Forem: Amit Upreti

Hands-on Web Scraping: Building your own twitter dataset with python and scrapy

Amit Upreti — Sun, 05 Jan 2020 04:25:21 +0000

I get it — You are tired of searching for datasets online for your machine learning project or maybe for analyzing a popular twitter trend.

Today we will learn how to generate your own custom dataset from twitter by using hashtag search.

Want to skip the post and see the good stuff directly? Here is the Github repo for you

amitupreti / Hands-on-WebScraping

This repo is a part of blog series on several web scraping projects where we will explore scraping techniques to crawl data from simple websites to websites using advanced protection.

Understanding the twitter policies for robots

First things first, let's make sure we follow the policies made by twitter.com for robots to follow while crawling so that we don’t run into any legal issues. Let's go to https://twitter.com/robots.txt

We can see twitter has allowed all the robots ( look at User-agent line ) to use the hashtag search (look at Allow: /hashtag… line) and requested to make a 1-second delay (look at Crawl-delay line) between the crawl requests.

User-agent: *
Allow: /hashtag/*?src=
Crawl-delay: 1

Walking through the code

This assumes that you have some basic knowledge of python and scrapy. If you are interested in only generating your own dataset, skip this section and go to sample crawl section directly or visit GitHub repo.

Gathering tweets URL by searching through hashtags

For searching for tweets we will be using the legacy twitter website. Let’s try searching for #cat https://mobile.twitter.com/hashtag/cats. This legacy version doesn't use javascript to load the data which makes our job super easy.

I choose a lazy person to do a hard job. Because a lazy person will find an easy way to do it.

― Bill Gates

Finding all the tweet URL for our hashtag search — find_tweets( ) function

If you want to look at the complete code at once. Here’s Github repo.

def find_tweets(self, response):
    tweets = response.xpath('//table[@class="tweet  "]/@href').getall()
    logging.info(f'{len(tweets)} tweets found')
    for tweet_id in tweets:
        tweet_id = re.findall("\d+", tweet_id)[-1]
        tweet_url = 'https://twitter.com/anyuser/status/' + \
                    str(tweet_id)
        yield scrapy.Request(tweet_url, callback=self.parse_tweet)

    # finding and visiting next page
    next_page = response.xpath(
        '//*[@class="w-button-more"]/a/@href').get(default='')
    logging.info('Next page found:')
    if next_page != '':
        next_page = 'https://mobile.twitter.com' + next_page
        yield scrapy.Request(next_page, callback=self.find_tweets)

Explanation: We will find the xpaths for all the tweets URL in the current page and crawl these URLs and send the response to our 2nd function parse_tweet ()

Now we will find the next URL for Load older Tweets button and crawl it and send the response to our current find_tweets() function. This way our crawler will keep clicking on the Load older Tweet button recursively if it is available on every crawl. This means we will visit all the results pages one by one.

**Finding all the data from an individual tweet — *parse_tweet( ) function***

def parse_tweet(self, response):
    logging.info('Processing --> ' + response.url)
    username = response.xpath(
        '//*[@class="permalink-inner permalink-tweet-container"]//*[@class="username u-dir u-textTruncate"]/b/text()').get(
        default='')
    full_name = response.xpath(
        '//*[@class="permalink-inner permalink-tweet-container"]//*[@class="FullNameGroup"]/strong/text()').get(
        default='')
    tweet_text = ' '.join(response.xpath(
        '//*[contains(@class,"permalink-inner permalink-tweet-container")]//*[@class="js-tweet-text-container"]/p//text()').getall()).strip()
    image_list = response.xpath(
        '//*[contains(@class,"permalink-inner permalink-tweet-container")]//*[@class="AdaptiveMediaOuterContainer"]//img/@src').getall()
    date_time = response.xpath(
        '//*[contains(@class,"permalink-inner permalink-tweet-container")]//*[@class="js-tweet-details-fixer tweet-details-fixer"]/div[@class="client-and-actions"]/span[@class="metadata"]/span/text()').get(
        default='')

    date_time = parser.parse(date_time.replace('-', '')).strftime('%Y-%m-%d %H:%M:%S')
    retweets = response.xpath(
        '//*[contains(@class,"permalink-inner permalink-tweet-container")]//*[@class="js-tweet-details-fixer tweet-details-fixer"]/div[@class="js-tweet-stats-container tweet-stats-container"]//*[@class="js-stat-count js-stat-retweets stat-count"]/a/strong/text()').get(
        default='')

    likes = response.xpath(
        '//*[contains(@class,"permalink-inner permalink-tweet-container")]//*[@class="js-tweet-details-fixer tweet-details-fixer"]/div[@class="js-tweet-stats-container tweet-stats-container"]//*[@class="js-stat-count js-stat-favorites stat-count"]/a/strong/text()').get(
        default='')
    replies = response.xpath(
        '//*[contains(@class,"permalink-inner permalink-tweet-container")]//*[contains(@id,"profile-tweet-action-reply-count")]/parent::span/@data-tweet-stat-count').get(
        default='')

    mentions = get_mentions(tweet_text)
    hashtags = get_hashtags(tweet_text)
    cta = get_links(tweet_text)

    result = {
        'username': username.lower(),
        'full_name': full_name,
        'twitter_url': response.url,
        'tweet_text': tweet_text,
        'tweet_time': str(date_time),
        'number_of_likes': str(likes),
        'no_of_retweets': str(retweets),
        'no_of_replies': str(replies),
        'mentions': ' | '.join(mentions),
        'no_of_mentions': str(len(mentions)),
        'hashtags': ' | '.join(hashtags),
        'no_of_hashtags': str(len(hashtags)),
        'call_to_action': ' | '.join(cta),
        'image_url': ' | '.join(image_list),

    }
    yield result
    print(result)

Explanation: Here we will use the current version of twitter (example: https://twitter.com/catsfootprint/status/1213603795663491075) as it turns our it loads all the data even without rendering javascript.

Let's do a sample crawl now.

First of all, lets set up the crawler

Installation

Download the project(for OS X and Linux)

git clone https://github.com/amitupreti/Hands-on-WebScraping

cd Hands-on-WebScraping/project1_twitter_hashtag_crawler

For Windows users

Download the project (Windows) here.
— Extract the project
— Open cmd and navigate inside the project directory

cd Hands-on-WebScraping/project1_twitter_hashtag_crawler

Install the dependencies

pip install -r requirements.txt --user

Verify the crawler spider exists

scrapy list

if you see twittercrawler , then you are all set.

Using your hashtags

Open myhashtags.csv file and put your hashtags separated by new line

Crawl and save data in your desired format(JSON,JSON lines,CSV,XML)

For CSV

scrapy crawl twittercrawler -a filename=myhashtags.csv -o mydata.csv

For JSON

scrapy crawl twittercrawler -a filename=myhashtags.csv -o mydata.json

For Jsonlines

scrapy crawl twittercrawler -a filename=myhashtags.csv -o mydata.jl

For XML

scrapy crawl twittercrawler -a filename=myhashtags.csv -o mydata.xml

You should now see the data being saved the output file with the format you choose

Speeding up the crawls

If you feel like the crawler is a little slow then find the hashtag.py file in the project and edit the custom settings.

custom_settings = {
    'USER_AGENT': 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36',
    'CONCURRENT_REQUESTS': 2, 'DOWNLOAD_DELAY': 1, 'LOG_LEVEL': 'INFO'}

Here CONCURRENT_REQUESTS is the number of URLs that will be processed parallelly and DOWNLOAD_DELAY is a wait between each request. So, Increase CONCURRENT_REQUESTS and decrease DOWNLOAD_DELAY (minimum value for download delay is 0).

If you have suggestions or find some issues.

Feel free to open an issue or a Pull Request on GitHub.

Thank you for reading.

YOU ARE AWESOME

This was originally posted on myblog.

Automating Lead Generation/Email Crawling with python

Amit Upreti — Wed, 13 Nov 2019 18:08:02 +0000

Want to skip the post and see the good stuff directly? Here is the Github repo

amitupreti / Email-Crawler-Lead-Generator

This email crawler will visit all pages of a provided website and parse and save emails found to a csv file.

Lead Generation is a very Lucrative business and people earn a ton of money just by finding emails to their client.

Today we will automate Lead Generation/Email Crawling with a simple python script.

Let's see what our end product will look like so that I won't waste your time in case you don't find this interesting.

Our crawler will visit all sub-pages of the provided website and look for emails and then save them in a CSV file.

See the code

First, let's see the code and then I will explain each step.

import re
import requests
import requests.exceptions
from urllib.parse import urlsplit, urljoin
from lxml import html
import sys
import csv


class EmailCrawler:

    processed_urls = set()
    unprocessed_urls = set()
    emails = set()

    def __init__(self, website: str):
        self.website = website
        self.unprocessed_urls.add(website)
        self.headers = {
            'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/78.0.3904.70 Chrome/78.0.3904.70 Safari/537.36',
        }
        self.base_url = urlsplit(self.website).netloc
        self.outputfile = self.base_url.replace('.','_')+'.csv'
        # we will use this list to skip urls that contain one of these extension. This will save us a lot of bandwidth and speedup the crawling process
        # for example: www.example.com/image.png --> this url is useless for us. we cannot possibly parse email from images and all other types of files.
        self.garbage_extensions = ['.aif','.cda','.mid','.midi','.mp3','.mpa','.ogg','.wav','.wma','.wpl','.7z','.arj','.deb','.pkg','.rar','.rpm','.tar.gz','.z','.zip','.bin','.dmg','.iso','.toast','.vcd','.csv','.dat','.db','.dbf','.log','.mdb','.sav','.sql','.tar','.apk','.bat','.bin','.cgi','.pl','.exe','.gadget','.jar','.py','.wsf','.fnt','.fon','.otf','.ttf','.ai','.bmp','.gif','.ico','.jpeg','.jpg','.png','.ps','.psd','.svg','.tif','.tiff','.asp','.cer','.cfm','.cgi','.pl','.part','.py','.rss','.key','.odp','.pps','.ppt','.pptx','.c','.class','.cpp','.cs','.h','.java','.sh','.swift','.vb','.ods','.xlr','.xls','.xlsx','.bak','.cab','.cfg','.cpl','.cur','.dll','.dmp','.drv','.icns','.ico','.ini','.lnk','.msi','.sys','.tmp','.3g2','.3gp','.avi','.flv','.h264','.m4v','.mkv','.mov','.mp4','.mpg','.mpeg','.rm','.swf','.vob','.wmv','.doc','.docx','.odt','.pdf','.rtf','.tex','.txt','.wks','.wps','.wpd']
        self.email_count = 0

    def crawl(self):
        """
        It will continue crawling untill the list unprocessed urls list is empty
        """

        url = self.unprocessed_urls.pop()
        print("CRAWL : {}".format(url))
        self.parse_url(url)


        if len(self.unprocessed_urls)!=0:
            self.crawl()
        else:
            print('End of crawling for {} '.format(self.website))
            print('Total urls visited {}'.format(len(self.processed_urls)))
            print('Total Emails found {}'.format(self.email_count))
            print('Dumping processed urls to {}'.format(self.base_url.replace('.','_')+'.txt'))
            with open(self.base_url.replace('.','_')+'.txt' ,'w') as f:
                f.write('\n'.join(self.processed_urls))

    def parse_url(self, current_url: str):
        """
        It will load and parse a given url. Loads it and finds all the url in this page. 
        It also filters the urls and adds them to unprocessed url list.
        Finally it scrapes the emails if found on the page and the updates the email list

        INPUT:
            current_url: URL to parse
        RETURN:
            None
        """

        #we will retry to visit a url for 5 times in case it fails. after that we will skip it in case if it still fails to load
        response = requests.get(current_url, headers=self.headers)
        tree = html.fromstring(response.content)
        urls = tree.xpath('//a/@href')  # getting all urls in the page


        #Here we will make sure that we convert the sub domain to full urls
        # example --> /about.html--> https://www.website.com/about.html
        urls = [urljoin(self.website,url) for url in urls]
        # now lets make sure that we only include the urls that fall under our domain i.e filtering urls that point outside our main website.
        urls = [url for url in urls if self.base_url == urlsplit(url).netloc]


        #removing duplicates
        urls = list(set(urls))


        #filtering  urls that point to files such as images, videos and other as listed on garbage_extensions
        #Here will loop through all the urls and skip them if they contain one of the extension
        parsed_url = []
        for url in urls:
            skip = False
            for extension in self.garbage_extensions:
                if not url.endswith(extension) and  not url.endswith(extension+'/'):
                    pass
                else:
                    skip = True
                    break
            if not skip:
                parsed_url.append(url)

        # finally filtering urls that are already in queue or already visited
        for url in parsed_url:
            if url not in self.processed_urls and url not in self.unprocessed_urls:
                self.unprocessed_urls.add(url)


        #parsing email
        self.parse_emails(response.text)
        # adding the current url to processed list
        self.processed_urls.add(current_url)



    def parse_emails(self, text: str):
        """
        It scans the given texts to find email address and then writes them to csv
        Input:
            text: text to parse emails from
        Returns:
            bool: True or false (True if email was found on page)
        """
        # parsing emails and then saving to csv
        emails = set(re.findall(r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+', text, re.I))
        #TODO: sometime "gFJS3amhZEg_z39D5EErVg@2x.png" gets accepted as email with the above regex. so for now i will check if email ends with jpeg,png and jpg

        for email in emails:
            skip_email = False
            for checker in ['jpg','jpeg','png']:
                if email.endswith(checker):
                    skip_email = True
                    break

            if not skip_email:    
                if email not in self.emails:
                    with open(self.outputfile, 'a', newline='') as csvf:
                        csv_writer = csv.writer(csvf)
                        csv_writer.writerow([email])
                    self.email_count +=1
                    self.emails.add(email)
                    print(' {} Email found {}'.format(self.email_count,email))

        if len(emails)!=0:
            return True
        else:
            return False



print('WELCOME TO EMAIL CRAWLER')
try:
    website = sys.argv[1]
except:
    website = input("Please enter a website to crawl for emails:")
crawl = EmailCrawler(website)
crawl.crawl()

Let's understand what is happening here

First part init function

class EmailCrawler:

    processed_urls = set()
    unprocessed_urls = set()
    emails = set()

    def __init__(self, website: str):
        self.website = website
        self.unprocessed_urls.add(website)
        self.headers = {
            'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/78.0.3904.70 Chrome/78.0.3904.70 Safari/537.36',
        }
        self.base_url = urlsplit(self.website).netloc
        self.outputfile = self.base_url.replace('.','_')+'.csv'
        # we will use this list to skip urls that contain one of these extension. This will save us a lot of bandwidth and speedup the crawling process
        # for example: www.example.com/image.png --> this url is useless for us. we cannot possibly parse email from images and all other types of files.
        self.garbage_extensions = ['.aif','.cda','.mid','.midi','.mp3','.mpa','.ogg','.wav','.wma','.wpl','.7z','.arj','.deb','.pkg','.rar','.rpm','.tar.gz','.z','.zip','.bin','.dmg','.iso','.toast','.vcd','.csv','.dat','.db','.dbf','.log','.mdb','.sav','.sql','.tar','.apk','.bat','.bin','.cgi','.pl','.exe','.gadget','.jar','.py','.wsf','.fnt','.fon','.otf','.ttf','.ai','.bmp','.gif','.ico','.jpeg','.jpg','.png','.ps','.psd','.svg','.tif','.tiff','.asp','.cer','.cfm','.cgi','.pl','.part','.py','.rss','.key','.odp','.pps','.ppt','.pptx','.c','.class','.cpp','.cs','.h','.java','.sh','.swift','.vb','.ods','.xlr','.xls','.xlsx','.bak','.cab','.cfg','.cpl','.cur','.dll','.dmp','.drv','.icns','.ico','.ini','.lnk','.msi','.sys','.tmp','.3g2','.3gp','.avi','.flv','.h264','.m4v','.mkv','.mov','.mp4','.mpg','.mpeg','.rm','.swf','.vob','.wmv','.doc','.docx','.odt','.pdf','.rtf','.tex','.txt','.wks','.wps','.wpd']
        self.email_count = 0

We have defined the following Sets

processed_urls --> will hold the URLs that we have visited(so that we won't visit the same URL twice)
unprocessed_urls --> will hold the URLs that are on the queue to parse
emails --> will hold the parsed emails.

We will use the base URL later to make sure our crawler doesn't visit outside URLs.
For example: if the user passes https://www.medium.com then the base URL would be medium.com. We will use this later to ensure that our crawler will only visit the URL within this domain.

crawl function

The crawl function is a starting point of our crawler. It will keep visiting all the URLs in the queue until we have visited every URL on the website.

def crawl(self):
        """
        It will continue crawling untill the list unprocessed urls list is empty
        """

        url = self.unprocessed_urls.pop()
        print("CRAWL : {}".format(url))
        self.parse_url(url)


        if len(self.unprocessed_urls)!=0:
            self.crawl()
        else:
            print('End of crawling for {} '.format(self.website))
            print('Total urls visited {}'.format(len(self.processed_urls)))
            print('Total Emails found {}'.format(self.email_count))
            print('Dumping processed urls to {}'.format(self.base_url.replace('.','_')+'.txt'))
            with open(self.base_url.replace('.','_')+'.txt' ,'w') as f:
                f.write('\n'.join(self.processed_urls))

parse url function

Our parse_urls function is where extraction happens. Here we

parse and filter all the URLs found on the given page.
We filtered duplicate URLs, URLs outside the domain and already visited URLs
We will also make sure that we don't try to visit urls that lead to files such as jpg,mp4,zips.
We finally parse the page for emails and then write them to a CSV file.

def parse_url(self, current_url: str):
        """
        It will load and parse a given url. Loads it and finds all the url in this page. 
        It also filters the urls and adds them to unprocessed url list.
        Finally it scrapes the emails if found on the page and the updates the email list

        INPUT:
            current_url: URL to parse
        RETURN:
            None
        """
        # visiting the page
        response = requests.get(current_url, headers=self.headers)
        tree = html.fromstring(response.content)
        urls = tree.xpath('//a/@href')  # getting all urls in the page


        #Here we will make sure that we convert the sub domain to full urls
        # example --> /about.html--> https://www.website.com/about.html
        urls = [urljoin(self.website,url) for url in urls]
        # now lets make sure that we only include the urls that fall under our domain i.e filtering urls that point outside our main website.
        urls = [url for url in urls if self.base_url == urlsplit(url).netloc]


        #removing duplicates
        urls = list(set(urls))


        #filtering  urls that point to files such as images, videos and other as listed on garbage_extensions
        #Here will loop through all the urls and skip them if they contain one of the extension
        parsed_url = []
        for url in urls:
            skip = False
            for extension in self.garbage_extensions:
                if not url.endswith(extension) and  not url.endswith(extension+'/'):
                    pass
                else:
                    skip = True
                    break
            if not skip:
                parsed_url.append(url)

        # finally filtering urls that are already in queue or already visited
        for url in parsed_url:
            if url not in self.processed_urls and url not in self.unprocessed_urls:
                self.unprocessed_urls.add(url)


        #parsing email
        self.parse_emails(response.text)
        # adding the current url to processed list
        self.processed_urls.add(current_url)

parse email function

It takes a text input and then finds emails on that text and finally writes these emails to a CSV file.



def parse_emails(self, text: str):
    """
    It scans the given texts to find email address and then writes them to csv
    Input:
        text: text to parse emails from
    Returns:
        bool: True or false (True if email was found on page)
    """
    # parsing emails and then saving to csv
    emails = set(re.findall(r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+', text, re.I))
    #TODO: sometime "gFJS3amhZEg_z39D5EErVg@2x.png" gets accepted as email with the above regex. so for now i will check if email ends with jpeg,png and jpg

    for email in emails:
        skip_email = False
        for checker in ['jpg','jpeg','png']:
            if email.endswith(checker):
                skip_email = True
                break

        if not skip_email:    
            if email not in self.emails:
                with open(self.outputfile, 'a', newline='') as csvf:
                    csv_writer = csv.writer(csvf)
                    csv_writer.writerow([email])
                self.email_count +=1
                self.emails.add(email)
                print(' {} Email found {}'.format(self.email_count,email))

    if len(emails)!=0:
        return True
    else:
        return False

How do i run this code?

To get a local copy up and running follow these simple steps.

Installation

Clone the Email-Crawler-Lead-Generator

git clone https://github.com/nOOBIE-nOOBIE/Email-Crawler-Lead-Generator.git

Install dependencies

pip install -r requirements.txt

Usage

Simply pass the url as an argument

python email_crawler.py https://medium.com/

### Output

  ➜  email_crawler python3 email_crawler.py https://medium.com/
WELCOME TO EMAIL CRAWLER
CRAWL : https://medium.com/
 1 Email found press@medium.com
 2 Email found u002F589e367c28ca47b195ce200d1507d18b@sentry.io
CRAWL : https://medium.com/creators
 3 Email found joshsrose@me.com
 4 Email found yourfriends@medium.com
 5 Email found partnerprogram@medium.com
 6 Email found dominiquemattiwrites@gmail.com
 7 Email found hihumanparts@gmail.com
CRAWL : https://medium.com/@mshannabrooks
CRAWL : https://medium.com/m/signin?operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40mshannabrooks&source=listing-----5f0204823a1e---------------------bookmark_sidebar-
CRAWL : https://medium.com/m/signin?operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40mshannabrooks&source=-----e5d9a7ef4033----6------------------

If you have suggestions or find some issues.

Feel free to open issues or a Pull Request on github.

Thank you for reading

YOU ARE AWESOME

Building an ImageDownloader with multiprocessing in python

Amit Upreti — Sun, 10 Nov 2019 09:55:11 +0000

Want to skip the post and see the full code directly: here is the github repo

amitupreti / image_downloader_multiprocessing_python

Downloading Images with multiprocessing using python requests and multiprocessing.

I get it, you are tired of waiting for your program to download images. It takes hours sometimes when I have to download thousands of images and you possibly can't keep waiting for your program to finish downloading these dumb images. You have a lot of important stuff to do.

Let's build a simple image downloader script that will read a text file and download all the images listed nicely in a folder superfast.

Final result

This is what we will be building at the end.

Install the dependencies

let's install the everybody's favorite requests library.

pip install requests

Now, we will see some basic code to download a single URL and try to automatically find the name of the image and also how to use retries.

import requests

res = requests.get(img_url, stream=True)
count = 1
while res.status_code != 200 and count <= 5:
    res = requests.get(img_url, stream=True)
    print(f'Retry: {count} {img_url}')
    count += 1

Here we retry to download the images 5 times in case it fails. Now let's try to automatically find the name of image and save it

#import more required library

import io
from PIL import Image

#lets try to find the image name
image_name = str(img_url[(img_url.rfind('/')) + 1:])
if '?' in image_name:
    image_name = image_name[:image_name.find('?')]

Explanation

Let's consider the url we are trying to download is

https://instagram.fktm7-1.fna.fbcdn.net/vp/581ba4467732636cab13a38c7bd9f796/5E4C7217/t51.2885-15/sh0.08/e35/s640x640/65872070_1200425330158967_6201268309743367902_n.jpg?_nc_ht=instagram.fktm7-1.fna.fbcdn.net&_nc_cat=111

Well, this is a mess. Let's break what code does to the url. We first find the last frontslash(/) with rfind and then select everything after that. This is the result

65872070_1200425330158967_6201268309743367902_n.jpg?_nc_ht=instagram.fktm7-1.fna.fbcdn.net&_nc_cat=111

Now our second part finds a ? and then only takes whatever is in front of it.

This is our final image name

65872070_1200425330158967_6201268309743367902_n.jpg

This result is pretty good and will work for most of cases.

Now that we have our image name and image downloaded. We will now save it.

i = Image.open(io.BytesIO(res.content))
i.save(image_name)

If you are thinking how the heck am I supposed to use the code above, then you are thinking correct. Here is a pretty function with everything we did above squished. Here we also tested if the downloaded type is an image and in case we couldn't find the image name

def image_downloader(img_url: str):
    """
    Input:
    param: img_url  str (Image url)
    Tries to download the image url and use name provided in headers. Else it randomly picks a name
    """
    print(f'Downloading: {img_url}')
    res = requests.get(img_url, stream=True)
    count = 1
    while res.status_code != 200 and count <= 5:
        res = requests.get(img_url, stream=True)
        print(f'Retry: {count} {img_url}')
        count += 1
    # checking the type for image
    if 'image' not in res.headers.get("content-type", ''):
        print('ERROR: URL doesnot appear to be an image')
        return False
    # Trying to red image name from response headers
    try:
        image_name = str(img_url[(img_url.rfind('/')) + 1:])
        if '?' in image_name:
            image_name = image_name[:image_name.find('?')]
    except:
        image_name = str(random.randint(11111, 99999))+'.jpg'

    i = Image.open(io.BytesIO(res.content))
    download_location = 'cats'
    i.save(download_location + '/'+image_name)
    return f'Download complete: {img_url}'

Now, "where is this multiprocessing this guy was talking about?" you might say.

This is very simple. We will simply define our pool and pass our function and image URL to it.

results = ThreadPool(process).imap_unordered(image_downloader, images_url)
for r in results:
    print(r)

Let's put this in a function

def run_downloader(process:int, images_url:list):
    """
    Inputs:
        process: (int) number of process to run
        images_url:(list) list of images url
    """
    print(f'MESSAGE: Running {process} process')
    results = ThreadPool(process).imap_unordered(image_downloader, images_url)
    for r in results:
        print(r)

Again, you might say this is all okay stuff. But I want to start to download my list of 1000 images right away. I don't want to copy and paste these codes and try figuring out merging everything.

Here is a full script. It does the following

Takes image list text file and number of the process as input
Downloads them as per your desired speed.
And also prints the total time taken to download the files.
There are also a few nice functions that helps us to read the filename and handle errors and stuff.

Full script

# -*- coding: utf-8 -*-
import io
import random
import shutil
import sys
from multiprocessing.pool import ThreadPool
import pathlib

import requests
from PIL import Image
import time


start = time.time()


def get_download_location():
    try:
        url_input = sys.argv[1]
    except IndexError:
        print('ERROR: Please provide the txt file\n$python image_downloader.py cats.txt')
    name = url_input.split('.')[0]
    pathlib.Path(name).mkdir(parents=True, exist_ok=True)
    return name


def get_urls():
    """
    Returns a list of urls by reading the txt file supplied as argument in terminal
    """
    try:
        url_input = sys.argv[1]
    except IndexError:
        print('ERROR: Please provide the txt file\n Example \n\n$python image_downloader.py dogs.txt \n\n')
        sys.exit()
    with open(url_input, 'r') as f:
        images_url = f.read().splitlines()

    print('{} Images detected'.format(len(images_url)))
    return images_url


def image_downloader(img_url: str):
    """
    Input:
    param: img_url  str (Image url)
    Tries to download the image url and use name provided in headers. Else it randomly picks a name
    """
    print(f'Downloading: {img_url}')
    res = requests.get(img_url, stream=True)
    count = 1
    while res.status_code != 200 and count <= 5:
        res = requests.get(img_url, stream=True)
        print(f'Retry: {count} {img_url}')
        count += 1
    # checking the type for image
    if 'image' not in res.headers.get("content-type", ''):
        print('ERROR: URL doesnot appear to be an image')
        return False
    # Trying to red image name from response headers
    try:
        image_name = str(img_url[(img_url.rfind('/')) + 1:])
        if '?' in image_name:
            image_name = image_name[:image_name.find('?')]
    except:
        image_name = str(random.randint(11111, 99999))+'.jpg'

    i = Image.open(io.BytesIO(res.content))
    download_location = get_download_location()
    i.save(download_location + '/'+image_name)
    return f'Download complete: {img_url}'


def run_downloader(process:int, images_url:list):
    """
    Inputs:
        process: (int) number of process to run
        images_url:(list) list of images url
    """
    print(f'MESSAGE: Running {process} process')
    results = ThreadPool(process).imap_unordered(image_downloader, images_url)
    for r in results:
        print(r)


try:
    num_process = int(sys.argv[2])
except:
    num_process = 10

images_url = get_urls()
run_downloader(num_process, images_url)


end = time.time()
print('Time taken to download {}'.format(len(get_urls())))
print(end - start)

save it to a python file and let's run it.

python3 image_downloader.py cats.txt

Dude give me a Github link.

Here is a link to the Github repo
https://github.com/nOOBIE-nOOBIE/image_downloader_multiprocessing_python

Dude teach me how to use it

Usage

python3 image_downloader.py <filename_with_urls_seperated_by_newline.txt> <num_of_process>

This will read all the urls in the text file and download them into a folder with name same as the filename.

num_of_process is optional.(by default it uses 10 process)

Example

python3 image_downloader.py cats.txt

Would love any response on how this can be improved further.

Forem: Amit Upreti

Hands-on Web Scraping: Building your own twitter dataset with python and scrapy

amitupreti / Hands-on-WebScraping

This repo is a part of blog series on several web scraping projects where we will explore scraping techniques to crawl data from simple websites to websites using advanced protection.

Understanding the twitter policies for robots

Walking through the code

Gathering tweets URL by searching through hashtags

Finding all the tweet URL for our hashtag search — find_tweets( ) function

Finding all the data from an individual tweet — *parse_tweet( ) function*

Let's do a sample crawl now.

First of all, lets set up the crawler

YOU ARE AWESOME

Automating Lead Generation/Email Crawling with python

amitupreti / Email-Crawler-Lead-Generator

This email crawler will visit all pages of a provided website and parse and save emails found to a csv file.

See the code

First part init function

crawl function

parse url function

parse email function

How do i run this code?

Installation

Usage

YOU ARE AWESOME

Building an ImageDownloader with multiprocessing in python

amitupreti / image_downloader_multiprocessing_python

Downloading Images with multiprocessing using python requests and multiprocessing.

Final result

Install the dependencies

Explanation

Full script

save it to a python file and let's run it.

Dude give me a Github link.

Dude teach me how to use it

Usage

Example

# Happy coding!

**Finding all the data from an individual tweet — *parse_tweet( ) function***