Forem: Natasha Lekh

Web scraping in 2024: breakthroughs and challenges ahead

Natasha Lekh — Sun, 28 Jan 2024 23:00:00 +0000

This article was first published on December 15, 2023, and updated on January 29, 2024, to reflect recent updates in the legal landscape of web scraping.

How did 2023 treat the web scraping industry? Let's take a short walk through the bad, the good, and the different of yesteryear. Welcome to a summary of the key events and trends that emerged in 2023, setting the stage for the landscape of 2024.

🎄 Want to compare to what web scraping was like in 2022? Check out our overview from the last year .

🧑 Irony of the year

The year started off funny. In 2022, Meta was very keen on sui ng individuals and companies for web scraping; in 2023, it continued to zero in even on its recent allies. The culprit in question, Bright Data, got sued by Facebo ok for scraping Facebook data. The trick is that Facebook was using Bright Data's services previously for scraping data (just from other websites). Essentially, Meta inadvertently revealed its practice of collecting data from other websites through its lawsuit against a firm it employed for this very purpose. Quite some web scraping uroboros there. This situation once more highlighted the two aspects of an age-old industry question: who really owns publicly accessible data, and is it okay to gather it?

🆕 In 2024, the court ruled against Meta and in favor of web scraping . The judge dismissed Meta's breach of contract claim, arguing that even though Bright Data has accepted the terms of service of Facebook and Instagram, the company was not acting as a "user" of the services when it was scraping but only as a logged-out "visitor," who is not bound by the terms.

In a cruel twist of fate, later last year, Meta got another billion-sized fine (as big as the previous 6 combined, apparently) from the Irish DPC for not protecting the data of EU citizens from surveillance. The Irish Data Protection Commission and EU are not playing when it comes to data privacy. The penalty relates to an inquiry that was opened by the DPC back in 2020. And it seems like in 2024 Meta will be facing several other lawsuits regarding ad space and its pay-or-consent policy, this time from Spanish media and Austrian data protection authority.

The pack of plaintiffs claiming the violation of terms and conditions was extended by a new member, with Air Canada filing suit against travel search site seats.aero in a similar case, alleging unlawful scraping of its website and thus violating its terms of conditions. Interestingly however, Air Canada also claims breach of criminal law under the Computer Fraud and Abuse Act (CFAA). This move could signal that, although claims on these grounds have been in the past dismissed by courts in Van Buren 2021 and the hiQ April 2022 ruling built on it, the CFAA has still not lost its allure for all of the websites wanting to sue web scraping companies.

👀 The non-event of the year

There are always a few things in life to be grateful for because they did not happen: the extinction of the bees, the eruption of Yellowstone Volcano, and the Google WEI Proposal. The Web Environment Integrity (WEI) proposal, which was pushed by Google, was eventually abandoned this year (not in the least due to the protest of the defenders of the free web see screenshot from explainers-by-googlers issues).

Issues in explainers-by-googlers after the announcement of the WEI Proposal

Google was trying to follow the likes of Apple to replace Captchas with a digitally signed token, an API containing a digitally signed token, to be precise. The reason seemed innocuous: to help separate real users from bot users, and real traffic from bot traffic, and limit online fraud and abuse all this without enabling privacy issues like cross-site tracking or browser fingerprinting. Sounds like a dream, right?

However, while it might aid in reducing ad fraud, Google's proposed method of authentication also carries the risk of curtailing web freedom by allowing websites or third parties to directly influence the choice of browsers and software used by visitors. It could also potentially lead to misuses, such as rejecting visitors using certain tools like ad blockers or download managers.

Besides, Google intended to implement the Web Environment Integrity API in Chromium, the open-source base for Chrome and several other browsers, excluding Firefox and Safari. This, in comparison, makes Apple's Private Access Token seem way less dangerous, not in the least part because Safari has a much smaller browser market share than Chrome.

The drawbacks were quickly noticed by the open web proponents in the tech community. Critics quickly recognized the potential for this to evolve into a kind of digital rights/restriction management for the web. They also highlighted that this change would wildly benefit the ad companies but create high risks of disadvantaging the users. It would also make scraping and web automation activities significantly harder. Well, for everyone except for Google, of course.

The rejection of WEI by the tech community again highlights the importance of maintaining an open and accessible web.

🁫 First domino of the year

Scraping social media is the most common web scraping use case. In the old internet days, websites kept their APIs free and accessible, and even if they backed down from that, they often left a free version for the developers. The year started with X (Twitter)'s move to a paid API model, which meant discontinuing free access even for developers. A few months later, Reddit followed suit with its API transition to a paid model which caused significant uproar and protests.

X's API policy changes might have contributed to the more frequent occurrences of Twitter scraping. With many projects forced to shut down due to the three price tiers, it's very likely that some developers had to turn to web scraping and browser automation as an alternative. We tried to keep up with these changes ourselves as providers of a more affordable Twitter API and Reddit API. But it's becoming increasingly difficult or inconvenient to scrape these websites without a reliable infrastructure.

👺 Troublemaker of the year

Last year the web scraping case law made strides with the hiQ vs. LinkedIn case. 2023 had been rather calm on the legal side of things, if not for one particular persona. If, in 2022, Meta was the one suing individuals and companies for harvesting data, this year was a debut for X (Twitter). To be fair, the year 2023 was a debut for a lot of things for Twitter, but let's focus on the thing in question.

Elon Musk, the tech mogul, made headlines with public promises to take legal action against web scraping companies. This move was followed by X adding and then silently removing the login requirement for viewing posts (tweets) and following through with the promise by initiating lawsuits against 4 unknown individuals. And before all that, Musk has made Twitter API paid. But let's follow step-by-step here.

In July 2023, Elon Musk (well, X Corp, if we're being precise) gave us all some heat by initiating legal action against four anonymous entities who were scraping Twitter. Apparently, the four defendants overwhelmed Twitter's registration page with automated requests to such an extent that it caused a significant server strain and disruption of service for users. The culprits are accused of overburdening Twitter's servers, diminishing user experience, and profiting unjustly at the company's expense.

And as a regular cherry on top, the lawsuit further accuses them of scraping Twitter user data in breach of the platform's user agreement. These days, breach of Terms of Service has become companies favorite reference when instigating lawsuits against web scraping. Seconded only by scraping data for large language models training which is a concern raised by Elon Musk as well. Despite these latter allegations, he did confirm that his recently launched firm, xAI, will use X posts for training purposes. So go figure.

The lawsuit suggests that the intensive data scraping led to such severe performance issues that X had to enforce a login requirement for access for everyone. Users are now required to have an account to view tweets and must subscribe to Twitter Blue's "verified" service to see over 600 posts per day.

Now, we don't know for sure whether the AI data scraping was so intense that it could have impacted the website as much. However, this lawsuit and the argumentation behind it raise concerns about the potential for misrepresenting ethical data scraping practices, especially companies that adhere to legal and ethical standards in data collection.

Resources:

📈 Trend of the year

**AI brings a new way to easily process large amounts of data something that required developing complex and special machine learning models before. These days anybody can do, for instance, sentiment analysis with LLMs.

Marek Trunkat, CTO of Apify**

Web scraping really became the household term after the waves caused by ChatGPT and OpenAI this year. Why? Because web scraping was heavily involved in the training process. In Google Trends, among the regular adjacent topics such as point-and-click or proxy, we see AI. And this trend is here to stay.

We were happy to observe that making a one-off regular web scraper using AI is so easy these days. The AI hype makes it seem simple and accessible even without coding knowledge. It's the reliability and continuity of scraping that the AI cannot guarantee, especially with websites employing their own blocking measures based on AI.

AI is the adjacent trend of the year in web scraping

🦾 AI and the hunt for data

**The AI revolution of 2023 only underscored the already growing need for data from the web. All large language models (LLMs) like GPT-4 and LLaMA-2 were trained on data scraped from the web. As demand for AI and LLM applications will continue to grow, so will grow the demand for web scraping and data extraction.

Jan Curn, Apify Founder & CEO**

The large language models that power ChatGPT and other AI chatbots get their mastery of language from essentially two things: massive amounts of training data scraped from the web and massive amounts of compute power to learn from that data. That second ingredient is very expensive, but the first ingredient, so far, has been completely free.

However, creators, publishers, and businesses increasingly see the data they put on the web as their property. If some tech company wants to use it to train its LLMs, they want to be paid. Just ask the Associated Press, which struck a training data licensing deal with OpenAI. Meanwhile, X (ne Twitter) has taken steps to block AI companies from scraping content on the platform.

Web data and RAG

The knowledge of LLMs is limited to the public data they were trained on. Building AI applications that can retrieve proprietary data or public data introduced after a models cutoff date and generate content based on it requires augmenting the knowledge of the model with specific information. That process is known as retrieval-augmented generation (RAG), and it has revolutionized search and information retrieval.

While the likes of LangChain and LlamaIndex swiftly took center stage in this field, web scraping (being the most efficient way to collect web data) has remained a significant part of RAG solutions.

**To work around the training data cutoff date problem to provide models with up-to-date knowledge, LLM applications often need to extract data from the web. This so-called retrieval-augmented generation (RAG) is what gives the LLMs the superpowers and arguably this is the strongest use case of LLMs.

Jan Curn, Apify Founder & CEO**

Adding data to custom GPTs

OpenAI launched GPTs (custom versions of ChatGPT) in November 2023. This was a really big deal, as suddenly, everyone had the means to build their own AI models. These GPTs can be customized not only with instructions but also with extra knowledge (by uploading files) and a combination of skills (with API specifications). In other words, you can give such GPTs web scraping capabilities with the right specs or scrape websites to upload knowledge to a GPT so it can base generated content on that information.

The hype around GPTs was quickly replaced by a huge furore around the firing and return of OpenAIs CEO. As a result, the debut of GPT Store, which lets users monetize their GPTs, was postponed and finally launched in early 2024.

EU AI Act represents break-through legislation for AI and web scraping

After the global shake-up in the world of personal data protection represented by GDPR, the EU reached a provisional agreement on the EU AI Act, which has similar ambitions for the world of artificial intelligence as GDPR had for personal data. Hailed by EU officials as global first and historic, the Act positions the EU as a frontrunner in the field of AI regulation.

The EU adopted a risk-based approach, defining four different classes of AI systems. The AI systems are divided into four categories: (1) unacceptable risk, (2) high risk, (3) limited risk, and (4) minimal/no risk.

Firstly, in the unacceptable risk category will belong to those AI systems which contravene EU values and are considered to be a threat to fundamental rights. These systems will be banned entirely. Among others, this category will include:

biometric categorization systems that use sensitive characteristics (e.g., political, religious, philosophical beliefs, sexual orientation, race, etc.);
untargeted scraping of facial images from the Internet or CCTV footage to create facial recognition databases;
emotion recognition, social scoring, AI systems manipulating human behavior, or exploiting vulnerabilities of people (due to their age, disability, social or economic situation, etc.).

However, the EU regulators incorporated several exceptions to using AI systems in this category, such as the use of biometric identification systems for law enforcement purposes, which will be subject to prior judicial authorization and only for a strictly defined list of crimes.

Secondly, the Act will include some AI systems in the high-risk category due to their significant potential harm to health, safety, fundamental rights, the environment, democracy, and the rule of law. Among others, this category will include AI systems in the field of medical devices, certain critical infrastructure, systems used to influence the outcome of elections or voter behavior, and more. These systems will be subject to comprehensive mandatory compliance obligations, such as fundamental rights impact assessment, conducting model evaluations and testing, reporting serious incidents, etc.

Thirdly, the AI systems classified as limited risk, such as chatbots, will be subject to minimal obligations, such as the requirement to inform users that they are interacting with an AI system and the obligation to mark the image, audio, or video content generated by AI.

Lastly, all AI systems not classified in one of the other three categories will be classified as minimal/no risk. The Act allows for the free use of minimal and no-risk AI systems, with voluntary codes of conduct encouraged.

Violations of the Act will be subject to fines, depending on the type of AI system, the size of the company, and the severity of the infringement.

Resources:

🌟 Apify's contributions

Of course, we could not pass up an opportunity to contribute to the party. For the 8 years that Apify has been on the market from the early days in Y Combinator to the transition from Apifier to now, it's been our goal to develop the cloud computing platform for automation and web scraping tools. So here's what we did this year to come a little bit closer to that goal.

Support for Python users: SDK, code templates, and Scrapy spiders

We've started the year off pretty strong by taking a significant and probably unexpected step forward. In March 2023 (on Pi Day, to be precise), we launched Python SDK to expand our toolset for Python developers. Now, if you know anything about Apify, you know that we have traditionally been on Node.js/JavaScript side of things. But things change, and so does the market and requests from our users. Being a start-up means venturing into different directions and trying different things when the situation calls for it. And since we consistently work on becoming the platform for web scraping and automation, launching a library for Python developers, giving them something to start from, just made sense.

As a follow-up step, we've rolled out web scraping templates aimed to simplify and improve the developer experience on our platform. We've realized not everyone wants to use ready-made tools in the Store or have complete control over every single aspect when building a scraper like with Crawlee. Web scraping templates seemed like a great third option, so here they are: in JavaScript, TypeScript, and Python. We've also launched the $1/month Creator Plan to support our most avid and enthusiastic users who are interested in building Actors.

Last but not least, we've made it possible to deploy Scrapy spiders to our cloud platform. All you have to do is use a Scrapy wrapper. The platform provides proxies and API and allows our Python users to run, schedule, monitor, and monetize their spiders.

Store and community growth

This year we've had to deal with unprecedented interest and growth of our Actors published in Store. The number of users engaging with Public Actors in Store has doubled , soaring from 8,971 to 17,070. In terms of new contributions, we've seen a significant influx, with 657 new Actors being published this year, a substantial increase compared to the 290 in 2022. Moreover, our community has been enriched by the addition of 96 new community developers , who have joined us with their Public Actors, doubling the number from the 48 who joined in 2022. This growth not only reflects the rising popularity of our platform but also underscores the expanding ecosystem for web scraping and automation we're building together.

New integrations and AI ventures

We've launched integrations with LlamaIndex and LangChain, marking a notable expansion in its collaboration network. These integrations mean you can load scraped datasets directly into LangChain or LlamaIndex vector indexes and build AI chatbots such as Intercom's Fin or other apps that query text data crawled from websites.

We've also introduced 3 AI tools in our Store to help fuel large language models and the likes: GPT Scraper and Extended GPT Scraper, Website Content Crawler, and AI Web Agent. Last but not least, we've launched a not LLM-related but nevertheless web scraping solution with AI at its core, AI Product Matcher.

Blog and YouTube

Regarding content, you can notice that our blog switched to a more technical approach, as well as our YouTube tutorials. We've also recorded our first podcast about the legality of web scraping. We've held three webinars on pretty extensive topics and experimented with posting Shorts. Our internal user engagement is as strong as ever with our newsletter reaching over 68K people every month, with around a 65% open rate. You can now subscribe to an online version of it on LinkedIn if you don't like your inbox crowded.

Apify platform

Our crown jewel, the Apify platform, is evolving day by day, not only design and UX-wise but also functionality-wise. We are currently working on a video of a new tour of Apify that will showcase all the new features and changes made this past year. But for now, here's something to look back on and appreciate the progress:

See you in the new year!

10 Google search tricks (that are also Google scraping tricks)

Natasha Lekh — Thu, 23 Nov 2023 23:00:00 +0000

Can you apply Google search tricks to scraping and data extraction as well? Lets put it to the test!

We're Apify , the only full-stack web scraping platform. You can build, deploy, share, and monitor scrapers or APIs for any website on Apify.

The savvy Googlers among us always have a few tricks up their sleeve. The question is: can you apply these Google search tricks to scraping and data extraction as well? Lets put it to the test! But before we start, we need to address the elephant in the blog.

🤔 What do Google search shortcuts have to do with web scraping?

Millions of people rely on Google search every day. Be it for school, research, or simple entertainment, if you know a few Google search shortcuts, your search process is more efficient. The thing is, a lot of those Google search tricks can also apply to web scraping.

The reason for this is simple: what a Google scraper does is very similar to what a Google visitor does: it goes to the google.com website, types in a query (even if it contains a shortcut), and receives results. The only difference is that the scraper also copies the results at lightning speed and packages them into a file.

This means that, if you are familiar with Google tricks and shortcuts, you can use that knowledge to upgrade your Google scraping process. When we built our Google Search Scraper 🔗 back in the day, we didn't count on this. Now that Google Scraper (also known as Google SERP API) has over 40,000 users, we feel obliged to everyone know about this interesting peculiarity.

https://blog.apify.com/unofficial-google-search-api-from-apify-22a20537a951/

Learn how to use our Google Search Scraper without tricks

💡 10 Google search tricks (that are also Google scraping tricks)

So let's put 10 well-known tricks to the test and level up your Google scraping. In other words, let's learn how to scrape Google like a pro 😎

Use site: to scrape specific sites

#1. Add site: after your keyword to narrow down the search to a specific website. For example, visualize site:blog.apify.com

This is probably the most well-known Google search shortcut to narrow down your search on a specific website without visiting it. The thing is, you can use this same trick not only to search but also to scrape content from that particular website. The syntax is very simple keyword + site:website.com. The screenshot above shows how you can apply it to our Google Scraper 🔗.

Our query will scrape all content from Google related to the word visualize but only from our Blog, blog.apify.com. All other scraping results will be filtered out. If you need to scrape specific content from a particular site, this is the shortcut to go for.

Quotation marks for exact scraping queries

#2. Surround your phrase or word with " " quotation marks for exact scraping queries

For a regular search, Google (and Google Scraper by extension) will get content containing the words of your query in any order. But you can use quotes to make your Google scraping query laser-accurate. No similar phrases, no swapping words around, no adjacent topics, just word-by-word accuracy.

Let's see whether we can get away with this by choosing a specific, very long-tail keyword to scrape for: "Headless browsers, infrastructure scaling, sophisticated blocking. Meet the full-stack platform that makes it all easy." This whole phrase can only be found on the Apify homepage. Will the Google SERP Scraper find it?

It did! So, surrounding your scraping keyword with quotes will instruct the scraping tool to scrape Google with that specific phrase in mind. This tip can also piggyback on the previous one: you can include quotes to search for specific wording on any website. We'll come back to mixing up various tricks in tip #10.

Hyphen to exclude certain results

#3. Add - in front to exclude certain results beforehand

This shortcut is useful for cases when you want to scrape data about one topic but filter out content about another. In other words, when you don't want a specific term to show up among your Google scraped results. For example, you want to scrape information about web scraping (going slightly meta there) but exclude any Python-related pages.

You can set this up by using - in front of unwanted keywords. In our example, the hyphen instructs the Google scraping tool to ignore any content that contains the word Python. And you won't find any Python-related pages among the results. The best part about this trick is that you can filter out information that you want to keep even before you start scraping.

Link: to scrape websites with backlinks

#4. Use link: to scrape websites containing backlinks of your choice

This Google scraping tip is no.1 for all SEO enthusiasts out there. Tracking backlinks is one of the most basic SEO practices because, as a rule, the more backlinks your page has, the better your Google ranking. Even better if those backlinks are "high quality," as in coming from domains with high domain authority. Essentially, the number of backlinks is a number one indicator that your website's content is valuable (since it's trusted by websites that decide to share it).

So the gist of this Google scraping trick is: instead of just scraping a page, we're going to scrape all pages that link to that specific page. Let's extract pages with a backlink to apify.com, a.k.a all pages that mention apify.com on their page. Phew, that was a mouthful, but with a simple link: apify.com we were able to catch them all.

Keep in mind that the more targeted your query is (focusing on a specific URL, for example link: apify.com/product-matching-ai/faq), the fewer results you'll get. This happens because most pages link to the main domain page rather than specific ones.

Related: to scrape similar websites or competition

#5: use related: to scrape similar websites or competition

The related: trick is a scraping technique that could be a game-changer for market researchers. When you apply related to, let's say amazon.com, you won't scrape links to Amazon. Instead, what you'll get are links to online stores similar to Amazon. Think of any e-commerce platform, such as Walmart, Kohl's, and other retailers that sell goods online. The scraping results will depend on the domain you've chosen.

By scraping with related: you can see which companies, organizations, or other entities are perceived as competition to the page you've indicated. So you can think of this Google scraping trick as a fast way to identify competitors in a given industry at least the ones that count in a digital space.

OR to scrape Google using multiple keywords at once

#6. Use OR to scrape using multiple keywords at once

This Google scraping trick allows you to scrape for multiple queries at once. For instance, let's say we want to scrape pages featuring recipes for both mustard dressing and vinaigrette dressing. By placing a simple OR between these phrases, we can make sure that our search (and subsequent scraping query) includes pages containing both of these delicious terms. To make this Google scraping trick even more precise, consider using quotation marks around your queries.

Asterisk to scrape wildcard data from Google

#7. Use * asterisk to scrape wildcard data

The asterisk wildcard is another nifty trick for Google scraping. When you insert an * into your scraping query, it acts as a flexible placeholder, which the Google scraper can later fill in. This tip is particularly handy when you don't have all the words at your fingertips. To best illustrate this, let's use an example with song lyrics. So, for our example, let's search for the lyrics of a famous Queen song by taking two random parts from verse three and placing an asterisk between them.

As the scraping tool works its magic, it understands that the asterisks could represent any word or series of words bridging our queries. More often than not, the result will include the exact lyrics of the song we're targeting. But this trick isn't limited to just song lyrics. Whether it's a specific social media post, an elusive item, a lengthy name, or an article title that's on the tip of your tongue, the asterisk wildcard can make your Google scraping just a little bit more interesting.

Filetype: to scrape files of specific format

#8. Use keyword + filetype: to scrape files of specific format

filetype: is as simple as it sounds. This Google scraping trick will get you any file on the open web. Just enter your keyword + filetype: followed by a file extension type: PDF, DOCX, or HTML. So for example, for your scraping query harry potter filetype:pdf you'll get a collection of Harry Potter-related PDFs. But the scope of this scraping trick isn't confined to these formats alone. You can scrape Google for any type of file it accepts, including PowerPoint Presentations (PPT), LaTeX Documents (TEX), and even Google Earth maps (KML).

Scrape results before, after, and between periods of time using BEFORE, AFTER and . .

#9. Scrape results before, after, and between periods of time using BEFORE, AFTER and . .

This trick lets you scrape Google pages in chronological order. Enter your keyword followed by the desired time frame before, after, or within a specific period. For example, if we're aiming to scrape Google Maps scraping tutorials published after 2022, our query would be: google maps scraping tutorial AFTER:2022. After applying this, our Google scraping results will exclusively feature tutorials from 2022 onwards, sparing us the effort of sifting through older, irrelevant data. A little caveat, though: you can't scrape anything earlier than the dawn of the internet.

Mix them up!

#10. Challenge the Google Pages Scraper by mixing up " ", * *, and site: search

Last but not least, you can combine a lot of the scraping tricks you've just learned our Google SERP Scraper loves a challenge. In our example, we'll be looking for a very specific article that we don't fully remember the name of and narrow down our search to blog.apify.com. So we're using quotation marks, an asterisk, and a site search. Let's see if the search engine and scraper can find and get that article for us. It did! So go ahead and try out all of the tricks one by one or all at once.

🤹 Know any other tricks? Try them out on Google Scraper !

Google scraping fueled by the Apify platform

The best part of Google Scraper is that it enables you to scrape anything and everything you could ever need from the World Wide Web. It can do that because Google SERP Scraper is more than just a standalone tool; it's actually supercharged by the versatility of the Apify platform.

Because of the platform support, you're not limited to simply exporting scraped Google data in a range of formats or getting results for various Google domains. You also gain the convenience of accessing that data through an API, crafting custom integrations with other scrapers or your favorite apps, and scheduling and monitoring your scraping projects with ease.

Last but not least, the Apify platform makes sure our 40K+ users can scrape Google pages with confidence thanks to our specialized SERP proxies that are tailor-made for the job). All this to make data extraction from Google easy and reliable.

Top 5 social media scrapers in 2024

Natasha Lekh — Mon, 13 Nov 2023 23:00:00 +0000

Social media has become an integral part of modern marketing and research strategies. And while establishing a presence on social media platforms is a challenge of its own, there's a whole new dimension of insights waiting to be uncovered by web scraping and downloading the data these apps provide.But can anyone just scrape a website these days? In 2024, the answer is yes. Follow along to discover 5 tools that can do the heavy lifting for you.

Social media scrapers were created to extract and analyze data from platforms like Instagram, TikTok, Facebook, Reddit, and YouTube. Getting data from social media consistently and at scale enables digital media marketers, researchers, and business professionals to have a clear understanding of any audience and make more informed decisions about its engagement.

What exactly is web scraping?

Web scraping is the automated process of collecting data from websites. Usually, it involves accessing a website's HTML code and parsing it to extract specific information such as text, images, or pricing details. Web scraping is the go-to method for fast and systematic gathering of web data, especially data from social media. Find out more in our guide to web scraping 101.

🧑 Is scraping data from social media legal?

It's important to acknowledge that responsible and ethical scraping practices are key to preserving the integrity of the open web and the stability of scraped websites. If you want to adhere to these policies, it's useful to be familiar with each platform's usage terms, legal considerations, and privacy policies to ensure compliance. Our detailed guide on the do's and don'ts of web scraping is a great place to start.

Social media scraping generally falls into a legal gray area (especially when compared to scraping other website types), given that it often involves collecting personal information like names. However, there are numerous scenarios where scraping social media is not just permissible but necessary. Here is an example:

https://blog.apify.com/web-scraping-ai-missing-children/

When scraping Facebook pages and AI face recognitio n come to the rescue

🔖 5 social media scrapers and 5 use cases for each

Data itself is useless without a clear understanding of how to apply it. And the more data there is especially if its user-generated like in the case of social media platforms the more ways appear on how to turn it into a use case.

There's a reason why scraping social media remains one of the top use cases in the data extraction industry. Scraping comments from social media can be used for various purposes, offering valuable insights and opportunities for businesses and researchers.

So here are the five best social media scrapers to consider adding to your toolkit and 5 different use cases to which data can be applied. Let's use them to scrape the same post from NASA across five different social media platforms and see what comments we get.

1. Instagram Comment Scraper

Instagram is a treasure trove of user-generated content and engagement. Instagram Comment Scraper 🔗 allows you to gather information about posts, comments, likes, users, and more. Integrating this data into content platforms enables advanced audience segmentation, competitor analysis, campaign optimization and more.

You can use this tool to measure sentiment and analyze audience engagement. Even the simplest natural language processing techniques can help you with sentiment analysis. Categorizing comments as positive, negative, or neutral can give you a quantitative understanding of how your audience feels about the brand, products, or even topics, identify recurring themes, and assess the level of engagement.

📷 Follow along with our guide on how to scrape Instagram

Comments extracted from an Instagram post

💡Dataset with scraped Instagram comments here

2. TikTok Comments Scrape r

TikTok's rapid rise in popularity makes it an essential platform for looking into the needs and wants of a younger audience. TikTok Comments Scraper 🔗 enables you to collect data on trending hashtags, videos, user profiles, and engagement metrics.

This scraper can help you paint a larger picture of customer feedback. Comments often contain feedback and reviews about products or services. Scraping these comments can provide valuable insights into customer opinions, satisfaction levels, and pain points. This data can guide product improvements and customer service enhancements.

🎵 Follow along with our simple guide on how to scrape TikTok

Comments extracted from a TikTok video

💡Dataset with scraped TikTok comments

3. Facebook Comments Scraper

Facebook remains a crucial platform for connecting with a diverse audience. Facebook Comments Scraper 🔗 allows you to extract data from public pages, posts, and comments.

This tool can assist you with crisis management and content moderation. Monitoring comments during a crisis or PR incident can help you measure public sentiment and identify potential issues that need addressing. A prompt response to negative comments can help you manage and mitigate reputational damage in a time of crisis. Scraping comments can also help identify and flag inappropriate or spammy content for removal to ensure compliance with community guidelines.

📘 Follow along with our simple guide on how to scrape Facebook

Comments extracted from a Facebook post

💡Dataset with scraped Facebook comments

4. Reddit Comments Scraper

Reddit is a goldmine of niche communities and discussions. Reddit Scraper 🔗 lets you extract information from specific subreddits, posts, comments, and user profiles without using the official Reddit API.

Scraping Reddit can help you perform market research and product development. Comments on subreddits can reveal customer needs, preferences, and pain points. This data is invaluable for market research, allowing you to identify gaps in your specific market and develop products or services that address customer demands. Customer comments also often contain suggestions for product improvements or new features. Scraping and analyzing these suggestions can inform your product development roadmap.

🦾 Follow along with our simple guide on how to scrape Reddit

Comments extracted from a Reddit post

💡Dataset with scraped Reddit comments

5. YouTube Comments Scraper

YouTube remains a thriving entertainment platform and a large search engine. YouTube Comments Scraper 🔗 enables you to gather data on video views, likes, comments, and channel performance.

By extracting data from YouTube, you can find new trends and make decisions about collaboration opportunities. Comments can reflect emerging trends and topics within your industry or niche. By scraping comments from relevant videos, you can stay updated on what's currently popular, helping you customize your content and marketing strategies accordingly.

📹 Follow along with our simple guide on how to scrape YouTube

Comments extracted from a YouTube video

💡Dataset with scraped Youtube comments

Incorporating these social media scrapers into your marketing or research toolkit can provide you with a needed edge in understanding your audience, backing up your hunches for trends, and refining your strategies. However, always keep in mind that while scraping web data can provide valuable insights, it's essential to combine it with other data sources and your own expertise to make well-informed decisions.

💡 Other social media web scrapers

Well, if you thought 5 scrapers was a lot, you might want to explore our offer for every social media web scraping case. In Apify Store, you can find a tool to scrape Twitter, Telegram, Snapchat, Twitch, Pinterest, Quora, Onlyfans, Threads, Whatsapp, Mastodon, and even LinkedIn. Heck, we might even get a Clubhouse scraper someday go ahead if you want to build it and publish it. Is anything lacking on that list? Let us know on the Ideas page.

https://apify.com/store/categories/social-media-scrapers

Find a social media scraper for any use case

Top Google Search APIs

Natasha Lekh — Thu, 31 Aug 2023 22:00:00 +0000

Not all websites share their APIs with the public. Luckily for us, the biggest website in the world, Google Search, offers quite a few Google APIs. Or does it? Let's break down the most common misconceptions around Google Search API.

We're Apify . The Apify platform gives you access to 1,500+ data extraction tools and unofficial APIs for popular websites. Check us out .

Instead of going deep into what an API is, heres a short recap: APIs make the web open. The main point of an API is to connect different web programs with a link that they will use to communicate and exchange data. If the API is well-written, it enables two applications created with different technologies and languages to interact smoothly with each other without the need to resort to some sort of techno-compromise.

For various reasons, not all websites share their APIs with the public. Luckily for us, the biggest website in the world seems to be among the most generous, offering quite a few public APIs along with documentation on how to use them. Not without caveats of course, but well talk about that in due time.

Which API is used in Google?

Google offers many public APIs for different Google services; they are usually JSON APIs based on RESTful requests. These APIs are publicly available in Google API GitHub repository. As for a Google Search API, theres no such thing from Google. The Custom Search API provided by Google wont allow you to get a JSON from Google search results. What it does instead is perform a small version of Google Search on your own website.

Is there a Google Search API?

Not at the moment. Although the original Google Web Search API has been deprecated in 2011, you can still stumble upon its replacement, Custom Search API. However, this replacement API is different. While provided by Google as a stand-in for Google Search API, this API wont return you a detailed list of Google search results in JSON - which is what youd expect. Instead, it allows you to program your own little search engine by applying the logic of Google Search to your website.

In addition, this Google Search API imposes two key limitations: the number of queries and the number of websites you can search. It supports a limited number of queries per day (10K). It may also restrict search capabilities which means that you cant use it to search the whole web programmatically. Whichever limitation you choose, it will cost you $5/1K queries to search websites programmatically via the official Google API. More on the pricing in the next part.

While searching for the Google Search API, you may also come across the Google Search Console API, which is also not exactly what you might be looking for since it's geared specifically towards the GSC tool. It will allow you to interact with and control various aspects of Google Search Console programmatically but that's closer to a scraping session customized for your website than an objective look at what Google has to offer for a given query.

Is Google's API free?

Google has many APIs and all of them are free but with many limitations. The Custom Search API, for example, offers the first 100 search queries per day for free. If you wish to lift this limitation and make more queries, it will cost you $5 per 1,000 requests but still limits you to 10,000 queries per day. Theres no way to exceed those 10,000 requests per day even if youre ready to pay more. The only way to go above the 10K limitation is if you are applying your search engine to less than 10 websites.

How do I create a search API?

To create a search API of your own, you can follow Googles step-by-step tutorial on creating a Custom Search Engine which is freely available on the Google developers page. In short, regardless of the programming language, you will need to get an API Key, a Custom Search ID, and install a relevant API Client to get your first results. If you follow through with all the steps, youll be able to make your own search engine, customize your search engine's ranking, and do other surface-level modifications.

🎓More of a Bing scraping fan? See this tutorial

How do I search Google using an API?

Although Google does offer some sort of Search API, its impossible to use that one to send a search query to the whole wide web and get a machine-readable doc from this search. Which makes an official Google Search API, in the way youd usually understand it, non-existent. However, many developers have noticed it and have come up with alternatives. By the nature of their task, all those Google Search API alternatives are essentially scrapers, heres just a few of them:

Top Google Search API alternatives:

1. SERP API

Offers some web scraping infrastructure, regular organic results, structured data in JSON, reliable location search via encrypted parameters, solving all CAPTCHAs, guaranteed 20% throughput, and no request queues. Free plan can get you 100 searches/month. Pricing starts from $50/month per 5,000 searches/month and offers a so-called Legal US Shield.

2. Apify 's SERP API

Provides SERP proxies, reliable scraping infra and monitoring, freedom in request location and language, organic results, ads, prices, reviews, related queries, wide proxy network, datasets of 5 formats (JSON, Excel, CSV, XML, RSS), and offers Google APIs for other Google Services. Free trial for 30 days, then $49/month for a platform subscription plan. Try it out for free right now!

3. Serpstack

The free plan promises no request queue, customized search queries, proxy network, data in JSON format, SERP proxies, and up to 1,000 searches monthly. Able to scrape the web, images, videos, news and shopping data. Pricing starts at $29.99/month for expanded features of API premium.

4. Rapid API

One available API for Google Search suggests speed and accuracy, easy-to-use but advanced enough to support special parameters, with zero proxies but clean IPs from the US instead, as well as some monitoring. Free plan is limited to 50 requests/month and 1 request/second, paid plan starts at $15/monthly and 5,000 requests/month. Maximum capacity plan offers 50,000 requests per month.

Another one called Google Web Search offers Free plan with 100 requests/month and 5 requests/second. Paid plan starts at $30/monthly and 10,000 requests/month.

5. ScrapingBee

Scrapes organic results, ads, local results, and related queries using SERP proxies. Offers extensive documentation. Monthly subscription plan starts at $49 with 6,000 searches/month and 5 concurrent requests. Proxies availability depend on the plan.

So Google Search API is powered by web scraping?

Essentially, yes. That's why you can confidently call Google the world's biggest web scraping company. As Google stopped providing a proper Search API allowing to get machine-readable data, other solutions such as scrapers inevitably turned up. They now remain the only proper solution to extract data from Google via an API.

How does a Google SERP API work?

The Google SERP bot takes your query, [e.g. weather tomorrow], takes it to google.com, performs the search and extracts the raw Google data from the results pages. You can customize the language [e.g. English or Spanish], geolocation of the results [e.g. US or Mexico], number of scraped results pages [e.g. first 10] and many other parameters.

Watch this simple video explaining how a Google Search API aka Google SERP Scraper retrieves data from the search engine. It will only take you 3 minutes to learn how to extract the data you need from Google Search!

How to scrape LinkedIn profiles and companies

Natasha Lekh — Tue, 29 Aug 2023 22:00:00 +0000

Learn how easy it is to scrape LinkedIn for company and individual page URLs. Find out how LinkedIn scrapers can help your business and discover how you can use them in 5 simple steps.

With 950 million members, LinkedIn is a worldwide online platform that allows users to create professional profiles. Both individuals and companies use it to store resumes, connect to colleagues, and promote their brands. Individual users can search and apply for companies and jobs, while companies can headhunt and advertise open positions.

📩 Phone, Email and Contact Details Scraper · Apify

Provide a list of websites and download their emails, phone numbers, and social media details. Export results in Excel, CSV, JSON or with an API.

apify.com

Want to extract contact details from a different website? Try this tool instead.

LinkedIn also acts as a social media platform for its users, where they can share content and discuss their areas of expertise. This makes LinkedIn one of the largest databases of potential employers and employees and a rich source of public information. LinkedIn web scraping is one of the most effective ways to access this data.

Apify Store has two useful LinkedIn URL scrapers: one dedicated to finding companies and one for finding personal profiles. These scrapers make it easier for you to access and use that public LinkedIn data.

LinkedIn Company URL Finder scraping tool extracts LinkedIn company page URLs and gives them to you in a neat, organized list, ready to download as structured data. Just feed it a company name or a list of companies you want to find on LinkedIn, and you get a list of URLs back. Its quick, its simple, and its cheap.

LinkedIn People Finder does the same thing, but for personal profiles. You give the web scraper a name or a list of names, and it extracts the URLs of their LinkedIn profiles.

👮 Is it legal to scrape LinkedIn?

Web scraping is legal. This scraper is extracting publicly available URLs from a simple Google search. If you want to learn more about web scraping and its legal implications, you can find more information in this article.

📇 Why scrape LinkedIn URLs?

Having a list of LinkedIn URLs for companies or people that you are interested in can be useful in lots of ways. Here are our 4 favorites:

1. Job hunting

Youre hunting for the perfect job, and you want all the URLs of the companies you would like to work for at your fingertips. If you scrape a list of URLs, you wont have to search for the company pages every time you look for open positions. You can even get really organized and use the list in a spreadsheet or database. You can rapidly access their pages anytime and quickly apply for a position through their LinkedIn page.

2. Recruiting and headhunting

Over time, LinkedIn has become a go-to place for recruitment across the globe. If a list of candidates applied for a position at your company, you'll want to have their resumes easily available. With LinkedIn People Finder, you can scrape the profile URLs so you wont have to singularly look for them every time you need to consult them. Or maybe you want to go hunting for that perfect candidate, even if they're not currently looking for you.

3. Client database

If you're building a database of potential clients to contact, LinkedIn is a great networking tool for this purpose. It will definitely make it easier if you have all the LinkedIn page URLs in a list for your marketing or sales team, because looking for LinkedIn company pages manually can be really time-consuming. This scraper will give your sales team quick and easy access to the contacts so that they can spend more time chasing leads and making deals.

4. Market research

You want to stay up to date on what your competitors are up to and track their recruiting strategies. A list of all their URLs will make your market research much easier. You will be able to easily connect to their LinkedIn feeds and analyze their branding and the content they are sharing.

Scraping job listings data for a competitive edge

Read about why and how data-driven companies are using web scraping and RPA tools for collecting job information.

blog.apify.com

A short tutorial on how companies try to automate job listings data extraction

📌 How to scrape LinkedIn company pages

Lets start with a quick guide on how to use LinkedIn Company URL Finder. With these few easy steps, you will be able to extract all the company page URLs you want!

Step 1. Find LinkedIn Company URL Finder

Go to the LinkedIn Company URL Finder page on Apify Store and click the Try for free button.

If youre not signed in, youll find yourself on the sign-up page. Sign up using your email account, Google, or GitHub.

You will be redirected to the scrapers page on Apify Console. Apify Console is your workspace for running tasks for your scrapers.

Step 2. Type in or copypaste company names

LinkedIn Company URL Finder only requires two input fields:

Company names refer to the names of the companies of which you want to extract the LinkedIn page URL. You can type or copy and paste a list of company names here. Make sure that you only have one company name per line.
Speed refers to the number of bots you want to start in parallel. The higher the number, the faster the search. However, a higher number also means you will burn through your available proxies quickly.

Step 3. Click Start to begin scraping

Once youre all set, click the Start button. Notice that your task will change its status to Running, so wait for the scraper's run to finish. It will be just a minute before you see the status switch to Succeeded.

Step 4. View scraped data

The search will create as many results as the number of companies you listed in the input phase. Move to the Storage tab containing your scraped data in many formats, including JSON, CSV, Excel, XML, and RSS feed. You can preview the data by clicking the preview button or viewing it in a new tab if the dataset is too large.

Step 5. Download scraped LinkedIn company data

As you have noticed, the search will create as many results as the number of companies you listed in the input phase in our case, three of them. You can preview the data by clicking the preview button or viewing it in a new tab if the dataset is too large.

How to scrape Indeed jobs and company profiles

Step-by-step guide to automated data collection from Indeed.

blog.apify.com

See this similar tutorial for scraping Indeed

👩🔧 How to scrape LinkedIn for personal profile pages

And now for LinkedIn People Finder. You will notice most of the steps are very similar for the two scrapers.

Step 1. Find LinkedIn People Finder

Go to the LinkedIn People Finder page on Apify Store and click the Try for free button.

The LinkedIn People Finder page on Apify Store.

If youre not signed in, youll find yourself on the sign-up page. Sign up using your email account, Google, or GitHub.

The sign-up/log-in page on Apify Store.

You will be redirected to the scrapers page on your Apify Console. Apify Console is your workspace for running tasks for your scrapers. You can now click the Start your free trial button and confirm that you want to start using LinkedIn People Finder.

Step 2. Type in or copypaste profile names

LinkedIn People Finder only requires you to fill in the people names field, i.e. the names of the people for which you want to extract the LinkedIn page URL. You can type or copy and paste a list of peoples names here. Make sure that you only have one name per line.

If you want to restrict the search field for the scraper, in the next field, you can select a language for the profile. The scraper will only show you the results available in that language.

Step 3. Click Start to begin scraping

Step 4. View scraped data

Move to the Storage tab to see the results of your scraping. The search will create as many results as the number of people you listed in the input phase. In our case, it's two. The Storage tab contains your scraped data in many formats, including JSON, CSV, Excel, XML, and RSS feed.

Step 5. Download LinkedIn profile data

Preview the data by clicking the preview button or viewing it in a new tab if the dataset is too large. You can pre-select certain fields before downloading the dataset. You can now upload it onto your computer for further use as spreadsheets or in other apps and your projects.

If you found LinkedIn Company URL Finder and LinkedIn People Finder useful, you might also like to try out Email & Phone Extractor. It will extract emails, phone numbers, and other useful contact information from any list of websites you provide. Great for lead generation!

10 reasons tourists hate European landmarks (according to data from Google Maps)

Natasha Lekh — Mon, 31 Jul 2023 22:00:00 +0000

Ever wanted to see an over-glorified pile of rocks or as it is more commonly known, Stonehenge? And how about an overrated pile of metal, the Eiffel Tower - is it on your bucket list? Or my recent favorite, the Leaning Tower of Pisa, which is sadly not made of pizza. 1 star. - you can find some true comedy gold on the pages of Google Maps.

If youve ever come across an unimpressed comment about your favorite place, book, or movie, you are well aware that online reviewers can be merciless. Tourists on Google Maps - even more so. Youd think popular tourist landmarks would be immune to dramatically bad reviews, with dissatisfaction falling somewhere in the meh category. After all, those are the most monumental landmarks of the country, surrounded by history and meaning. So they are deemed to be a tourist attraction for a reason and deserve some slack. Youd be surprised to discover how strong some opinions about the Colosseum and Louvre are. So why are people unhappy with their experiences?

Lets do something different today and use a simple AI text analysis tool to sift through the most recent worst reviews of top-5 European landmarks. But unlike those reviewers, our sentiment analysis tool will help us in being objective with our little data project. With extracted data on our side and a few hefty tools to work with it, well be able to collect reviews, organize them by sentiment, visualize, analyze it by common words, and ultimately find out the truth. All within the same platform. And dont mistake me for a data analyst Im just a writer who likes data!

How will we do sentiment analysis for reviews?

First, well collect the reviews from places on Google Maps. We wont be doing that by sorting and copying them into a doc, well simply use a web scraping tool to get all of them automatically and create a dataset suitable for an AI.

Next, well use an AI text analyzer from Geneea to go through our reviews and identify the most commonly used keywords in each of them. This text analysis tool will make our dataset perfectly suitable for any data manipulations such as visualizing it.

This is our final point: well use a data viz tool to see our reviews in a word cloud and help us gain an understanding of different aspects of bad sentiment around all these places.

🌹 How to use a text analyzer tool for sentiment analysis in Google reviews

Step 1. Choose landmarks to analyze

Lets head over to Google Maps first and pick some landmarks. Lets go for the most touristy spots that would easily make it onto any European skyline postcard. Off the top of my head: Eiffel Tower, the Leaning Tower of Pisa, the Colosseum, the Sistine Chapel, and of course, the over-glorified pile of rocks itself, Stonehenge.

Usually, you would need a Google Maps API to extract reviews. Luckily, there are cheaper and more convenient ways to collect data such as Google Maps Scraper 🔗 or Google Maps Reviews Scraper 🔗.

There are also many ways to scrape Google Maps data but well be going with the one that fits us most in this case: by providing a Google place URLs. Well simply copy-paste the URL of each place that interests us.

Step 2. Extract Google reviews

Now that weve chosen the places to scrape, its time to configure the reviews part. Well scroll down to configure reviews data extraction. As for our requirements for reviews, they have to be:

bad 👎
recent 🍃
place-bound 📍
well-organized 🗂

All of this is possible to set up with the scraper. We will choose to scrape 100 reviews per place (so, 500 reviews in total), then make sure we only get the ones posted from 2020 and onwards, and finally, prefer that the worst ones (lowest-ranking) come first. Last but not least, to make sure our resulting dataset will be well-organized and easy to process well use the One review per row toggle.

In just one minute, we have all 500 very candid reviews in one neat dataset, starting with the first batch of good-old Stonehenge bashing.

Step 3. Configure the AI text analyzer

Once weve extracted the reviews, you can move on to the AI text analyzer tool 🔗 itself. In the analyzer's input, we need to provide the dataset ID of the scraped results, so well just copy-paste it from the previous tab. Additionally, we have the option to modify the Industry field to get results tailored to specific industries such as General, Banking, or Hospitality. Well go with General.

Well also instruct our text analyzer to process all of our 500 reviews and skip reviews with no text.

Step 4. Look at the analyzed results

In a few minutes, well see the results of basic sentiment analysis. The resulting dataset includes the original data that weve seen already along with a new NLP field called text analysis which contains valuable attributes such as sentiment and tags.

Step 4. Have fun looking at the analyzed results

Step 5. Visualize the results

To make the most of the scraped data, you can visualize it using your preferred visualization tool. The resulting dataset can be seamlessly integrated into various visualization platforms to create informative dashboards and reports. So let's take our data a step further and see what weve got using a visualization tool:

Sentiment analysis visualized in word clouds: 500 bad reviews of 5 landmarks

Step 6. Analyze the results

Analyzing all 500 of our reviews shows that some bad experiences are universal for all 5 places: rude staff, long lines, and tourist traps can turn any trip into a nightmare, or at least comedy gold material. Among the top 1o are also: expensive or extortionate prices, poor management, racist staff, crowdedness, grim weather, poor security, and of course, complete scams. The analysis also shows that even bad reviews contain some positive and neutral characteristics shown in green and white colors; Google Maps reviewers are not strangers to nuance.

Note that the text analysis tool we've used leaned heavily on the power of adjectives and the frequency of mentioning specific aspects of the places: just something to keep in mind when analyzing Google Places reviews - or writing one.

📸 The value of analyzing online reviews for sentiment

So is it worth going to see the Tower of Pisa? Or waiting in queues to see Paris from a birds view? We dont know. This review analyzer only checked for bad reviews 🤷🏻. To be fair to a popular tourist attraction, wed need to analyze the whole scope of the reviews: the good, the bad, and the meh ones.

Online reviews play a crucial role in shaping consumer decisions and can make or break a business. Of course, European landmarks will not suffer from a few sour opinions and the stream of tourists will continue. However, analyzing these reviews can provide valuable insights for businesses looking to understand customer sentiments and improve their offerings.

Whether you're interested in monitoring your or your competitor's reviews or simply doing a small data project (like yours truly here), the combination of real-time reviews from the web and sentiment analysis tool is the way you turn opinions into data. If you're looking for a powerful text analysis tool, the Google Maps Scraper in conjunction with AI Text Analyzer 🔗 can be an excellent option for you.

You can use this AI analyzer for text not only to analyze reviews of tourist attractions and public spaces. You can choose to analyze restaurants, banks, shops, hospitals, galleries, and whatever else there is on Google Maps that people care to leave a review about. By extracting, translating, and doing even the simplest analysis of reviews, you can understand visitor sentiments better, identify patterns faster, and make your business stronger.

🌠 Want to know more about how to scrape Google Maps? Check out this comprehensive guide .

FAQ

Can I use this AI text analyzer with Python?

Yes, you can use this text analyzer with Python via Apify API. The Apify API is organized around RESTful HTTP endpoints that enable you to manage, schedule, and run any tool on the platform, as well as access any datasets. To access the API using Python, use the apify-client PyPi package.

Can I use this tool for sentiment analysis on Twitter?

No, this tool is specifically designed for sentiment analysis for Google Maps reviews. However, Geneea offers other tools that might be suitable for your needs. If you scrape tweets, profiles, or a specific topic using a Twitter scraper, you can continue the sentiment analysis using Geneea's tools.

Alternatively, there are other social media sentiment analysis options available as well. Take a look at other social media scrapers to scrape comments and use them together with Geneea's NLP tools for text analysis and visualization.

What other tools for text analysis do you have?

Apart from the text analyzer tool, we also offer an article download tool. Additionally, Geneea's visualization tool, Frida, can be applied for sentiment analysis, as demonstrated in this journalistic work where this tool was used for visualizing the results of sentiment analysis in the context of Zelenskyi vs. Putin in Czech media.

Google Maps scraping manual: how to extract reviews, images, restaurants, and more 📍 📚

Natasha Lekh — Tue, 25 Jul 2023 22:00:00 +0000

We're Apify . You can build, deploy, share, and monitor any scrapers on the Apify platform. Check us out .

Welcome to your comprehensive guide to extracting valuable data from Google Maps. In this manual, we will walk you through various techniques and tools to help you scrape images, extract restaurant data, gather contact details, scrape reviews, and much more. Whether you're a researcher, developer, or simply someone looking to make good use of publicly available Google Maps data, this manual is designed to assist you in achieving your data-related goals.

What is this manual for?

📍 Google Maps Scraper was one of the first Apify scrapers developed. As the Google Maps website underwent changes and evolved over time, our scraper adapted to reflect those changes and make sure it extracted high-quality Google Maps data.

Over the years, our scraper has gained a fan base of over 45,000 users, along with numerous new features. Many of these features may not be immediately apparent due to the complexity of the scraper. So we've created this manual to help you out with specific Google Maps scraping cases. We do, however, have other general tutorials on how to use the scraper:

How to scrape data from Google Maps: general tutorial on scraping Google Maps
How to scrape Google Maps by URLs, coordinates or other geolocation parameters
How to extract emails, social profiles, phone numbers and addresses from Google Maps
How to extract gas prices from Google Maps
Google Maps webinar with a demo of using Google Maps Crawler

Feel free to explore them. If you notice that the manual might be lacking a case or that the guide instructions don't work for you, please let us know.

Restaurants

👩🍳 How to extract restaurant data from Google Maps?

By using 📍 Google Maps Scraper. All you have to do is put restaurants into the Search term field, City name in the Location field, and Number of results. The resulting dataset will contain the scraped restaurants from the city you've indicated.

Put restaurants into Search term field, City name in the Location field, and Number of results. Then scrape all restaurants from the city you've indicated.

🍟 How to scrape restaurant chains?

📍 Google Maps Scraper allows you to easily scrape not only restaurants in general but also restaurant chains such as McDonalds or Starbucks. All you have to do is use an input for scraping places that are called the same way. First of all, you should put the name of the chain in Search terms, and add Location and Number of results.

Then scroll down to 🔍 Search options and choose to scrape only places that include the search term in their title. Click Start. After the scrape, you should get a result with all franchise restaurants in one dataset.

To scrape a franchise from Google Maps, indicate its name and location. Then make a change in the Search options.

🥘 How to scrape restaurant menus from Google Maps?

You can use Google Maps scraper to scrape cafes, restaurants, pubs, and any other gastronomic spots. Theres no need to specify menu scraping in input, you can scrape restaurants as usual any amount and anywhere you want. The output dataset will contain the links to menus if they are indicated on Google Maps.

You can find the menu URL in the scraped dataset in restaurants include it on their Google Maps card

Contact Details

☎️ How to extract phone numbers from Google Maps?

Use Google Maps data extraction tool. No matter what, where, and how you will decide to scrape using this tool (by URL, search term+location or coordinates), it will make sure you always get phone numbers and addresses of the places you were targeting. As long as these listings mention their contact details on Google Maps website, you will be able to extract them by default.

Here's an example of extracting phone numbers from 100 places that sell bagels in NYC:

Example of extracted phone numbers from 100 places that sell bagels in NYC

To make sure you get all listed phone numbers from business listings in Google Maps (not only the default ones), you might want to run the Google Maps Email Extractor on top. It will visit the websites of these listings and get all the phone numbers mentioned there.

Enhanced dataset containing all phone numbers from Google business listings

📍How to scrape Google Maps for addresses?

Use Google Maps data extraction tool. No matter what method, location, or scraping technique you choose (by URL, search term or coordinates), this tool will make sure you always extract the addresses of the places you're targeting. As long as those listings have their contact info listed on the Google Maps website, you'll be able to extract them without any hassle.

Example of location and address scraped from Google Maps: one place vs. 200 places

If you want to enhance the addresses from business listings on Google Maps with their social media details, you should add the Google Maps Email Extractor on top. It goes the extra mile by visiting the websites linked to those listings and snatching up all the socials mentioned there: Facebook, Twitter, Instagram, etc. It's a great way to create a relevant database full of contact details.

Example of social media details scraped from listings on Google Maps

📩 How to extract emails from Google Maps?

Your best bet is to use the Google Maps Scraper and then Google Maps Email Extractor. The reason for using two tools is, you cannot scrape business emails from Google Maps directly. You have to visit every place on Google Maps separately, see if they indicate their website address, and then visit these websites one by one to check for email in Contact Us section. The process of visiting every place is covered by the Maps Scraper and the process of web scraping every website of that place by the Email Extractor.

📮 You can read how we've extracted emails from Google Maps businesses along with their social media profiles in this tutorial .

Extracted contact details from restaurants on Google Maps

⬇️ Can I download emails of Google Maps business listings in Excel?

Yes. After extracting contact data from Google Maps you can also filter and download it in Excel. In order to do that, pick relevant categories in Selected fields (such as contactDetails, address, phone, etc.) and the file format you prefer.

An example of filtering out contact details on Google Maps and downloading them in Excel

Reviews

🌟 How to scrape Google reviews?

The easiest way is to use the reviews option in📍 Google Maps Scraper. First pick places to scrape either by URL or search term. Then you can set up everything around reviews, starting from review freshness, URL, and number of them per place to the reviewer's details. You can also scrape only reviews containing specific keywords and sort them by relevance, ranking, or newness.

As a result, you'll get a dataset with Google reviews and all information surrounding them: review text, the response from the owner, number of upvotes, posting time, how often reviewer leaves reviews, and whether the review is marked as a Local Guide on Google Maps. For our example, we've decided to scrape Google restaurant reviews.

Fill out the Reviews part of the input to get Google review data

⬇️ Can I download only the reviews from Google Maps in Excel?

Yes. If youve extracted all kinds of Google Maps data, and you dont want to clean the dataset for reviews, you can do it directly on the platform. To preview the reviews before downloading them just pick the Reviews tab in Output. Now to only download Google reviews, all you have to do is choose Reviews in Export tab and Excel format.

Pick the Reviews tab in Output. Choose to download Google Reviews in Export tab.

Visualizations

🗺 Can I visualize data scraped from Google Maps?

Yes. You can view your scraped data on a map as you're extracting it and export it later. Visualization of your data on OpenStreetMap can always be found in Live View tab. After the run has finished, you'll find your map safe and sound in Key-Value Store as results-map.html file for further use.

Visualization of data extracted from Google Maps

The data you extract from Google Maps with📍 Google Maps Scraper is highly adaptable. You can export it in JSON, CSV, HTML, and XLS and plug it into any visualization tool or platform of your choice. Here's an example of how we used scraped gas prices with kepler.pl

Kepler.gl embedded map

natashalekh.github.io

Big ambitions

🌆 Can I scrape an entire city on Google Maps? I want to extract all places.

Yes. You can scrape all the Google places from the city by using special parameters on 📍 Google Maps Scraper. Bear in mind, it's going to take some time. However, all you need to provide is the name of the city and a big number. Then scroll down to choose the Scraping places without search terms -> Scrape by OCR tool in the dropdown list.

Example of input of how to scrape all places from a city on Google Maps

Here's an example of web scraping all places on Google Maps from Albuquerque, New Mexico. As you can see from the dataset, we have gotten a wide range of places that include but are not limited to sculptures, mailing services, airlines, and restaurants. We've got 459 places in total and although it took us over an hour and much of our memory, scraping an entire city cost us only $2.50 in credits. This means you can scrape an entire city basically for free within the Free plan.

Example of scraping all places from an entire city

🌐 Can I scrape an entire country on Google Maps?

Yes. All you have to provide are the search terms and the name of the country in Google Maps Scraper tool. Set the maximum number as high as possible and you're all set. Here's an example of scraping all hospitals across Switzerland.

Example of scraping all hospitals across Switzerland

It took us almost an hour but we've got all the hospital facilities in Switzerland, which ended up being 166. We have their general info, contact info, location, rating, and if we want to we can add reviews as well. Crawling Google Maps and extracting that data across the entire albeit small country cost us only $1.70 credits.

Example of extracting Google Maps data across the territory of an entire country

🏙 Can I scrape only cities and skip unpopulated areas on Google Maps?

Yes. You can extract more places from Google Maps with the Deeper region scrape option in Google Maps Scraper. If you're extracting maps data from countries like Australia or Canada, or just regions with vast areas of low population, this feature will be essential for uncovering every single Google Maps place.

With this setting, the scraper will skip unpopulated areas; this includes deserts, forests, mountains, lakes. The scraper will instead focus on the places that actually include Google Maps pins - which are usually cities. The areas will be divided into cities and zoomed in enough, ensuring you don't miss a single pin on Google Maps.

Setting for city-focused scraping on Google Maps

Note: enabling this feature may increase runtime, as more places will be extracted. But trust us, it's worth it since you'll be able to efficiently scrape huge areas like states or even an entire continent!

Sentiment analysis

🌹 Can I use this tool to do sentiment analysis of Google Maps reviews?

You can combine the capabilities of Google Maps Reviews Scraper and 🤖 AI Text Analyzer to do full-text analytics of Google reviews. You can scrape any type of reviews you'd like: Google restaurant reviews, museum reviews, business reviews, etc., ratings, and replies from business owners. However, for Google sentiment analysis, it's not enough just to just scrape the reviews. The second Actor is a text analysis tool built to go through the whole dataset and identify the most important attributes of each review. That way the dataset will be ready to be further processed by any sentiment analysis tools of your choice.

🏰 You can read how we've analyzed hundreds of negative Google reviews using the AI text analysis tool mentioned above.

Example of extracted Google reviews that are pre-processed for sentiment analysis

Geolocation

📡 How to extract longitude and latitude from Google Maps places?

It's enough to scrape a place on Google Maps using📍Google Maps Scraper. Usually, the Google Maps URLs already include the coordinates of any chosen place. All we have to do is to extract them.

An example of Google Maps coordinates already seen in the URL

Head over to Google Maps URL 🔗 section and copy-paste the URL of Google place that interests you. You can add as many places as you need coordinates from. In the dataset, you'll see both longitude and latitude of the place.

Extracted longitude and latitude of the Google Maps

The scraper will extract geolocation data even if you use the first section with Search, Location, and number of results. The option with URLs is just more specific to easier to demonstrate the scraping results on.

Specific Google Maps data

📇 How to get Google place id?

Every place on Google Maps has a placeid tied to it, a unique identifier in the Google Maps database. Place IDs are a part of the URL and are really useful for organizing your Google Maps data. Knowing just the ID, you can retrieve the whole information about the place from Google Maps. You can extract placeid using 📍 Google Maps Scraper. No matter which way you will chose to scrape the places (by URL, geolocation or search term), the dataset will always include a placeid. Here's an example of placeid after scraping Vasco da Gama Bridge.

Where to find a place id in the Google Maps dataset

If you have a dataset with many places scraped and you want to keep only a few fields such as place name and place ID, simply preselect the fields you prefer before downloading them.

Example of sorting the dataset that contains 200 scraped places to get only their titles and place ids

Google Maps Images

📸 How to scrape Google Maps images and photos?

To scrape images from Google Maps places, you will need to use two Actors: 📍 Google Maps Scraper to extract the images and ⬇️Download Images from Dataset Actor to download them in bulk. You need the second Actor because the scraper will extract not the image files but only their URLs. Since you wouldnt want to open each image URL and Save the image as a file, the ⬇️Download Images from Dataset is here to do it for you. Alternatively, you can build a Google Images Scraper yourself.

In short, this is your action plan:

Extract data from Google places to make a dataset
Open the 📍 Google Maps Scraper.
Configure the scraper to extract data from the places, including the images.
Run the scraper and find the dataset containing extracted Google Maps data.
Note the dataset ID for future reference.

2. Find the image attribute

Image attribute of Google Maps

Preview the resulting dataset in JSON.
Use Ctrl+F or Cmd+F to find a specific attribute with image URLs. Typically for Google Maps data, this field is called imageUrls

3. Move on to image download

Open Download Images from Dataset and copy the dataset ID and image field

Open Download Images from Dataset Actor.
Copy the dataset ID from the previous steps and paste it into the respective field.
Paste the image field (e.g. imageUrls) into the other field.
Specify the number of images you want to download.
Click Start.

Open Download Images from Dataset. Paste dataset ID and imageUrlsinto the fields, choose number of images, and click Start to download.

4.⬇️ Download the images

Click on result to download a zip folder with all the images from Google Maps

Click on Zip File URL to start downloading the images from Google Maps.

If in doubt, follow the steps in the ⬇️Download Images from Dataset readme.

🌟 How to scrape images from Google Maps reviews?

Scraping images from Google Maps reviews can provide valuable insights and visual content for your projects. In this part of the manual, we will guide you through the process of scraping images from Google Maps reviews using the necessary tools and steps. Here's your action plan to get started.

Get the required tools:

📍 Google Maps Scraper: this tool will help you extract the reviews and relevant information from Google Maps.
⬇️ Download Images from Dataset Actor: this Actor will assist you in downloading the images in bulk from the dataset.

Extract reviews and create a dataset

Open the 📍 Google Maps Scraper tool.
Set up the scraper to extract the places, their reviews, and images.
Run the scraper and open the dataset to check for images and reviews.
Note the dataset ID for future reference.

Set up Google Maps Scraper to scrape reviews and their images

2. Identify the image field

Image attribute of Google Maps reviews

Open the resulting dataset containing the extracted reviews.
Find the specific attribute that holds the image URLs of reviews. Typically for Google Maps reviews, this field is named reviewImageUrls

3. Move on to image download

Copy the dataset ID containing reviews and review image attribute

Open the Download Images from Dataset.
Copy the dataset ID from the previous steps and paste it into the respective field.
Paste the image field (e.g., reviewImageUrls) into the appropriate field.
Specify the number of images you want to download (if needed).
Click Start to initiate the image download process.

4. Download the images

Click on result to download a zip folder with all the images from Google Maps reviews

After the download process completes, click on the Zip File URL.
This will download a zip file containing all the images from the Google Maps reviews.

If you encounter any issues with the part of downloading images in bulk, check the visual instructions in the Actor's readme.

That's it - you're ready to scrape Google Maps!

We hope this guide has equipped you with the knowledge and tools necessary to effectively scrape, extract, and analyze data from Google Maps. Whether you're interested in scraping images, extracting contact details, or delving into specific Google Maps data, you now have the know-how to embark on your own scraping journey.

Remember, with each data extraction, it's important to ensure compliance with Google's terms of service and respect the privacy and rights of others. As technology evolves, so do the possibilities offered by Google Maps. Stay curious, explore new features, and continue to leverage the wealth of information available through this powerful platform.

🛸If you still want more Google Maps scraping tips, how about finding out how to scrape Area 51 on Google Maps?

How to scrape Indeed jobs and company profiles

Natasha Lekh — Mon, 24 Jul 2023 22:00:00 +0000

Indeed's APIs don't cover the job listings available on the site. The good news is that because of its clear categories and customizable parameters, the Indeed website is perfect for web scraping.

Indeed helps people find their dream jobs. Or at least the ones that meet their standards. Indeed has a few of its own APIs, but they don't cover the job listings available on the site. The good news is that because of its clear categories and customizable parameters, the Indeed website is perfect for web scraping. So, let's give scraping Indeed a try using a specialized scraping tool.

Can I scrape jobs using Indeed API?

Not really. There are several Indeed APIs available. In the past, Indeed used to have an API group designed to scrape Indeed job listings, it was called Publisher Jobs API (Get Job API and Job Search API). However, these APIs have been deprecated. The Job Search API allowed users to access data like job titles, company names, locations, posting times, and job descriptions. Due to these changes, users have been looking for alternatives to Indeed APIs, such as this Indeed web scraper 🔗

Can I scrape applicant info using Indeed API?

To an extent. The available Indeed APIs are useful for tasks like integrating Indeed data into applicant tracking systems, monitoring applicant conversions, or managing interview schedules. They mainly cater to the hiring audience of the platform. Unfortunately, they have a number of limitations and are not suitable for job searching purposes.

Sadly, limitations on website APIs have become very common these days. Websites with huge user bases rarely leave their APIs available and open, even if they did in the past. This is where scraping comes in. Because of the rising trend in API limits, over time we've built hundreds of ready-to-use tools for scraping and automation to keep the public data available for automated extraction. If youre interested in keeping track of job listings on Indeed, you can try using Indeed Scraper 🔗 to automate the process of collecting and extracting the data.

🗂 How does Indeed scraper work?

Indeed Scraper 🔗 works in a way similar to manually searching through the Indeed website and extracting information from each page. But it's way faster!

It starts by going over to the Indeed website and navigating to the specified location. Then, it enters your search query into the search bar and proceeds to scroll down until it reaches the final page. The scraper queues up all the job listings it finds and efficiently copies and organizes all the visible data into a well-structured document. All you have to do is download it.

🖇 How can I use the data extracted from Indeed?

For job seekers	For recruiters
🎖 Extract job listings by titles, benefits, skills, certifications	👥 Collect candidate data and automate the applicant management process
⏱ Extract Indeed job openings with relevant positions the moment they appear	💰 Compare salaries in the industry to determine fair market rates for similar roles
🏡 Find job positions with specific requirements (part-time, WFH, no experience, etc.)	🔍 Research to support investment decisions for future office space needs
💶 Get salary estimates on job positions posted on Indeed	📊 Analyze competitors, track hiring trends for skills and technology gaps
📇 Analyze hiring trends, forecast skills, and job role demands	😊 Assess employee sentiment

📇 Step-by-step guide to Indeed scraping

Step 1. Find Indeed Scraper

First, go to Indeed Scraper's page, and click the Try for free button.

Step 1. Find Indeed Scraper

If you're not signed in to the Apify platform, clicking the button will take you to the signup page. You can sign up using your email account, Google, or GitHub and try the Indeed scraper for free.

Sign in to the Apify platform. Don't worry, we don't bite 😁

Step 2. Find Indeed jobs to scrape

You can now access the scrapers page in Apify Console. This is where you can set up, run, and download data from scraping. Now let's fill in the required input fields, mainly:

Positions: e.g. let's look for data analyst jobs on Indeed
Country: for our example, were going with the US
Location: were using San Francisco
Max items: let's get 100 job listings.

We will also use the Scrape company details toggle to get the hiring companies' info such as name, rating, number of reviews .

Step 2. Find Indeed jobs to scrape

Step 3. Start scraping Indeed

Once you are all set, click the Start button. You will notice that your task will change its status to Running 🏃. It will be just a minute before you see the status switch to Succeeded✅

Step 3. Start scraping Indeed

Step 4. Download Indeed job listings

When the status changes to Succeeded ✅, you can check the results in the Storage tab.

We've managed to scrape 100 data analysis job postings together with information on the companies that offer these positions. Now we can preview and download this data as Excel, HTML table, JSON, CSV, or XML. We can also share the scraped Indeed data directly via an API.

Step 4. Download Indeed job listings

Step 5. Bonus: set up job scraping Indeed every hour, day, week, or month

You can set up the whole scraping job to run automatically every day. First, you have to save your scraping parameters (specific location, job position, number of listings) as a task. This step will make sure the scraper always has the right input which you now don't have to type in yourself. Then click Actions -> Schedule.

Now set up the timing conditions for your scraping task: how frequently it should scrape Indeed, at what time, and in which time zone. All you have left to do is click Create. Last but not least, you can also set up a notification to get an email every time your scheduled Indeed scraping has been completed.

Step 5. Set up job scraping every hour, day, week, or month

❓FAQ

Can I scrape data from LinkedIn?

Yes. If you dont want to scrape Indeed job postings and rather get data from another job-hunting platform, you can use some of the other available job listing scrapers. You can scrape reviews on Glassdoor, scrape data from LinkedIn, or freelancers offers on Upwork or Fiverr. Just browse our 30+ available job listing scrapers.

🔍 Google Jobs Scraper	🚪 Glassdoor Scraper
🚀 Crunchbase.com Scraper Scraper	💼 Upwork Scraper
👹 Monster Job Search Scraper	🎨 Fiverr Scraper
👔 LinkedIn Company URL - Mass Profile Finder	💼 LinkedIn Company Ads
📰 LinkedIn Posts Scraper	🏙️ Craigslist Scraper

❓How to scrape LinkedIn companies and profiles

Learn how to scrape LinkedIn for URLs with these scrapers.

Is it legal to scrape Indeed data?

Web scraping is legal. However, different rules set by regulations such as GDPR or CCPA may apply based on where you are in the world. So be careful when scraping personal data (users, resumes, and other sensitive information) and avoid websites that are not publicly available or replicating copyrighted content. Read our is web scraping legal article to learn more about the subject.

Can I scrape Indeed with Python?

Yes. To carry out Indeed scraping using Python, you can use Apify API. It gives you programmatic access to the Apify platform: any datasets with scraped data, performance metrics, results, versions, and more. You can access the API using Python via apify-client PyPI package.

Can I scrape Indeed company reviews?

In theory, you can scrape anything available on the website. So Indeed reviews could be scraped. Indeed Scraper doesn't have that functionality yet, but you can request it using the Issues tab. Another way would be to send a request to one of our community devs on Discord; they'll make a custom scraper for Indeed reviews for a fee.

Last but not least, you can build this Indeed Reviews scraper yourself - by using scraper templates or the open-source scraping library Crawlee.

How to scrape Threads

Natasha Lekh — Tue, 18 Jul 2023 22:00:00 +0000

Threads app has become a game changer for social media in 2023, reaching an astonishing 100 million users in the first five days of launch. Thats as many people as the population of Egypt or Vietnam 🤯

And although growth in the number of Threads users is reported to be beginning to slow down, and despite the report of Instagram-like restrictions on freedom of speech, frustration over Twitters recent API policies could keep fueling Threads user acquisition.

For web scraping, because Instagram and Threads users share usernames across both platforms, scraping Threads is an untapped opportunity for sentiment analysis, research, and understanding customer behavior. Its also the easiest example of doing comprehensive cross-platform social media scraping since the profile names are the same.

So since social media scraping is one of the most common business cases for data extraction, weve prepared an early-bird tutorial for Threads. In this blog post, we'll show you how to use a web scraping tool to scrape Threads and get as much public data as the Threads rate limits will allow us. Lets get on with it!

📲 What Threads data can I get using this scraper?

Here's the contact data you can get using Meta Threads Profile Scraper 🔗

🌐 Profile URL and ID
📷 Picture URL and ID
👤 Username
📛 Full name
🔢 Follower count
📝 Bio and bio link

🔍 How to scrape data from Threads app

Step 1. Choose Meta Threads Profile Scraper

Let's head over to Apify Store, our library of more than 1,000 ready-to-use tools able to get any data from the open web. Try searching for 'meta there, and you'll find the right tool. Use Meta Threads Profile Scraper 🔗. It is basically an unofficial version of Threads API which is free to try. Click the Try actor button.

Step 1. Choose Meta Threads Profile Scraper

If you dont have your Apify account yet, all you need to sign up is your email address, Gmail, or GitHub account. And the best part is, you can try our Threads Scraper for free right away.

Sign up to scrape Threads for free

Step 2. Add Threads usernames

Head over to the Threads app and pick the profiles you want to scrape. Copy and paste each username into the scrapers input. You can add as many profiles as you want using the + Add button, or upload a whole list at once using the Bulk edit button nearby.

Step 2. Add Threads usernames

Step 3. Click Start to begin scraping

Once you click the Start button, you'll see the output as the data begins to come in while our Threads scraper tool is operating, but it won't be done until the status changes to Succeeded. We've got our results in a total of just 20 seconds!

Step 3. Click Start to begin scraping

Step 4. Download Threads data

Once the data extraction is complete, you can view and download the extracted Threads data in Excel, JSON, CSV, and XML. You can now click the Storage tab -> Export button to download your data.

Step 4. Download Threads data

Same data but as HTML table

You can also clean and preprocess the scraped data before downloading it by removing any irrelevant information. Here, in our example, we only want username, full name, follower count, and some other details such as bio and URL in our dataset. In order to export only that data, well choose the relevant tags from the list of Selected fields before downloading them.

Clean and preprocess the scraped data before downloading it

Step 5. Bonus: export Threads data via API

If you dont want to download the data but plug it directly into your app or project after extraction, you can use the Apify API option. Go over to Storage and click on the API button in the top-right corner. There you can explore multiple API export options, so choose Get output.

Step 5. Bonus: export Threads data via API

If you need to, you can also see all API Endpoints and API Client tokens in Apify Console.

API Endpoints and API Client tokens in Console

❓FAQ

⚖️Is it legal to scrape Threads data?

Since they all belong to the same company, when extracting data from Threads, the same rule applies to scraping Facebook or Instagram. The safest is to make sure that you are not extracting personal data (even if its publicly available for anyone on the web) or at least that you have a good reason to do that. For more information on the topic of ethical scraping, see the article from our lawyers.

🛡 Do you need proxies to scrape Threads?

You generally need proxies to have a successful and generally reliable scraping process nowadays. But in the current situation with Threads app, you cant even use it if youre from the EU for example, let alone scrape it. So there are at least two major reasons you do need proxies for scraping Threads. We recommend using residential proxies as the most reliable way to go about scraping anything from Meta in 2023. Fortunately, our free plan offers a free trial of residential proxies , so you can fully test this scraper tool on getting data from Threads, Instagram or Facebook.

📹 Can I download Threads videos?

Yes, apart from the Meta Threads Profile Scraper that this tutorial is about, there are several other tools available for scraping Threads or similar content from different platforms. Here are a few examples:

These tools are just a few examples, and there may be other specialized Thread scrapers available in the future. We always recommend exploring the available options and choosing a tool that aligns with your scraping needs.

🎂 Can I scrape both Threads users and Instagram users?

Yes. You can scrape both Threads users and Instagram users since they share the same usernames and profiles. By using scraping techniques, you can extract data from both platforms simultaneously and get insights into user profiles and their activities on both Meta platforms. You may want to check out our Instagram Profile Scraper as well.

❓How to extract contact details from Google Maps
Download Google Maps contact details in Excel.
Learn how to extract emails, social profiles, phone numbers and addresses from Google Maps

Top 10 web scrapers for lead generation

Natasha Lekh — Tue, 06 Jun 2023 15:44:43 +0000

Learn to automate contact data collection and outreach using web automation tools.

Lead generation plays a vital role in the success of any business. However, it can be quite a time-consuming process to find potential customers and gather their contact information. Thankfully, there are tools available that can make the reach-out task much easier and more efficient. In this blog post, we'll delve into some of the best tools for lead generation, with a special focus on the incredibly versatile scraping tools able to make lead collecting easier.

These data extraction tools automate the collection of contact details like email addresses and phone numbers from a variety of websites, enabling you to filter and target your leads, maintain organized and trustworthy databases, and streamline your sales funnel. The end result? Improved conversions and substantial revenue growth for your business.

❓How can you use web scraping tools for lead generation?

Here are some reasons why web scrapers (or web crawlers) can be helpful when generating leads:

🕰 Saving time. Web scraping tools automate the process of gathering data from websites, allowing you to quickly extract relevant information without spending excessive time and effort. This frees up your valuable resources to focus on other important aspects of your business.

🔫 Laser-focused targeting. Using data extraction tools, you can specifically target websites that are particularly relevant to your industry, niche, or target audience. You can pre-select them before reaching out to them. By extracting data from these sources specifically, you can identify potential leads that align with your ideal customer profile, increasing your chances of conversion.

🍃 Up-to-date information. Some web scrapers can be set up to scrape data at regular intervals (daily, hourly, weekly, etc.), ensuring that you have access to the most current and accurate information without any active participation or keeping an eye on it. This real-time data will enable you to reach out to leads promptly and via the correct credentials, helping you to stay ahead of the competition and increasing your chances of success.

🎁 Personalized outreach. By using web scraping software, you can gather valuable insights about your leads, such as their interests, preferences, or struggles. This information empowers you to tailor your outreach efforts and deliver personalized messages which would resonate with your prospects.

💪 Competitive advantage. Last but not least, by leveraging web scraping tools effectively, you can gain a competitive edge by accessing data that your competitors may not have. This unique information can give you valuable insights into market trends, customer behavior, and emerging opportunities, allowing you to make informed decisions and stay ahead in your industry.

How to use business lead scrapers for lead generation

How business lead scrapers save you time on lead generation

blog.apify.com

❓Which web scraping tools are best for lead generation?

📬 Case 1. Automated lead collection

Probably the most classic use case. Web scraping is an essential technique for automating the process of extracting contact information from potential leads. Emails, addresses, phone numbers, social media details - you name it. Here's a range of tools that enable you to extract those and other contact details from websites effortlessly, all available online and to the public

10. Contact Details Scraper .

The strength of this tool is its universality. This web scraper allows you to extract contact information from any website, including email addresses, phone numbers, and more.

9. Google Maps Email Extractor .

This scraper is built for getting contact data from Google Maps. You can extract contact details from any Google Maps listings in particular, helping you find potential leads in specific locations - countries, cities, or areas.

8. Facebook Pages Scraper .

So many businesses list their contact data on their Facebook Pages. This scraper can be used to extract those contact details from Facebook Pages and expand your lead database with social media data.

7. Facebook Likes Scraper .

Using this data extractor, you can reach out to the audience which is showing interest in particular topics discussed on Facebook.

How to scrape contact details from any website on the web 📩

🔻 Case 2. Streamlined sales funnel

It's no less important to organize and streamline your sales funnel. Here are a few tools that allow you to automate the process of collecting contact details, helping you accelerate your sales processes and achieve better results:

6. Email & Phone Extractor .

Extract email addresses and phone numbers from various sources, including websites and documents, to build targeted lead lists.

5. LinkedIn Company URL Finder.

LinkedIn lead generation command: you can extract URLs of LinkedIn company pages, generate a list and import that LinkedIn contact data into your database for effective lead management.

4. Yellow Pages Scraper .

Yellow Pages has been a great source of leads for American-based businesses. You can scrape Yellow Pages directories to bundle potential leads and create targeted email lists.

3. Crunchbase.com Scraper .

Get email leads from Crunchbase.com by extracting valuable company data. create a B2B lead database

LinkedIn Company URL Finder · Apify

LinkedIn Company URL Finder extracts the URLs of LinkedIn company pages instead of you searching for it. From a list of company names, you get a list of their LinkedIn URLs. A must have tool when you have hundreds of linkedin company url to retrieve : this tool makes this fast and automatic !

apify.com

Collect contact details and generate targeted lead lists using web scrapers

🗃 Case 3. Building reliable databases

It's one thing to get your leads, but managing them and maintaining a reliable database is no less crucial for successful marketing campaigns. Tools that allow you to send the extracted data directly to your CRM or other marketing tools in the right format can ensure your databases always remain accurate and up-to-date, giving your sales team a head start.

2. Craigslist Scraper .

This tool enables you to scrape contact details from Craigslist listings, providing you with a valuable source of leads and making sure your CRMs are always up-to-date.

1. Yellow Pages Scraper with Email .

With this Yellow Pages email scraper, you can extract full contact information from Yellow Pages listings, including but not limited to email addresses, and easily populate your databases.

Craigslist Ad Scraper · Apify

Extract data from classified advertisements on Craigslist. Scrape contact details from jobs, housing, items wanted, items for sale, services, community service, gigs, events and resumes listed on Craigslist. Download listings data in JSON, XML, Excel, and other versatile

apify.com

🥾 Steps to introduce web automation into your lead generation process

To make the most of these lead-generation tools, follow these simple steps:

Sign up for an Apify account: create a free Apify account and get $5 free prepaid platform usage every month.
Choose a scraper or web automation tool: browse Apify Store to find pre-built tools suited to lead generation.
Scrape data from the website of your choice.
Download web data in various formats, as well as schedule, integrate, and monitor the data extraction. All for seamless data handling and monitoring of the process for continuous lead generation.

Automating lead generation is crucial for maximizing your marketing efforts and driving revenue growth. Our versatile tools provide effective solutions for web scraping, database management, and streamlining your sales funnel. By utilizing these top lead generation tools, you can save time, target your marketing campaigns more efficiently, and stay ahead of the competition in the dynamic business landscape.

6 Automation ideas for sales teams

Much of the busy work sales agents engage in on a daily basis can be performed by bots. Discover 6 ways how sales automation allows sales team members to shed the burden of busy work and focus fully on those tasks that really help their companies grow.

blog.apify.com

How to deploy and schedule your Cypress tests in the cloud

Natasha Lekh — Sun, 04 Jun 2023 19:17:01 +0000

If you're already using Apify and have Cypress code that needs a home, you're in luck. We created Cypress Test Runner to make running Cypress tests in the Apify cloud a breeze.

If you're already using Apify and have Cypress code that needs a home, you're in luck. We created Cypress Test Runner to make running Cypress tests on the Apify cloud platform a breeze. Let's see how you can use it.

☁️Cloud alternatives for testing

When it comes to running your Cypress tests in the cloud, there are a few options available:

Cypress Cloud and its alternatives. Cypress Cloud provides a hosted solution for running your Cypress tests, allowing you to easily execute your tests on a remote infrastructure. There are also open-source alternatives for running Cypress tests and collecting test results.
Run it in CI/CD pipelines. Integrating your Cypress tests into your CI/CD pipelines is another effective approach tested and used by us here at Apify on a daily basis. By plugging your tests into your existing processes, such as to GitHub Actions, you can seamlessly incorporate testing into your development workflow.

Example of our CI/CD pipeline we have on GitHub

🥾Guide on how to deploy and schedule Cypress tests in the cloud

If you want to set regular runs for your Cypress tests in the Apify cloud, follow these steps:

Step 1. Choose Cypress Test Runner

Step 1. Choose Cypress Test Runner

Head over to Apify Store and search for Cypress. You'll find the Cypress Test Runner 🔗. Click Try for free and sign in using your email to get started.

Step 2. Record or write your tests

How to record a Cypress test using Cypress Recorder

Of course, you can add manually written tests as well. But by using the Cypress Test Recorder extension, you can make your test writing a bit faster. Check out our guide 🔗 on how to record your tests using the Cypress Recorder Extension in Chrome.

How to use Cypress Test Recorder

Is Cypress Recorder the tool to automate writing manual tests?

blog.apify.com

Step 3. Export and paste your tests

Step 3. Export and paste your tests

After recording your tests, export them from the Recorder and paste the test code into Cypress Test Runner. This step allows you to set up the runner with your specific tests for execution.

Step 4. Start the run

Run result: one test failed and one test succeeded

Now click on Start and wait for the run to complete the test. Once your tests are done running, you'll see the Status of the run change to Succeeded or Failed. Note that if any of your Cypress tests fail, the whole run will be marked as Failed.

You can also check the full log of every test run.

Step 5. Check the dashboard

To see the test results just copy the dashboard link on top of the Run panel. The dashboard will provide you some valuable insights into your test runs, including detailed test results, metrics, and visual aid. You can use this information to analyze the performance and identify any issues or areas for improving your tests.

Step 5. Check the dashboard

Step 6. Save your run as a task

Step 6. Save your run as a task

If you want to schedule your test to run on a regular basis, you'll have to create a Task. To create a Task, simply head for the Input of your Run and click Save input to new task.

Now name your task, tweak the Input if needed, and click Schedule.

Step 7. Schedule your test to run hourly, daily, etc.

One last thing left to do: choose how often you want the Cypress test to run (weekly, monthly, daily, hourly, custom time or by cron expression). Don't forget to pick your time zone so the run happens exactly when you need it to.

Step 7. Schedule your test to run hourly, daily, etc.

By following these steps with Cypress Test Runner, you can effortlessly deploy and schedule your Cypress tests in the Apify cloud. Enjoy the benefits of running your tests remotely, gaining valuable insights, and improving the quality of your software.

📇 Template for Cypress Actors

If you prefer to build your own Cypress Actor, we have a template available to help you get started. This template serves as the foundation for any container with Cypress tests (including the one this tutorial is about) and helps our users to create their own Cypress testing solutions faster. Try it and see if it helps you too!

Template for creating Cypress Actors

How to extract emails, social profiles, phone numbers and addresses from Google Maps

Natasha Lekh — Wed, 31 May 2023 17:42:37 +0000

Sometimes you might notice that the contact details of places on Google Maps seem incomplete. Heres a tutorial on how to complete your database with phone numbers, emails, addresses, ZIP codes, and social media details from places registered on Google Maps.

🗺 Does Google Maps include all business contact details?

No. Google Maps is an ever-expanding database of free business data: marketers can find place names, addresses, phone numbers, and official websites. However, Google Maps does not provide information on business email addresses or social media accounts to reach out through.

Luckily, most businesses do list their contact details in one place their websites which are often mentioned on their Google Maps detail cards. For instance, this New York place does not indicate their email or socials on Google Maps, but they do list their official website.

A Google Maps place usually contains its address, phone number, and website. To find more contact details, one needs to actually visit their website.

When we head over to their website, and its Contact section, we can often find all the missing contact details. So are we going to go with this flow for every place we are interested in?

Places website contains its email address, Facebook and Instagram.

No, let's save time and use a tool able to extract and combine the basic contact information from Google Maps with extra contact info found on places websites. By using Google Maps Email Extractor 🔗 , you can extract business contacts data from Google Maps on a large scale. The tool allows you not only to retrieve comprehensive contact information including business address, all listed phone numbers, emails, social account names, but also any other listed details from companies registered on the platform.

🤨 How does the Google Maps Email Extractor work?

Google Maps Contact Details Scraper combines the capabilities of three tools: Google Maps Scraper 🔗, Contact Details Scraper 🔗, and the Merge, Dedup & Transform Datasets Actor 🔗. Here's how the process works:

Google Maps Scraper extracts available business data from Google Maps 📍
Contact Details Scraper visits the businesses' websites and supplements contact details with missing info.
Merge Datasets Actor cleans up the dataset, removes duplicates (phone number, for instance), and keeps only the most important Google Maps contact data.

What contact details from Google Maps can I get with this tool?

Here's the contact data you can get using Google Maps Email Extractor 🔗

📍 addresses (city, country, ZIP code)
☎️ phone numbers
📩 emails
🌐 indicated website
📱 social media links: Instagram, YouTube, Facebook, LinkedIn, and Twitter handles
⭐️ all the other information available on Google Maps: place name, description and URL, reviews, geolocation, delivery options, popular visiting times, etc.

🥾 How to export business contacts and emails from Google Maps

Step 1. Use Google Maps Scraper to extract place data

We can use Google Maps Scraper 🔗 to crawl Google Maps and extract information. Google Maps Scraper is a powerful web scraping tool that allows you to extract contact details from Google Maps on a large scale. This tool will scrape all places data from the chosen area on Google Maps.

Here's how you can use this tool to extract contact details: You can provide the URL of the place you want to scrape as input to the scraper. Or fill in the scraper input yourself.

Step 1. Use Google Maps Scraper to extract place data.

Input for Google Maps scraper to scrape data from 200 restaurants in NYC.

Extracted data from 200 restaurants in NYC.

The dataset you get will already contain some contact information with the data available on Google Maps cards: address, phone, and website. Now lets add the emails and socials to complete the set.

How to scrape Google Maps data

Extract data without limits with this unofficial Google Maps API.

blog.apify.com

Step 2. Copy the Google Maps dataset ID

Copy the task, run, or dataset ID with the extracted information from the Google Maps Scraper.

Step 2. Copy the Google Maps dataset ID.

Step 3. Open Google Maps Email Extractor tool

In Apify Console, head over to the Store tab and navigate to Google Maps Email Extractor. You can find it in Lead generation section or by searching.

Step 3. Open Google Maps with Contact Details tool

Step 4. Paste the ID into Google Maps Email Extractor input

Then, paste the ID into the designated field of Google Maps Email Extractor 🔗 and click on Start.

Step 4. Paste the ID into Google Maps Email Extractor input

It might take some time for the scraper to finish its job, depending on the length of your dataset.

Step 5. Get the emails, addresses, and social media details

Once the scraping process is complete, you can preview and download the dataset containing the extracted contact details.

Example of a large Google Maps dataset containing 200 contacts of scraped NYC restaurants. To see the extracted business data, seek out Contact Details column

Step 6. Download the contact details dataset

You can also clean up your dataset before exporting it. Since the dataset can contain a lot of fields from Google Maps, it might be convenient to preselect the fields that you want to keep in your final dataset. You can download your data in any of the given formats: Excel, XML, JSON, CSV, HTML.

Step 6. Download the contact details dataset.

🖇 How to use the extracted contact details from Google Maps

Extracting contact details from Google Maps can be incredibly useful in various scenarios, including:

Extracting business contacts and emails at scale.
Building a database with up-to-date contact information.
Updating an old database of contact information.
Creating b2b cold email marketing campaigns.
Finding sales, partnership, and sponsorship prospects.
Mining for local sales leads.

👮 Is it legal to scrape emails and addresses from Google Maps?

Our Google Maps scrapers are ethical and do not extract any private user data. They only extract what businesses have chosen to share publicly on the web.

However, you should be aware that some results such as reviews could contain personal data. You should not scrape personal data unless you have a legitimate reason to do so. For legitimate details in more details, read our blog post on the legality of web scraping and ethical scraping.

FAQ

Do I need to install an extension to scrape contacts from Google Maps?

No. Google Maps Email Extractor runs in the cloud, you dont need to install anything. All you need to do to start using the tool is to create a free Apify account using your email.

🌆 Can I scrape an entire city for contact details?

Yes. But this type of Google Maps scraping will take some time and resources. You can choose to extract all places in that area or a specific type of place (restaurants, museums, cafes, universities, hospitals, grocery shops, pharmacies, etc.) Moreover, you can scrape an entire country for contact details too. Just don't forget to use Google Maps Email Extractor 🔗 to get all the contact details missing from Google Maps.

🧑🍳 Can I scrape restaurant chains and their contact details?

Yes. You can scrape, for instance, all Starbucks or all McDonald's in the area by using the Search options parameter in Google Maps Scraper 🔗. Then proceed to enhance your dataset with all the contact details with Google Maps Email Extractor 🔗 as we did in this tutorial.

How to scrape restaurant chains from a city, state or country.

🛰 Can I extract longitude and latitude from Google Maps?

Yes, every place on Google has longitude and latitude assigned to them. You can easily scrape those using Google Maps Scraper 🔗.

📡 Can I scrape entire areas of Google Maps for contact details?

Yes. If your area has a name (City, State or Country), you can scrape Google Maps places by simply inputting the name. But you can also extract data by geolocation (multiple points of longitude and latitude). you can create various irregular shapes for specific areas on the map such as circles, squares, and polygons. We recommend using Geojson.io for an easy coordinates definition and following our guide to see how to apply those coordinates in Google Maps Scraper 🔗