Forem: CoderHXL

Traditional crawler or AI-assisted crawler? How to choose?

CoderHXL — Tue, 23 Apr 2024 06:33:52 +0000

Foreword

In the field of data fetching, traditional crawlers and AI-assisted crawlers have their own advantages. The traditional crawler crawls data based on rules, which is suitable for websites with stable structure and clear rules. However, with the frequent changes and complications of the website structure, traditional crawlers have gradually exposed their limitations. In contrast, AI-assisted crawlers use artificial intelligence technology to intelligently analyze web pages and adapt to changes, showing higher flexibility and accuracy. So, in the face of different grasping needs, how should we choose? This article will deeply discuss the characteristics and advantages and disadvantages of traditional crawlers and AI-assisted crawlers, so as to provide you with decision-making reference.
What are the traditional crawlers and AI-assisted crawlers respectively

Traditional crawler

Traditional crawlers mainly rely on fixed rules or patterns to grab web data. They typically locate and extract the required information by identifying specific elements in a web page, such as class names, tags, or structures. However, the limitations of this approach are obvious. Once a site is updated to change the original class name, label, or structure, traditional crawlers will fail because they cannot recognize the new elements, resulting in data fetching failures or errors.

AI-assisted crawler

Ai-assisted crawlers intelligently analyze and understand the content of web pages to more accurately locate and extract the required information. Through technologies such as natural language processing, they are able to understand the semantic information of the web page, so that they can more precisely locate the required data, and even after the website is updated, AI-assisted crawlers can continue to effectively crawl data.

Example

The crawler uses x-crawl. The crawled websites are all real. To avoid disputes, https://www.example.com is used instead.

Traditional crawler to obtain movie information of movie rankings through specific elements in the web page
Crawler + AI, the crawler is paired with AI to obtain movie information from the movie rankings

Traditional crawler

Traditional crawler, obtains movie information of movie rankings through specific elements in the webpage

import { createCrawl } from 'x-crawl'

//Create a crawler application
const crawlApp = createCrawl()

// crawlPage is used to crawl pages
crawlApp.crawlPage('https://www.example.com').then(async (res) => {
   const { page, browser } = res.data

   // Wait for the element to appear on the page
   await page.waitForSelector('#wrapper #content .article')
   const filmHandleList = await page.$$('#wrapper #content .article table')

   const pendingTask = []
   for (const filmHandle of filmHandleList) {
     // Cover link (picture)
     const picturePending = filmHandle.$eval('td img', (img) => img.src)
     //Movie name(name)
     const namePending = filmHandle.$eval(
       'td:nth-child(2) a',
       (el) => el.innerText.split(' / ')[0]
     )
     // Introduction (info)
     const infoPending = filmHandle.$eval(
       'td:nth-child(2) .pl',
       (el) => el.textContent
     )
     // score
     const scorePending = filmHandle.$eval(
       'td:nth-child(2) .star .rating_nums',
       (el) => el.textContent
     )
     // Number of comments (commentsNumber)
     const commentsNumberPending = filmHandle.$eval(
       'td:nth-child(2) .star .pl',
       (el) => el.textContent?.replace(/\(|\)/g, '')
     )

     pendingTask.push([
       namePending,
       picturePending,
       infoPending,
       scorePending,
       commentsNumberPending
     ])
   }

   const filmInfoResult = []
   let i = 0
   for (const item of pendingTask) {
     Promise.all(item).then((res) => {
       // filmInfo is a movie information object, the order is determined before
       const filmInfo = [
         'name',
         'picture',
         'info',
         'score',
         'commentsNumber'
       ].reduce((pre, key, i) => {
         pre[key] = res[i]
         return pre
       }, {})

       //Save each movie information
       filmInfoResult.push(filmInfo)

       //Last processing
       if (pendingTask.length === ++i) {
         browser.close()

         // Arrange, decide whether it is more or less based on the quantity
         const filmResult = {
           element: filmInfoResult,
           type: filmInfoResult.length > 1 ? 'multiple' : 'single'
         }

         console.log(filmResult)
       }
     })
   }
})

AI assisted crawler

Crawler + AI, let crawler and AI obtain movie information from movie rankings

import { createCrawl, createCrawlOpenAI } from 'x-crawl'

//Create a crawler application
const crawlApp = createCrawl()

//Create AI application
const crawlOpenAIApp = createCrawlOpenAI({
   clientOptions: { apiKey: process.env['OPENAI_API_KEY'] },
   defaultModel: { chatModel: 'gpt-4-turbo-preview' }
})

// crawlPage is used to crawl pages
crawlApp.crawlPage('https://movie.douban.com/chart').then(async (res) => {
   const { page, browser } = res.data

   // Wait for the element to appear on the page and get the HTML
   await page.waitForSelector('#wrapper #content .article')
   const targetHTML = await page.$eval(
     '#wrapper #content .article',
     (e) => e.outerHTML
   )

   browser.close()

   // Let AI obtain movie information (the more detailed the description, the better)
   const filmResult = await crawlOpenAIApp.parseElements(
     targetHTML,
     `This is a list of movies. You need to get the movie name (name), cover link (picture), introduction (info), rating (score), and number of comments (commentsNumber). Use bracketed words as attribute names`
   )

   console.log(filmResult)
})

Results of two examples

Movie information ultimately displayed by two examples

{
   "elements": [
     {
       "name": "Old Fox",
       "picture": "https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2900908599.webp",
       "info": "2023-10-27 (Tokyo International Film Festival) / 2023-11-24 (Taiwan, China) / Bai Runyin / Liu Guanting / Chen Muyi / Liu Yier / Kadowaki Mai / Huang Jianwei / Wen Shenghao / Ban Tiexiang / Yang Liyin / Fu Mengbai/Gao Yingxuan/Zhuang Yizeng/Zhang Zaixing/Xu Bowei/Guan Qing/Zhong Yao/You Jiaxuan/Zheng Yangen/Dai Yazhi/Jiang Ren/Xiao Hongwen...",
       "score": "8.1",
       "commentsNumber": "29211 people commented"
     },
     {
       "name": "Robot Dream",
       "picture": "https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2899644068.webp",
       "info": "2023-05-20(Cannes Film Festival) / 2023-12-06(Spain) / 2024(Mainland China) / Ivan Labanda/Albert Trevor Segarra/ Rafa Calvo/José Garcia Toss/José Luis Mediavilla/Garcia Molina/Esther Sollance/Spain/France/Pablo· Berger...",
       "score": "9.1",
       "commentsNumber": "64650 people commented"
     },
     {
       "name": "Under the sun",
       "picture": "https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2904961420.webp",
       "info": "2023-06-11 (Shanghai International Film Festival) / 2023-11-02 (Hong Kong, China) / 2024-04-12 (Mainland China) / David Jiang / Yu Xiangning / Lin Baoyi / Liang Zhongheng / Chen Zhanwen / Zhou Hanning / Liang Yongting/Gong Cien/Bao Peiru/Zhu Baiqian/Zhu Baikang/Xu Yuexiang/Hu Feng/Bao Qijing/Gao Hanwen/Peng Xingying/Luo Haoming/Tan Yuying...",
       "score": "8.0",
       "commentsNumber": "36540 people commented"
     },
     {
       "name": "Poor thing",
       "picture": "https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2897662939.webp",
       "info": "2023-09-01 (Venice Film Festival) / 2023-12-08 (USA) / Emma Stone / Mark Ruffalo / Willem Dafoe / Rami Yusuf / Christopher Abbott/Susie Bemba/Jerrod Carmichael/Katherine Hunter/Vicki Pepperdine/Margaret Qualley/Hannah Schigula/Jack Patton... .",
       "score": "7.0",
       "commentsNumber": "130113 people commented"
     },
     {
       "name": "Perfect Day",
       "picture": "https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2898894527.webp",
       "info": "2023-05-25 (Cannes Film Festival) / 2023-12-21 (Germany) / 2023-12-22 (Japan) / Yakusho Koji / Emoto Tokio / Nakano Arisa / Yamada Aoi / Aso Yumi/Sayuri Ishikawa/Tomokazu Miura/Mini Tanaka/Hiroto Oshita/Inuko Inuyama/Motomi Makiguchi/Tan Nagai/Ken Naoko/Moro Shioka/Ken Moriyu/Iru Katagiri/Xinguto Serizawa...",
       "score": "8.3",
       "commentsNumber": "33562 people commented"
     },
     {
       "name": "New Dragon Killing Formation",
       "picture": "https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2905374090.webp",
       "info": "2024-03-08 (South by Southwest Film Festival) / 2024-03-21 (USA Network) / Jake Gyllenhaal / Conor McGregor / Jessica Williams / Billy Magnussen/Daniela Melchior/Jimisola Ekumero/Lucas Gage/Travis Van Winkle/Darren Barnett/Joe Quem de Almeida...",
       "score": "6.3",
       "commentsNumber": "9980 people commented"
     },
     {
       "name": "Seoul Spring",
       "picture": "https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2905204009.webp",
       "info": "2023-11-22 (South Korea) / Hwang Jung-min / Jung Woo-sung / Lee Sung-min / Park Hae-jun / Kim Sung-joon / Park Hoon / An Se-ho / Jung Yun-ha / Jung Hae-in / Nam Yun-ho / Jeon Soo-ji / South Korea / Kim Sung-soo / 141 Minutes/Seoul Spring/Drama/Sung-su Kim/Korean",
       "score": "8.8",
       "commentsNumber": "171858 people commented"
     },
     {
       "name": "Goldfinger",
       "picture": "https://img1.doubanio.com/view/photo/s_ratio_poster/public/p2901830629.webp",
       "info": "2023-12-30 (Mainland China) / Tony Leung / Andy Lau / Charlene Choi / Simon Yam / Alex Fong / Chan Ka Lok / Bai Zhi / Jiang Haowen / Pacific Insurance / Chin Ka Lok / Anita Yuen / Chow Jiayi / Sam Jiaqi / Li Jingjun / Ng Siu Hin / Ke Weilin / Feng Yongxian / Du Yaoyu/Li Jiancheng/Gu Yongfeng/Hong Kong, China/Mainland China/Chuang Wenqiang...",
       "score": "6.1",
       "commentsNumber": "135956 people commented"
     },
     {
       "name": "American novel",
       "picture": "https://img9.doubanio.com/view/photo/s_ratio_poster/public/p2902166424.webp",
       "info": "2023-09-08(Toronto International Film Festival) / 2023-12-15(USA) / Jeffrey Wright/ Tracey Ellis Ross/ John Ortiz/ Issa· Ray/Sterling K. Brown/Erica Alexander/Leslie Goseth/Adam Brody/Keith David/Myra Lucretia Taylor/Raymond Anthony Thomas...",
       "score": "7.7",
       "commentsNumber": "26223 people commented"
     },
     {
       "name": "interest area",
       "picture": "https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2899514583.webp",
       "info": "2023-05-19 (Cannes Film Festival) / 2023-12-15 (USA) / Christian Fouridower / Sandra Wheeler / John Carter House / Ralph Herbert Erfurt/Freya Kreutzkamm/Max Baker/Imorgan Kugger/Stephanie Petrowicz/Ralph Zillman/Mary Rosa Tityan. ..",
       "score": "7.4",
       "commentsNumber": "24875 people commented"
     }
   ],
   "type": "multiple"
}

Compare

Steps required for traditional crawlers to extract information

const pendingTask = []
for (const filmHandle of filmHandleList) {
   const picturePending = filmHandle.$eval('td img', (img) => img.src)
   const namePending = filmHandle.$eval(
     'td:nth-child(2) a',
     (el) => el.innerText.split(' / ')[0]
   )
   const infoPending = filmHandle.$eval(
     'td:nth-child(2) .pl',
     (el) => el.textContent
   )
   const scorePending = filmHandle.$eval(
     'td:nth-child(2) .star .rating_nums',
     (el) => el.textContent
   )
   const commentsNumberPending = filmHandle.$eval(
     'td:nth-child(2) .star .pl',
     (el) => el.textContent?.replace(/\(|\)/g, '')
   )

   pendingTask.push([
     namePending,
     picturePending,
     infoPending,
     scorePending,
     commentsNumberPending
   ])
}

const filmInfoResult = []
let i = 0
for (const item of pendingTask) {
   Promise.all(item).then((res) => {
     const filmInfo =
       ['name', 'picture', 'info', 'score', 'commentsNumber'].reduce <
       any>
       ((pre, key, i) => {
         pre[key] = res[i]
         return pre
       },
       {})

     filmInfoResult.push(filmInfo)

     if (pendingTask.length === ++i) {
       const filmResult = {
         element: filmInfoResult,
         type: filmInfoResult.length > 1 ? 'multiple' : 'single'
       }
     }
   })
}

Relying on fixed class names and structures, the process is still relatively cumbersome.

Steps required for AI-assisted crawlers to extract information

const filmResult = await crawlOpenAIApp.parseElements(
   targetHTML,
   `This is a list of movies. You need to get the movie name (name), cover link (picture), introduction (info), rating (score), and number of comments (commentsNumber). Use bracketed words as attribute names`
)

A matter of words.

Traditional crawlers need to rely on fixed class names and various cumbersome operations to obtain data. If the website is updated frequently, changes in class names or structures after the website is updated may cause the traditional crawler crawling strategy to fail. It is necessary to re-obtain the latest class name and update various operations to crawl data.
AI-assisted crawler only needs a paragraph to obtain the required data more efficiently, intelligently and conveniently. You can even pass the entire HTML to AI to help us operate it. Since the website content is more complex, the location to be retrieved needs to be described more accurately, and a large amount of Tokens will be consumed. However, even if subsequent updates to the website cause the class name or structure to change, it can still be crawled normally. to data, because we can no longer rely on fixed class names or structures to locate and extract the required information, but let AI understand and parse the semantic information of web pages, thereby extracting the required data more efficiently, intelligently and conveniently.

If more content is required, then the traditional crawler will take more steps, while the AI-assisted crawler can do it by just adding a few sentences, and there is no need to worry about whether the class name and structure of the website will change after the website is updated.

Summarize

Traditional crawlers mainly rely on preset rules or patterns to crawl web page data, and they perform well on websites with stable structures and clear rules. However, with the rapid development of network technology and frequent updates of website structure, traditional crawlers are facing more and more challenges. Once the website structure changes, traditional crawlers usually need to readjust the rules, which may even lead to crawling failure, which greatly reduces its efficiency and accuracy.

In contrast, AI-assisted crawlers combine artificial intelligence technology to intelligently analyze the structure and semantics of web pages and adapt to changes in the website. Through technologies such as machine learning and natural language processing, AI-assisted crawlers can identify and learn features in web pages to more accurately locate and extract the required data. This allows AI-assisted crawlers to maintain efficient crawling capabilities when facing complex and changing website structures.

In general, traditional crawlers and AI-assisted crawlers each have their own applicable scenarios. For websites with stable structure and clear rules, traditional crawlers may be a more economical and direct choice. However, for websites with complex structures and frequent updates, AI-assisted crawlers have shown greater flexibility and accuracy advantages. When choosing, we need to comprehensively consider factors such as specific crawling needs, website characteristics, and resource investment.

resource

The crawlers used in the examples in this article are all from x-crawl. Whether it is a traditional crawler or an AI-assisted crawler, it can satisfy you, and it also has many useful functions.

x-crawl

x-crawl is a flexible Node.js AI-assisted crawler library. Flexible usage and powerful AI assistance functions make crawler work more efficient, intelligent and convenient.

It consists of two parts:

Crawler: It consists of crawler API and various functions, which can work normally even without relying on AI.
AI: Currently based on the large AI model provided by OpenAI, AI simplifies many tedious operations.

Features

🤖 AI Assistance - Powerful AI assistance function makes crawler work more efficient, intelligent and convenient.
🖋️ Flexible writing - A single crawling API is suitable for multiple configurations, and each configuration method has its own advantages.
⚙️Multiple uses - Supports crawling dynamic pages, static pages, interface data and file data.
⚒️ Control page - Crawling dynamic pages supports automated operations, keyboard input, event operations, etc.
👀 Device Fingerprint - Zero configuration or custom configuration to avoid fingerprint recognition to identify and track us from different locations.
🔥 Asynchronous Synchronization - Asynchronous or synchronous crawling mode can be performed without switching the crawling API.
⏱️ Interval crawling - no interval, fixed interval and random interval, determine whether to crawl with high concurrency.
🔄 Failed retry - Customize the number of retries to avoid crawling failures due to temporary problems.
➡️ Rotation proxy - Automatic proxy rotation with retries on failure, custom error times and HTTP status codes.
🚀 Priority Queue - According to the priority of a single crawling target, it can be crawled ahead of other targets.
🧾Crawling information - Controllable crawling information will output colored string information in the terminal.
🦾 TypeScript - Own types and implement complete types through generics.

If you think x-crawl is helpful to you, or if you like x-crawl, you can give x-crawl repository a point on GitHub A star. Your support is the driving force for our continuous improvement! thank you for your support!

x-crawl GitHub: https://github.com/coder-hxl/x-crawl

x-crawl documentation: https://coder-hxl.github.io/x-crawl/

AI+Node.js x-crawl crawler: Why are traditional crawlers no longer the first choice for data crawling?

CoderHXL — Wed, 17 Apr 2024 04:59:13 +0000

AI and Node.js crawler combination

When AI is paired with Node.js crawlers, this combination makes data collection smarter and more efficient. AI can help Node.js crawler to achieve more accurate target positioning. Traditional crawlers often rely on fixed rules or templates to capture data, but this way is often powerless in the face of complex and changeable web structure.

Why do we need AI-assisted crawlers

With the rapid development of network technology, website updates become more and more frequent, and the change of class name or structure often brings no small challenge to the crawlers that rely on these elements. In this context, crawlers combined with AI technology have become a powerful weapon to deal with this challenge.

First, a change in class name or structure after a website is updated can render traditional crawling strategies ineffective. This is because crawlers often rely on fixed class names or structures to locate and extract the information they need. Once these elements change, crawlers may not be able to accurately find the required data, which affects the effectiveness and accuracy of data fetching.

However, crawlers that incorporate AI technology are better able to cope with this change. AI can also understand and parse the semantic information of web pages through technologies such as natural language processing, so as to extract the required data more accurately.

In summary, crawlers combined with AI technology can better cope with the problem of class names or structural changes after website updates.

What is x-crawl?

x-crawl is a flexible Node.js AI-assisted crawler library. Flexible use and powerful AI-assisted functions make the crawler work more efficient, intelligent and convenient.

It consists of two parts:

Crawler: Composed of crawler API and various functions, it can work properly even without relying on AI.
AI: Currently based on the large AI model provided by OpenAI, AI simplifies many tedious operations.

x-crawl GitHub: https://github.com/coder-hxl/x-crawl

x-crawl Documentation: https://coder-hxl.github.io/x-crawl/cn/

Features

🤖 AI Assistance - Powerful AI assistance function makes crawler work more efficient, intelligent and convenient.
🖋️ Flexible writing - A single crawling API is suitable for multiple configurations, and each configuration method has its own advantages.
⚙️Multiple uses - Supports crawling dynamic pages, static pages, interface data and file data.
⚒️ Control page - Crawling dynamic pages supports automated operations, keyboard input, event operations, etc.
👀 Device Fingerprinting - Zero configuration or custom configuration to avoid fingerprint recognition to identify and track us from different locations.
🔥 Asynchronous Sync - Asynchronous or synchronous crawling mode without switching crawling API.
⏱️ Interval crawling - no interval, fixed interval and random interval, determine whether to crawl with high concurrency.
🔄 Failed Retry - Customize the number of retries to avoid crawling failures due to temporary problems.
➡️ Rotation proxy - Automatic proxy rotation with failed retries, custom error times and HTTP status codes.
🚀 Priority Queue - Based on the priority of a single crawl target, it can be crawled ahead of other targets.
🧾 Crawl information - Controllable crawl information, which will output colored string information in the terminal.
🦾 TypeScript - Own types and implement complete types through generics.

Example of AI and x-crawl crawler combination

The combination of crawler and AI allows the crawler and AI to obtain pictures of high-rated vacation rentals according to our instructions:

import { createCrawl, createCrawlOpenAI } from 'x-crawl'

//Create a crawler application
const crawlApp = createCrawl({
  maxRetry: 3,
  intervalTime: { max: 2000, min: 1000 }
})

//Create AI application
const crawlOpenAIApp = createCrawlOpenAI({
  clientOptions: { apiKey: process.env['OPENAI_API_KEY'] },
  defaultModel: { chatModel: 'gpt-4-turbo-preview' }
})

// crawlPage is used to crawl pages
crawlApp.crawlPage('https://www.airbnb.cn/s/select_homes').then(async (res) => {
  const { page, browser } = res.data

  // Wait for the element to appear on the page and get the HTML
  const targetSelector = '[data-tracking-id="TOP_REVIEWED_LISTINGS"]'
  await page.waitForSelector(targetSelector)
  const highlyHTML = await page.$eval(targetSelector, (el) => el.innerHTML)

  // Let the AI get the image link and de-duplicate it (the more detailed the description, the better)
  const srcResult = await crawlOpenAIApp.parseElements(
    highlyHTML,
    `Get the image link, don't source it inside, and de-duplicate it`
  )

  browser.close()

  // crawlFile is used to crawl file resources
  crawlApp.crawlFile({
    targets: srcResult.elements.map((item) => item.src),
    storeDirs: './upload'
  })
})

Even if the subsequent update of the website causes the class name or structure to change, it can climb to the data normally, because we no longer rely on the fixed class name or structure to locate and extract the required information, but let the AI understand and parse the semantic information of the web page, so as to extract the required data more efficiently, intelligently and conveniently.

You can even pass the entire HTML to AI to help us operate, because the website content is more complex you also need to more accurately describe the location to take, and will consume a lot of Tokens..

Procedure:

If you want to see the HTML the AI needs to process or see the srcResult (img url) returned by the AI after parsing the HTML according to our instructions:
here at the bottom, because there are too many HTML fragments inconvenient to view at the bottom of this example, you can go to see.

AI Intelligent on-demand analysis elements

No need to manually analyze the HTML page structure to extract the required element attributes or values. Now you just need to input the HTML code into AI and tell AI which elements you want to get information about, and AI will automatically analyze the page structure and extract the corresponding element attributes or values.

import { createXCrawlOpenAI } from 'x-crawl'

const xCrawlOpenAIApp = createXCrawlOpenAI({
  clientOptions: { apiKey: 'Your API Key' }
})

const HTMLContent = `
   <div class="scroll-list">
     <div class="list-item">Women's hooded sweatshirt</div>
     <div class="list-item">Men's sweatshirts</div>
     <div class="list-item">Women's sweatshirt</div>
     <div class="list-item">Men's hooded sweatshirt</div>
   </div>
   <div class="scroll-list">
     <div class="list-item">Men's pure cotton short sleeves</div>
     <div class="list-item">Men's pure cotton short sleeves</div>
     <div class="list-item">Women's pure cotton short sleeves</div>
     <div class="list-item">Men's ice silk short sleeves</div>
     <div class="list-item">Men's round neck short sleeves</div>
   </div>
`

xCrawlOpenAIApp
  .parseElements(HTMLContent, `Take all men's clothing and remove duplicates`)
  .then((res) => {
    console.log(res)
    /*
      res:
      {
        elements: [
          { content: "Men's hooded sweatshirt" },
          { content: "Men's sweatshirts" },
          { content: "Men's pure cotton short sleeves" },
          { content: "Men's ice silk short sleeves" },
          { content: "Men's round neck short sleeves" }
        ],
        type: 'multiple'
      }
    */
  })

Intelligent generation of element selectors

It can help us quickly locate specific elements on the page. Just enter the HTML code into AI and tell AI which elements you want to get selectors for, and AI will automatically generate appropriate selectors for you based on the page structure, greatly simplifying the tedious process of determining selectors.

import { createXCrawlOpenAI } from 'x-crawl'

const xCrawlOpenAIApp = createXCrawlOpenAI({
  clientOptions: { apiKey: 'Your API Key' }
})

const HTMLContent = `
   <div class="scroll-list">
     <div class="list-item">Women's hooded sweatshirt</div>
     <div class="list-item">Men's sweatshirts</div>
     <div class="list-item">Women's sweatshirt</div>
     <div class="list-item">Men's hooded sweatshirt</div>
   </div>
   <div class="scroll-list">
     <div class="list-item">Men's pure cotton short sleeves</div>
     <div class="list-item">Men's pure cotton short sleeves</div>
     <div class="list-item">Women's pure cotton short sleeves</div>
     <div class="list-item">Men's ice silk short sleeves</div>
     <div class="list-item">Men's round neck short sleeves</div>
   </div>
`

xCrawlOpenAIApp
  .getElementSelectors(HTMLContent, `all Women's wear`)
  .then((res) => {
    console.log(res)
    /*
      res:
      {
        selectors: '.scroll-list:nth-child(2) .list-item:nth-child(3)',
        type: 'multiple'
      }
    */
  })

Intelligent reply to crawler questions

Can provide you with intelligent answers and suggestions. Whether it is about crawling strategies, anti-crawling techniques or data processing, you can ask AI questions, and AI will provide you with professional answers and suggestions based on its powerful learning and reasoning capabilities to help you complete your tasks better. Reptile task.

import { createXCrawlOpenAI } from 'x-crawl'

const xCrawlOpenAIApp = createXCrawlOpenAI({
  clientOptions: { apiKey: 'Your API Key' }
})

xCrawlOpenAIApp.help('What is x-crawl').then((res) => {
  console.log(res)
  /*
    res:
    x-crawl is a flexible Node.js AI-assisted web crawling library. It offers powerful AI-assisted features that make web crawling more efficient, intelligent, and convenient. You can find more information and the source code on x-crawl's GitHub page: https://github.com/coder-hxl/x-crawl.
   */
})

xCrawlOpenAIApp
  .help('Three major things to note about crawlers')
  .then((res) => {
    console.log(res)
    /*
      res:
      There are several important aspects to consider when working with crawlers:

      1. **Robots.txt:** It's important to respect the rules set in a website's robots.txt file. This file specifies which parts of a website can be crawled by search engines and other bots. Not following these rules can lead to your crawler being blocked or even legal issues.

      2. **Crawl Delay:** It's a good practice to implement a crawl delay between your requests to a website. This helps to reduce the load on the server and also shows respect for the server resources.

      3. **User-Agent:** Always set a descriptive User-Agent header for your crawler. This helps websites identify your crawler and allows them to contact you if there are any issues. Using a generic or misleading User-Agent can also lead to your crawler being blocked.

      By keeping these points in mind, you can ensure that your crawler operates efficiently and ethically.
   */
  })

Summary

In the latest version of x-crawl, we have introduced powerful AI-assisted features to make crawler work more efficient, intelligent and convenient. This innovative feature is mainly reflected in the following aspects:

1. Intelligent on-demand analysis elements

Traditional crawler work often requires manual analysis of the HTML page structure to extract the desired element attributes or values. Now, with the AI assistance of x-crawl, you can easily implement intelligent on-demand analysis of elements. Just tell the AI which elements you want to get information about, and the AI will automatically analyze the page structure and extract the corresponding element attributes or values.

2. Intelligent generation element selector

Selectors are an integral part of the crawler's work, helping us quickly locate specific elements on the page. Now, x-crawl's AI assistant can intelligently generate element selectors for you. Simply input the HTML code into the AI, and the AI will automatically generate the appropriate selector for you based on the page structure, greatly simplifying the tedious process of determining the selector.

3. Intelligent Reply to crawler problems

In the crawling work, we will inevitably encounter various problems and challenges. And x-crawl's AI assistance can provide you with intelligent answers and suggestions. Whether it is about crawling strategy, anti-crawling skills or data processing, you can ask AI questions, AI will provide you with professional answers and suggestions based on its strong learning and reasoning ability to help you better complete the crawling task.

4. User-defined AI function

In order to meet the individual needs of different users, x-crawl also provides the ability to customize the AI. This means that you can customize and optimize the AI according to your needs, making it better suited to your crawler work. Whether it's adjusting the AI's analysis strategy, optimizing the generation algorithm of the selector, or adding new functional modules, you can do it with simple operations to make the AI more in line with your usage habits and workflow.

x-crawl GitHub: https://github.com/coder-hxl/x-crawl

x-crawl Documentation: https://coder-hxl.github.io/x-crawl

If you find x-crawl helpful, or if you like x-crawl, you can star x-crawl repository on GitHub. Your support is our motivation for continuous improvement! Thank you for your support!

Recommend a flexible Node.js multi-functional crawler library —— x-crawl

CoderHXL — Wed, 20 Mar 2024 09:03:40 +0000

x-crawl

x-crawl is a flexible Node.js multifunctional crawler library. Flexible usage and numerous functions can help you quickly, safely, and stably crawl pages, interfaces, and files.

If you also like x-crawl, you can give the x-crawl repository a star on GitHub to support it. Thank you for your support!

Features

🔥 Asynchronous Synchronous - Just change the mode property to toggle asynchronous or synchronous crawling mode.
⚙️Multiple uses - Supports crawling dynamic pages, static pages, interface data, files and polling operations.
⚒️ Control page - Crawling dynamic pages supports automated operations, keyboard input, event operations, etc.
🖋️ Flexible writing style - The same crawling API can be adapted to multiple configurations, and each configuration method is very unique.
⏱️ Interval Crawling - No interval, fixed interval and random interval to generate or avoid high concurrent crawling.
🔄 Failed Retry - Avoid crawling failure due to short-term problems, and customize the number of retries.
➡️ Proxy Rotation - Auto-rotate proxies with failure retry, custom error times and HTTP status codes.
👀 Device Fingerprinting - Zero configuration or custom configuration, avoid fingerprinting to identify and track us from different locations.
🚀 Priority Queue - According to the priority of a single crawling target, it can be crawled ahead of other targets.
🧾 crawl log - Logs the crawl and uses colored string reminders at the terminal.
🦾 TypeScript - Own types, implement complete types through generics.

Example

Take the automatic acquisition of some photos of experiences and homes around the world every day as an example:

// 1. Import module ES/CJS
import xCrawl from 'x-crawl'

// 2. Create a crawler instance
const myXCrawl = xCrawl({ maxRetry: 3, intervalTime: { max: 2000, min: 1000 } })

// 3. Set the crawling task
/*
  Call the startPolling API to start the polling function,
  and the callback function will be called every other day
*/
myXCrawl.startPolling({ d: 1 }, async (count, stopPolling) => {
  // Call the crawlPage API to crawl the page
  const pageResults = await myXCrawl.crawlPage({
    targets: [
      'https://www.airbnb.cn/s/*/experiences',
      'https://www.airbnb.cn/s/plus_homes'
    ],
    viewport: { width: 1920, height: 1080 }
  })

  // Obtain the image URL by traversing the crawled page results
  const imgUrls = []
  for (const item of pageResults) {
    const { id } = item
    const { page } = item.data
    const elSelector = id === 1 ? '.i9cqrtb' : '.c4mnd7m'

    // wait for the page element to appear
    await page.waitForSelector(elSelector)

    // Get the URL of the page image
    const urls = await page.$$eval(`${elSelector} picture img`, (imgEls) =>
      imgEls.map((item) => item.src)
    )
    imgUrls.push(...urls.slice(0, 6))

    // close the page
    page.close()
  }

  // Call crawlFile API to crawl pictures
  await myXCrawl.crawlFile({ targets: imgUrls, storeDirs: './upload' })
})

running result:

Note: Please do not crawl randomly, you can check the robots.txt protocol before crawling. The class name of the website may change, this is just to demonstrate how to use x-crawl.

More content can be viewed: https://github.com/coder-hxl/x-crawl

Use x-crawl to help you fetch data

CoderHXL — Mon, 10 Apr 2023 01:43:36 +0000

x-crawl

x-crawl is a flexible Node.js multifunctional crawler library. Flexible usage and numerous functions can help you quickly, safely, and stably crawl pages, interfaces, and files.

If you also like x-crawl, you can give x-crawl repository a star to support it, thank you for your support!

Features

🔥 Asynchronous Synchronous - Just change the mode property to toggle asynchronous or synchronous crawling mode.
⚙️ Multiple purposes - It can crawl pages, crawl interfaces, crawl files and poll crawls to meet the needs of various scenarios.
🖋️ Flexible writing style - The same crawling API can be adapted to multiple configurations, and each configuration method is very unique.
⏱️ Interval Crawling - No interval, fixed interval and random interval to generate or avoid high concurrent crawling.
🔄 Failed Retry - Avoid crawling failure due to short-term problems, and customize the number of retries.
➡️ Proxy Rotation - Auto-rotate proxies with failure retry, custom error times and HTTP status codes.
👀 Device Fingerprinting - Zero configuration or custom configuration, avoid fingerprinting to identify and track us from different locations.
🚀 Priority Queue - According to the priority of a single crawling target, it can be crawled ahead of other targets.
☁️ Crawl SPA - Crawl SPA (Single Page Application) to generate pre-rendered content (aka "SSR" (Server Side Rendering)).
⚒️ Control Page - You can submit form, keyboard input, event operation, generate screenshots of the page, etc.
🧾 Capture Record - Capture and record crawling, and use colored strings to remind in the terminal.
🦾 TypeScript - Own types, implement complete types through generics.

Example

Take the automatic acquisition of some photos of experiences and homes around the world every day as an example:

// 1.Import module ES/CJS
import xCrawl from 'x-crawl'

// 2.Create a crawler instance
const myXCrawl = xCrawl({maxRetry: 3,intervalTime: { max: 3000, min: 2000 }})

// 3.Set the crawling task
/*
  Call the startPolling API to start the polling function,
  and the callback function will be called every other day
*/
myXCrawl.startPolling({ d: 1 }, async (count, stopPolling) => {
  // Call crawlPage API to crawl Page
  const res = await myXCrawl.crawlPage({
    targets: [
      'https://www.airbnb.cn/s/experiences',
      'https://www.airbnb.cn/s/plus_homes'
    ],
    viewport: { width: 1920, height: 1080 }
  })

  // Store the image URL to targets
  const targets = []
  const elSelectorMap = ['._fig15y', '._aov0j6']
  for (const item of res) {
    const { id } = item
    const { page } = item.data

    // Wait for the page to load
    await new Promise((r) => setTimeout(r, 300))

    // Gets the URL of the page image
    const urls = await page!.$$eval(
      `${elSelectorMap[id - 1]} img`,
      (imgEls) => {
        return imgEls.map((item) => item.src)
      }
    )
    targets.push(...urls)

    // Close page
    page.close()
  }

  // Call the crawlFile API to crawl pictures
  myXCrawl.crawlFile({ targets, storeDir: './upload' })
})

running result:

Note: Do not crawl at will, you can check the robots.txt protocol before crawling. This is just to demonstrate how to use x-crawl.

For more detailed documentation, please check: https://github.com/coder-hxl/x-crawl