A collection of awesome web crawler,spider and resources in different languages. Contents. Python; Java; C#; JavaScript; PHP; C++; C; Ruby; Rust ...

A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs.

Crawl4AI simplifies asynchronous web crawling and data extraction, making it accessible for large language models (LLMs) and AI applications. Issues 69 · README.md · Dockerfile · README.sync.md

Scrapy is a BSD-licensed fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. Gitignore · License · Issues 450 · Pull requests 218

A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Issues 118 · CHANGELOG.md · README.md · Discussions

Axtarish Crawler is a tool used to discover target URLs, select the relevant content, and have it delivered in bulk. It crawls websites in real-time and at scale ...

Browsertrix Crawler is a standalone browser-based high-fidelity crawling system, designed to run a complex, customizable browser-based crawl in a single Docker ... README.md · Issues 94 · CHANGES.md · Docker-compose.yml

A web crawler and scraper, building blocks for data curation workloads. Getting Started The simplest way to get started is to use the Spider Cloud hosted ... Pull requests 0 · Issues 3 · Discussions

Backend part of the Axtarish Crawler application. The Axtarish Crawler app takes an input from the user such as a link maximum number of pages and depth.

Features · Fast And fully configurable web crawling · Standard and Headless mode · JavaScript parsing / crawling · Customizable automatic form filling · Scope ... Releases 11 · README.md · Pull requests 8 · Discussions