A collection of awesome web crawler,spider and resources in different languages. Contents. Python; Java; C#; JavaScript; PHP; C++; C; Ruby; Rust ...

A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs.

Scrapy is a BSD-licensed fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. Gitignore · License · Issues 445 · Pull requests 219

Crawl4AI simplifies asynchronous web crawling and data extraction, making it accessible for large language models (LLMs) and AI applications. Issues 65 · README.sync.md · README.md · Main.py

A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Issues 117 · CHANGELOG.md · README.md · Package.json

Axtarish Crawler is a tool used to discover target URLs, select the relevant content, and have it delivered in bulk. It crawls websites in real-time and at scale ...

Browsertrix Crawler is a standalone browser-based high-fidelity crawling system, designed to run a complex, customizable browser-based crawl in a single Docker ... README.md · Issues 94 · CHANGES.md · Docker-compose.yml

A web crawler and scraper, building blocks for data curation workloads. Getting Started The simplest way to get started is to use the Spider Cloud hosted ... Pull requests 0 · Issues 3 · Discussions

Backend part of the Axtarish Crawler application. The Axtarish Crawler app takes an input from the user such as a link maximum number of pages and depth.

Features · Fast And fully configurable web crawling · Standard and Headless mode · JavaScript parsing / crawling · Customizable automatic form filling · Scope ... Releases 11 · README.md · Discussions · Pull requests 8