The crawler opened source by tap4.ai

Simple and powerfull all in one Telegram Bot to scrap / crawl webpages using Requests, html5lib and Beautifulsoup

SpideyX a multipurpose Web Penetration Testing tool with asynchronous concurrent performance with multiple mode and configurations.

Use browser to re-copy a web page

your friendly neighborhood web crawler

🤖 robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API

Web crawler for extracting internal site links info for SEO auditing & optimization purposes

Declarative, scriptable web robot (crawler) and scrapper

Generic Interfaces to Addressable Objects

Tegenaria is a crawler framework based on golang

Fast Crawlbase API crawling library

武汉东湖高新片区光谷&软件园二手房房价爬虫。data source: 房天下

Example to demonstrate the usage of cached queues across multiple requests.

基于Spring Boot、Scrapy 的可视化爬虫配置与管理

Useful functions for connecting to the network in the PHP based applications.

Shark (Plunder)可配置、插件化的爬虫引擎，二次开发框架。Configurable, pluginable crawler engine, secondary development framework.

An advanced web-crawler written in PHP.

Simple crawler using apache nutch and elasticsearch

Blazingly Fast, High Performant, Scalable Web Crawler Engine 💨

An Android app crawling framework, making automatic crawling mobile apps super easy! (if possible, iOS will be supported after Android version)