Scrapy, a fast high-level web crawling & scraping framework for Python.

Python ProxyPool for web spider

A Powerful Spider(Web Crawler) System in Python.

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

Incredibly fast crawler designed for OSINT.

Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

基于搜狗微信搜索的微信公众号爬虫接口

Redis-based components for Scrapy.

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:

A community-driven way to read and chat with AI bots - powered by chatGPT.

Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭

Every web site provides APIs.

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

新浪微博爬虫，用python爬取新浪微博数据，并下载微博图片和微博视频