Beautiful Soup: We called him Tortoise because he taught us

• html5-parser

  2 671 6.3 C

  Fast C based HTML 5 parsing for python

You want a proper html 5 parser that can handle non valid documents. And the fastest one is https://github.com/kovidgoyal/html5-parser over 30x faster than html5lib

• SeleniumBase

  9 4,350 9.7 Python

  📊 Python's all-in-one framework for web crawling, scraping, testing, and reporting. Supports pytest. UC Mode provides stealth. Includes many tools.

  

In those cases you might want to check out SeleniumBase: https://seleniumbase.io/

• Scout Monitoring

  www.scoutapm.com featured

  Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.

  

• colly

  39 22,392 5.7 Go

  Elegant Scraper and Crawler Framework for Golang

  

• shot-scraper

  16 1,548 7.1 Python

  A command-line utility for taking automated screenshots of websites

Playwright for Python has really good documentation: https://playwright.dev/python/

I used it for my https://shot-scraper.datasette.io/ tool, and wrote a bit about CLI-driven scraping using that tool here: https://simonwillison.net/2022/Mar/14/scraping-web-pages-sho...

• playwright-python

  31 10,919 9.1 Python

  Python version of the Playwright testing and automation library.

  

Playwright for Python has really good documentation: https://playwright.dev/python/

I used it for my https://shot-scraper.datasette.io/ tool, and wrote a bit about CLI-driven scraping using that tool here: https://simonwillison.net/2022/Mar/14/scraping-web-pages-sho...

• soup

  4 2,135 0.0 Go

  Web Scraper in Go, similar to BeautifulSoup

> Does anyone know if there as a good equivalent for Go

Yes: https://github.com/anaskhan96/soup

It works well.

Suggest a related project