Python warc

Open-source Python projects categorized as warc
Edit details
Topics: web-archiving Python internet-archiving wget Docker
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured

Top 10 Python warc Projects

  • ArchiveBox

    🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

    • Project mention: 38% of webpages that existed in 2013 are no longer accessible a decade later | news.ycombinator.com | 2024-05-18

    There's also https://archivebox.io which can take your bookmarks and archive them in many ways. Unfortunately back when I tried it last time it was a big buggy, I wish there was a better solution to build a nice archive of the sites I visit more often just in case.

  • conifer

    Collect and revisit web pages.

    • Project mention: YaCy, a distributed Web Search Engine, based on a peer-to-peer network | news.ycombinator.com | 2024-03-05

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • grab-site

    The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

    • Project mention: Ask HN: How can I back up an old vBulletin forum without admin access? | news.ycombinator.com | 2024-01-29

    The format you want is WARC. Even the Library of Congress uses it. There are many many WARC scrapers. I'd look at what the Internet Archive recommends. A quick search turned up this from the Archive Team and Jason Scott https://github.com/ArchiveTeam/grab-site (https://wiki.archiveteam.org/index.php/Who_We_Are) but I found that in less than 15 seconds of searching so do your own diligence.

  • ipwb

    InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS

  • WarcDB

    WarcDB: Web crawl data as SQLite databases.

    • Project mention: An Introduction to the WARC File | news.ycombinator.com | 2024-01-29

  • warcio

    Streaming WARC/ARC library for fast web archive IO

  • bitextor

    Bitextor generates translation memories from multilingual websites

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo

  • cdx_toolkit

    A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

  • forum-dl

    Scrape posts, threads from forums, news aggregators, mail archives, export to JSONL, mailbox, WARC

    • Project mention: Ask HN: How can I back up an old vBulletin forum without admin access? | news.ycombinator.com | 2024-01-29

    You can try forum-dl, a forum scraping tool I've been writing for this purpose: https://github.com/mikwielgus/forum-dl

    It's single-threaded, alpha-quality software, and still isn't compatible with many forums and themes. But it can export WARCs and may just happen to work for you.

  • warc2zim

    Command line tool to convert a file in the WARC format to a file in the ZIM format

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Python warc related posts

Index

What are some of the best open-source warc projects in Python? This list will help you:

Project Stars
1 ArchiveBox 20,023
2 conifer 1,458
3 grab-site 1,273
4 ipwb 591
5 WarcDB 384
6 warcio 350
7 bitextor 282
8 cdx_toolkit 153
9 forum-dl 61
10 warc2zim 36

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com