Cached Chrome Top Million Websites

• ClickHouse

  211 34,836 10.0 C++

  ClickHouse® is a real-time analytics DBMS

  

If you are interested in the research on technologies used on the Internet, I recommend playing with the "Minicrawl" dataset.

It contains data about ~7 million top websites, and for every website, it also contains: - the full content of the main page; - the verbose output of curl, containing various timing info; the HTTP headers, protocol info...

Using this dataset, you can build a service similar to https://builtwith.com/ for your research.

Data: https://clickhouse-public-datasets.s3.amazonaws.com/minicraw... (129 GB compressed, ~1 TB uncompressed).

Description: https://github.com/ClickHouse/ClickHouse/issues/18842

You can easily try it with clickhouse-local without downloading:

  $ curl https://clickhouse.com/ | sh

• crux-top-lists

  6 712 6.0 Python

  Downloadable snapshots of the Chrome Top Million Websites pulled from public CrUX data in Google BigQuery.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• hn-search

  1,662 526 2.9 TypeScript

  Hacker News Search

  

It's a tough thing to balance, but generally, bringing in someone's personal details as ammunition in an internet argument is not ok on HN (https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...). I'm not saying those are never relevant, but

[editing...]

• github-explorer

  13 133 4.3 HTML

  Everything You Always Wanted To Know About GitHub (But Were Afraid To Ask)

  

Yes, it's continuously updated.

The source code is here: https://github.com/ClickHouse/github-explorer

This shell scripts updates it: https://github.com/ClickHouse/github-explorer/blob/main/upda...

Suggest a related project