Data Ingestion - Build Your Own "Map Reduce"?

• mrjob

  1 2,609 0.0 Python

  Run MapReduce jobs on Hadoop or Amazon Web Services

  

Mrjob

• mmh3

  2 308 7.5 C

  Python extension for MurmurHash (MurmurHash3), a set of fast and robust hash functions.

Some notes: We don't need Sha256 and not evey base64; nothing will happen if keys will not distribute very equally. we could take MMH3; googling "python murmurhash" gives 2 interesting results; and since both use the same cpp code, let's take the one with most stars Other options would be to simply do (% NUM_SHARDS) or even shift right (however must have shards count == power of 2).

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• murmurhash

  2 42 5.0 C++

  💥 Cython bindings for MurmurHash2 (by explosion)

  

Some notes: We don't need Sha256 and not evey base64; nothing will happen if keys will not distribute very equally. we could take MMH3; googling "python murmurhash" gives 2 interesting results; and since both use the same cpp code, let's take the one with most stars Other options would be to simply do (% NUM_SHARDS) or even shift right (however must have shards count == power of 2).

• py-spy

  25 11,985 5.8 Rust

  Sampling profiler for Python programs

Q: Are we sure about it? A: we could use py-spy and see.

Suggest a related project