Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Top 23 C++ Compression Projects
-
draco
Draco is a library for compressing and decompressing 3D geometric meshes and point clouds. It is intended to improve the storage and transmission of 3D graphics.
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
caesium-image-compressor
Caesium is an image compression software that helps you store, send and share digital pictures, supporting JPG, PNG and WebP formats. You can quickly reduce the file size (and resolution, if you want) by preserving the overall quality of the image.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
compressonator
Tool suite for Texture and 3D Model Compression, Optimization and Analysis using CPUs, GPUs and APUs
-
Bareos
Bareos is a cross-network Open Source backup solution (licensed under AGPLv3) which preserves, archives, and recovers data from all major operating systems.
-
PGM-index
🏅State-of-the-art learned data structure that enables fast lookup, predecessor, range searches and updates in arrays of billions of items using orders of magnitude less space than traditional indexes
-
clp
Compressed Log Processor (CLP) is a free tool capable of compressing text logs and searching the compressed logs without decompression. (by y-scope)
-
LZHAM
Lossless data compression codec with LZMA-like ratios but 1.5x-8x faster decompression speed, C/C++
-
SIMDCompressionAndIntersection
A C++ library to compress and intersect sorted lists of integers using SIMD instructions
-
GDCM
Grassroots DICOM read-only mirror. Only for Pull Request. Please report bug at http://sf.net/p/gdcm
-
zpaqfranz
Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
I use this: https://saerasoft.com/caesium
Project mention: DwarFS – The Deduplicating Warp-Speed Advanced Read-Only File System | news.ycombinator.com | 2024-04-11https://github.com/mhx/dwarfs/blob/main/doc/mkdwarfs.md#nils...
Project mention: How do other motion capture suits manage large file sizes? | /r/gamedev | 2023-12-04Cleaned and compressed are whatever they turn out to be. Use ACL, https://github.com/nfrechette/acl
Project mention: An Empirical Evaluation of Columnar Storage Formats [pdf] | news.ycombinator.com | 2024-05-15Lance v2 looks interesting. I like their meta-data + container story. Lacking SOTA encoding schemes though.
There is also Vortex (https://github.com/fulcrum-so/vortex). That has modern encoding schemes that we want to use.
BtrBlocks (https://github.com/maxi-k/btrblocks) from the Germans is another Parquet alternative.
Nimble (formerly Alpha) is a complicated story. We worked with the Velox team for over a year to open-source and extend it. But plans got stymied by legal. This was in collaboration with Meta + CWI + Nvidia + Voltron. We decided to a separate path because Nimble code has no spec/docs. Too tightly coupled with Velox/Folly.
Given that, we are working on a new file format. We hope to share our ideas/code later this year.
I meant the Kiwix dump (https://download.kiwix.org/zim/wikipedia_en_all_nopic.zim – careful, 60GB!).
At a first glance, the Wikimedia XML dump does not look substantially different from what Kiwix/ZIM does with compressed HTML: They're both compressed (bz2 for the Wikimedia dump, zstd or LZMA for Kiwix/ZIM), and both compress multiple files at once, so inter-file redundancy should hopefully be significantly reduced.
HTML seems a bit more verbose than the Mediawiki syntax (plus the XML header for each article), but I'd be surprised if that actually accounted for a 3x difference in size.
Then again, Kiwix seems to have experimented with shared dictionary brotli compression, which supposedly yields an >2x improvement: https://github.com/openzim/libzim/issues/144
I wonder if their current zstd implementation also uses shared dictionaries. If not, that might just be the reason: If ZIM compression chunks are much smaller than the bz2 streams of the Wikimedia dumps, there would still be a lot of redundancy between chunks.
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
C++ Compression related posts
-
An Empirical Evaluation of Columnar Storage Formats [pdf]
-
GarlicOS lag
-
How do other motion capture suits manage large file sizes?
-
WikiReader
-
How are zlib, gzip and zip related?
-
BtrBlocks: Efficient Columnar Compression for Data Lakes [pdf]
-
Help! Does anyone know how to install johncena141 games on linux?
-
A note from our sponsor - InfluxDB
www.influxdata.com | 17 May 2024
Index
What are some of the best open-source Compression projects in C++? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | draco | 6,280 |
| 2 | Snappy | 6,012 |
| 3 | meshoptimizer | 5,151 |
| 4 | bolt | 2,463 |
| 5 | caesium-image-compressor | 2,266 |
| 6 | dwarfs | 1,978 |
| 7 | dandere2x | 1,404 |
| 8 | acl | 1,271 |
| 9 | compressonator | 1,226 |
| 10 | Bareos | 942 |
| 11 | PGM-index | 763 |
| 12 | clp | 723 |
| 13 | LZHAM | 679 |
| 14 | EWAHBoolArray | 436 |
| 15 | SIMDCompressionAndIntersection | 414 |
| 16 | GDCM | 337 |
| 17 | zpaqfranz | 230 |
| 18 | grok | 216 |
| 19 | CIDLib | 208 |
| 20 | ancient | 204 |
| 21 | btrblocks | 184 |
| 22 | sevenzipjbinding | 171 |
| 23 | libzim | 159 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com