CSV or Parquet File Format

• delta

  69 6,980 9.9 Scala

  An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs (by delta-io)

  

I prefer parquet (or delta for larger datasets. CSV for very small datasets, or the ones that will be later used/edited in Excel or Googke sheets.

• Apache Arrow

  76 13,698 10.0 C++

  Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing

  

In fact I have asked Apache Github how to read select column of particular row group of a parquet file. https://github.com/apache/arrow/issues/35688

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• duckdb

  52 17,924 10.0 C++

  DuckDB is an in-process SQL OLAP Database Management System

  

The Parquet-Go library is very complex, not yet success to use it. So I ask whether DuckDB can provide API https://github.com/duckdb/duckdb/issues/7776

Suggest a related project