I used multiprocessing and multithreading at the same time to drop the execution time of my code from 155+ seconds to just over 2+ seconds

This page summarizes the projects mentioned and recommended in the original post on /r/Python
Rust Python Database SQL Dataframe

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured

  • connector-x

    Fastest library to load data from DB to DataFrames in Rust and Python

    • There's packages like connector-x and polars that do a lot of what you're mentioning out of the box. I used these two to massively speed up an SQLalchemy + Pandas based ETL in the past as well.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • How moving from Pandas to Polars made me write better code without writing better code

    2 projects | dev.to | 5 Mar 2024

  • Polars

    11 projects | news.ycombinator.com | 8 Jan 2024

  • Pure Python Distributed SQL Engine

    9 projects | news.ycombinator.com | 30 Dec 2022

  • Working with more than 10gb csv

    3 projects | /r/datascience | 5 Oct 2022

  • A Critique of SQL, 40 Years Later

    6 projects | news.ycombinator.com | 24 Aug 2022