Top 23 Python ETL Projects

• Airflow

  170 34,705 10.0 Python

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  

Project mention: AI Strategy Guide: How to Scale AI Across Your Business | dev.to | 2024-05-11

Level 1 of MLOps is when you've put each lifecycle stage and their intefaces in an automated pipeline. The pipeline could be a python or bash script, or it could be a directed acyclic graph run by some orchestration framework like Airflow, dagster or one of the cloud-provider offerings. AI- or data-specific platforms like MLflow, ClearML and dvc also feature pipeline capabilities.

• airbyte

  140 14,296 10.0 Python

  The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

  

Project mention: How to Build a Chat App with Your Postgres Data using Agent Cloud | dev.to | 2024-05-13

AgentCloud uses Airbyte to build data pipelines, which allow us to split, chunk, and embed data from over 300 data sources, including Postgres.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• dagster

  47 10,382 10.0 Python

  An orchestration platform for the development, production, and observation of data assets.

  

Project mention: AI Strategy Guide: How to Scale AI Across Your Business | dev.to | 2024-05-11

Level 1 of MLOps is when you've put each lifecycle stage and their intefaces in an automated pipeline. The pipeline could be a python or bash script, or it could be a directed acyclic graph run by some orchestration framework like Airflow, dagster or one of the cloud-provider offerings. AI- or data-specific platforms like MLflow, ClearML and dvc also feature pipeline capabilities.

• Mage

  77 7,171 9.9 Python

  🧙 The modern replacement for Airflow. Mage is an open-source data pipeline tool for transforming and integrating data. https://github.com/mage-ai/mage-ai

  

Project mention: FLaNK AI-April 22, 2024 | dev.to | 2024-04-22

• AWS Data Wrangler

  9 3,816 9.4 Python

  pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

  

Project mention: Read files from s3 using Pandas/s3fs or AWS Data Wrangler? | /r/dataengineering | 2023-12-06

I had no problem with awswrangler (https://github.com/aws/aws-sdk-pandas) and it supports reading and writing partitions which was really helpful and a few other optimizations that made it a great tool

• ethereum-etl

  3 2,836 6.9 Python

  Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

  

Project mention: Blockchain transactions decoding: making wallet activity understandable | dev.to | 2023-10-27

Event is a log entity which EVM smart contracts can emit during transaction execution. Events are very good at signalling that an some action has taken place on-chain. Applications can subscribe and listen to events to trigger some off-chain logic or they can index, transform and store events in some off-chain storage (look at The Graph protocol or Ethereum ETL).

• mara-pipelines

  3 2,056 6.0 Python

  A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

  

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• pyspark-example-project

  1 1,370 0.0 Python

  Implementing best practices for PySpark ETL jobs and applications.

• sqlmesh

  12 1,334 9.9 Python

  Efficient data transformation and modeling framework that is backwards compatible with dbt.

  

Project mention: Launch HN: Serra (YC S23) – Open-source, Python-based dbt alternative | news.ycombinator.com | 2023-08-14

There is also sqlmesh (https://sqlmesh.com/). Pretty new as well. It introduces some interesting concepts. For smaller dbt projects it could be a drop-in replacement as it allows importing dbt projects.

• pgsync

  1 1,071 7.5 Python

  Postgres to Elasticsearch/OpenSearch sync (by toluaina)

• NeumAI

  2 788 8.7 Python

  Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.

  

Project mention: Show HN: Neum AI – Open-source large-scale RAG framework | news.ycombinator.com | 2023-11-21

Interesting to see that the semantic chunking in the tools library is a wrapper around GPT-4. Asks GPT for the python code and executes it: https://github.com/NeumTry/NeumAI/blob/main/neumai-tools/neu...

• eland

  4 617 8.5 Python

  Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch

  

• baby-names-analysis

  1 563 0.0 Python

  Data ETL & Analysis on the dataset 'Baby Names from Social Security Card Applications - National Data'.

  

• redun

  4 489 7.5 Python

  Yet another redundant workflow engine

  

Project mention: Redun: Yet another redundant workflow engine | news.ycombinator.com | 2023-08-11

• versatile-data-kit

  52 411 9.7 Python

  One framework to develop, deploy and operate data workflows with Python and SQL.

  

• bitcoin-etl

  1 388 0.0 Python

  ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

  

• ethereum-etl-airflow

  7 386 9.0 Python

  Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethereum-smart-contract-into-bigquery-in-8-mins-bab5db1fdeee

  

Project mention: ethereum-etl-airflow: NEW Data - star count:358.0 | /r/algoprojects | 2023-07-10

• astro-sdk

  7 323 8.5 Python

  Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

  

Project mention: Orchestration: Thoughts on Dagster, Airflow and Prefect? | /r/dataengineering | 2023-06-01

Have you tried the Astro SDK? https://github.com/astronomer/astro-sdk

• paperetl

  12 319 6.3 Python

  📄 ⚙️ ETL processes for medical and scientific papers

  

Project mention: Show HN: Open-source Rule-based PDF parser for RAG | news.ycombinator.com | 2024-01-23

• recap

  2 305 8.7 Python

  Work with your web service, database, and streaming schemas in a single format.

  

Project mention: Recap: A python library for describing database tables and serialization formats with minimal type coercion. | /r/dataengineering | 2023-07-12

The Github Repo: https://github.com/recap-build/recap

• usaspending-api

  2 283 9.9 Python

  Server application to serve U.S. federal spending data via a RESTful API

  

• trex

  3 238 6.6 Python

  Enforce structured output from LLMs 100% of the time (by automorphic-ai)

  

Project mention: Show HN: Generate JSON mock data for testing/initial app development | news.ycombinator.com | 2023-10-03

A friend of mine built a tool called Trex that you might find helpful, check it out here: https://github.com/automorphic-ai/trex

It's very consistent at generating templated data.

• dbt-coves

  1 210 9.2 Python

  CLI tool for dbt users to simplify creation of staging models (yml and sql) files

  

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Python ETL related posts

• Show HN: Open-source Rule-based PDF parser for RAG

  9 projects | news.ycombinator.com | 23 Jan 2024

• Prism: the easiest way to create robust data workflows. Accessible via CLI

  1 project | /r/coolgithubprojects | 21 Sep 2023

• Show HN: Prism – a framework for creating robust data science workflows

  1 project | news.ycombinator.com | 1 Sep 2023

• Show HN: Prism – Data Orchestration in Python

  1 project | news.ycombinator.com | 28 Jul 2023

• Introducing Prism: A Novel, Open-Source Data Orchestration Software. Feedback needed!

  2 projects | /r/EntrepreneurRideAlong | 27 Jul 2023

• Prism - a lightweight, yet powerful data orchestration platform in Python. Accessible via CLI

  1 project | /r/coolgithubprojects | 27 Jul 2023

• Intelligently transform unstructured to structured output (JSON, Regex, CFG)

  1 project | /r/ETL | 18 Jul 2023
• A note from our sponsor - SaaSHub
  www.saashub.com | 22 May 2024

  SaaSHub helps you find the best software and product alternatives Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com

Do not miss the trending Python projects with our weekly report!