Python elt

Open-source Python projects categorized as elt
Edit details
Topics: Python ETL SQL Data Analytics
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured

Top 16 Python elt Projects

  • Airflow

    Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

    • Project mention: AI Strategy Guide: How to Scale AI Across Your Business | dev.to | 2024-05-11

    Level 1 of MLOps is when you've put each lifecycle stage and their intefaces in an automated pipeline. The pipeline could be a python or bash script, or it could be a directed acyclic graph run by some orchestration framework like Airflow, dagster or one of the cloud-provider offerings. AI- or data-specific platforms like MLflow, ClearML and dvc also feature pipeline capabilities.

  • airbyte

    The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

    • Project mention: How to Build a Chat App with Your Postgres Data using Agent Cloud | dev.to | 2024-05-13

    AgentCloud uses Airbyte to build data pipelines, which allow us to split, chunk, and embed data from over 300 data sources, including Postgres.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • dbt-core

    dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

    • Project mention: Dbt | news.ycombinator.com | 2024-02-18

  • Mage

    🧙 The modern replacement for Airflow. Mage is an open-source data pipeline tool for transforming and integrating data. https://github.com/mage-ai/mage-ai

    • Project mention: FLaNK AI-April 22, 2024 | dev.to | 2024-04-22

  • dlt

    data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

    • Project mention: Ask HN: Freelancer? Seeking freelancer? (December 2023) | news.ycombinator.com | 2023-12-03

    SEEKING FREELANCER | REMOTE | GERMANY

    dltHub is looking for a freelance help in the following repos:

    - https://github.com/dlt-hub/dlt

  • meltano

    Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

    • Project mention: meltano VS cloudquery - a user suggested alternative | libhunt.com/r/meltano | 2023-06-02

  • sqlmesh

    Efficient data transformation and modeling framework that is backwards compatible with dbt.

    • Project mention: Launch HN: Serra (YC S23) – Open-source, Python-based dbt alternative | news.ycombinator.com | 2023-08-14

    There is also sqlmesh (https://sqlmesh.com/). Pretty new as well. It introduces some interesting concepts. For smaller dbt projects it could be a drop-in replacement as it allows importing dbt projects.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo

  • dbt-metabase

    dbt + Metabase integration

  • versatile-data-kit

    One framework to develop, deploy and operate data workflows with Python and SQL.

  • astro-sdk

    Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

    • Project mention: Orchestration: Thoughts on Dagster, Airflow and Prefect? | /r/dataengineering | 2023-06-01

    Have you tried the Astro SDK? https://github.com/astronomer/astro-sdk

  • dbt-coves

    CLI tool for dbt users to simplify creation of staging models (yml and sql) files

  • reddit-detective

    Play detective on Reddit: Discover political disinformation campaigns, secret influencers and more

  • sayn

    Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

  • Meltano Singer SDK

    Write 70% less code by using the SDK to build custom extractors and loaders that adhere to the Singer standard: https://sdk.meltano.com (by meltano)

  • dbd

    dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.

  • reddit_api_elt

    • Project mention: Reddit ELT Pipeline | /r/dataengineering | 2023-12-11

    Hi everyone, this is my first DE project. Baitur5/reddit_api_elt (github.com) . It is basically about a data pipeline that extracts Reddit data for a Google Data Studio report, focusing on a specific subreddit Can you guys check it out , and give some advice & tips on how to improve it or the next things I should add.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Python elt related posts

Index

What are some of the best open-source elt projects in Python? This list will help you:

Project Stars
1 Airflow 34,705
2 airbyte 14,296
3 dbt-core 8,985
4 Mage 7,171
5 dlt 1,792
6 meltano 1,617
7 sqlmesh 1,334
8 dbt-metabase 431
9 versatile-data-kit 412
10 astro-sdk 323
11 dbt-coves 210
12 reddit-detective 206
13 sayn 117
14 Meltano Singer SDK 86
15 dbd 56
16 reddit_api_elt 2

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com