Top 23 Python dbt Projects

• Mage

  77 7,202 9.9 Python

  🧙 The modern replacement for Airflow. Mage is an open-source data pipeline tool for transforming and integrating data. https://github.com/mage-ai/mage-ai

  

Project mention: FLaNK AI-April 22, 2024 | dev.to | 2024-04-22

• data-diff

  20 2,899 9.4 Python

  Discontinued Compare tables within or across databases

  

Project mention: How to Check 2 SQL Tables Are the Same | news.ycombinator.com | 2023-07-26

If the issue happen a lot, there is also: https://github.com/datafold/data-diff

That is a nice tool to do it cross database as well.

I think it's based on checksum method.

• Scout Monitoring

  www.scoutapm.com featured

  Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.

  

• soda-core

  5 1,786 8.9 Python

  :zap: Data quality testing for the modern data stack (SQL, Spark, and Pandas) https://www.soda.io

  

• sqlmesh

  12 1,383 9.9 Python

  Efficient data transformation and modeling framework that is backwards compatible with dbt.

  

Project mention: Launch HN: Serra (YC S23) – Open-source, Python-based dbt alternative | news.ycombinator.com | 2023-08-14

There is also sqlmesh (https://sqlmesh.com/). Pretty new as well. It introduces some interesting concepts. For smaller dbt projects it could be a drop-in replacement as it allows importing dbt projects.

• dbt-duckdb

  1 754 9.4 Python

  dbt (http://getdbt.com) adapter for DuckDB (http://duckdb.org)

  

• streamify

  4 474 0.0 Python

  A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

• piperider

  6 471 9.5 Python

  Code review for data in dbt

  

Project mention: Show HN: PipeRider – open-source Data Impact Analysis for dbt changes | news.ycombinator.com | 2023-09-06

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• astronomer-cosmos

  8 476 9.5 Python

  Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code

  

• dbt-metabase

  1 432 8.0 Python

  dbt + Metabase integration

• airflow-dbt

  1 382 0.0 Python

  Apache Airflow integration for dbt

  

• dbt-data-reliability

  2 349 9.7 Python

  dbt package that is part of Elementary, the dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.

  

• grai-core

  6 271 9.5 Python
  

Project mention: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform | news.ycombinator.com | 2023-07-17

Elastic v2 if one is interested in such things: https://github.com/grai-io/grai-core/blob/v0.1.33/LICENSE

• recs-at-resonable-scale

  2 218 2.3 Python

  Recommendations at "Reasonable Scale": joining dataOps with recSys through dbt, Merlin and Metaflow

Project mention: When writing ML software - how do you use TDD? | /r/mlops | 2023-06-25

Good paper, and in response to that one a team from Coveo, wrote this paper on behavioral tests for recommender systems... and also this repo.

• dbt-clickhouse

  1 222 8.5 Python

  The Clickhouse plugin for dbt (data build tool)

  

• dbt-coves

  1 216 9.2 Python

  CLI tool for dbt users to simplify creation of staging models (yml and sql) files

  

• dbt-athena

  1 194 9.2 Python

  The athena adapter plugin for dbt (https://getdbt.com) (by dbt-athena)

  

• dbt-databricks

  15 190 9.5 Python

  A dbt adapter for Databricks.

  

• post-modern-stack

  2 181 2.2 Python

  Joining the modern data stack with the modern ML stack

Project mention: [Advice] MLOps Course recommendations | /r/datascience | 2023-06-24

End-to-end stuff, full-fledge stacks: https://github.com/jacopotagliabue/post-modern-stack

• dbt-ml-preprocessing

  2 176 3.7 Python

  A SQL port of python's scikit-learn preprocessing module, provided as cross-database dbt macros.

  

• dbt-coverage

  2 174 6.3 Python

  One-stop-shop for docs and test coverage of dbt projects.

  

• dbt2looker

  1 171 0.0 Python

  Generate lookml for views from dbt models

  

• dbterd

  1 171 8.5 Python

  Generate the ERD as a code from dbt artifacts

• valmi-activation

  1 130 9.3 Python

  ⚡ valmi.io reverse ETL (data activation) is the open source ( OSS ) data activation platform to load data from warehouses into Webhooks and SaaS tools like Klaviyo, Facebook Ads, Salesforce, Braze etc. Valmi.io Customer Data Platform (CDP) helps track and ingest user activity events from websites, shopify, serverside events. https://cloud.valmi.io

  

Project mention: Show HN: Valmi.io Open Source Reverse-ETL Engine | news.ycombinator.com | 2023-06-21

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Python dbt related posts

• Launch HN: Grai (YC S22) – Open-Source Data Observability Platform

  3 projects | news.ycombinator.com | 17 Jul 2023

• When writing ML software - how do you use TDD?

  2 projects | /r/mlops | 25 Jun 2023

• [Advice] MLOps Course recommendations

  3 projects | /r/datascience | 24 Jun 2023

• Run dbt projects as Apache Airflow DAGs and Task Groups with a few lines of code

  1 project | news.ycombinator.com | 1 May 2023

• Curious if anyone has adopted a stack to do raw data ingestion in Databricks?

  2 projects | /r/dataengineering | 25 Apr 2023

• Running dbt core on airflow

  1 project | /r/dataengineering | 19 Apr 2023

• dolly-v2-12b

  3 projects | /r/LocalLLM | 13 Apr 2023
• A note from our sponsor - InfluxDB
  www.influxdata.com | 2 Jun 2024

  Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com

Do not miss the trending Python projects with our weekly report!