Best Orchestration Tool to run dbt projects?

• astronomer-cosmos

  8 471 9.4 Python

  Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code

  

Well, I would take a look at Astronomer Cosmos which is a framework to dynamically Airflow DAGs from dbt with a single operator.No need to manage connections, use true BashOperator etc. It’s dead simple and open source https://astronomer.github.io/astronomer-cosmos/ I’m going to make a video about it very soon.

• dagster

  47 10,382 10.0 Python

  An orchestration platform for the development, production, and observation of data assets.

  

Dagster seemed really cool when I looked into it as an alternative to airflow. I especially like the software defined assets and built-in lineage which I haven't seen in any other tool. However it seems it does not support RBAC which is a pretty big issue if you want a self-service type of architecture, see https://github.com/dagster-io/dagster/issues/2219. It does seem like it's available in their hosted version, but I wanted to run it myself on k8s.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• data-aware-orchestration

  1 28 10.0 Python

  Data-aware orchestration with dagster, dbt, and airbyte

Suggest a related project