Completed my first Data Engineering project with Kafka, Spark, GCP, Airflow, dbt, Terraform, Docker and more!

• streamify

  4 474 0.0 Python

  A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

I've documented on Git itself. It's slightly more focused on the setup part. But you can still get an idea on the data flow.

• eventsim

  1 481 0.0 Scala

  Event data simulator. Generates a stream of pseudo-random events from a set of users, designed to simulate web traffic.

  

Eventsim is a program that generates event data to replicate page requests for a fake music web site. The results look like real use data, but are totally fake. The docker image is borrowed from viirya's fork of it, as the original project has gone without maintenance for a few years now.

• Scout Monitoring

  www.scoutapm.com featured

  Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.

  

• eventsim

  3 60 0.0 Scala

  Event data simulator. Generates a stream of pseudo-random events from a set of users, designed to simulate web traffic. (by viirya)

Eventsim is a program that generates event data to replicate page requests for a fake music web site. The results look like real use data, but are totally fake. The docker image is borrowed from viirya's fork of it, as the original project has gone without maintenance for a few years now.

• terraform

  507 41,419 9.9 Go

  Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.

  

Infrastructure as Code software - Terraform

• CPython

  1,322 60,139 10.0 Python

  The Python programming language

  

Language - Python

• ApacheKafka

  104 29 0.0

  A curated re-sources list for awesome Apache Kafka

Stream Processing - Kafka, Spark Streaming

• Docker Compose

  391 32,644 9.6 Go

  Define and run multi-container applications with Docker

  

Containerization - Docker, Docker Compose

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• nodejs-bigquery

  43 460 8.0 TypeScript

  Node.js client for Google Cloud BigQuery: A fast, economical and fully-managed enterprise data warehouse for large-scale data analytics.

  

Data Warehouse - BigQuery

• Airflow

  170 34,877 10.0 Python

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  

Orchestration - Airflow

• spark-bigquery-connector

  2 352 8.9 Java

  BigQuery data source for Apache Spark: Read data from BigQuery into DataFrames, write DataFrames into BigQuery tables.

  

• corp

  12 413 4.6

  Assets related to the operation of Fishtown Analytics.

  

Just a slight critique, but I noticed some of the dbt models are a bit hard to read. Especially your dim_users SCD2 model, which uses lots of nested subqueries and multiple columns on the same line. You may want to refer to this style guide from dbt Labs. I find CTEs are a lot easier to parse and read.

Suggest a related project