Top 23 Python Bigquery Projects

• Redash

  38 25,057 9.5 Python

  Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.

  

Project mention: Redash: Connect to data source, easily visualize, dashboard and share your data | news.ycombinator.com | 2024-03-20

• airbyte

  140 14,296 10.0 Python

  The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

  

Project mention: How to Build a Chat App with Your Postgres Data using Agent Cloud | dev.to | 2024-05-13

AgentCloud uses Airbyte to build data pipelines, which allow us to split, chunk, and embed data from over 300 data sources, including Postgres.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• sqlglot

  56 5,679 9.9 Python

  Python SQL Parser and Transpiler

Project mention: The Future of MySQL is PostgreSQL: an extension for the MySQL wire protocol | news.ycombinator.com | 2024-04-26

This is probably referring to "zero changes to your driver code" and not "zero changes to the SQL you send over this driver".

Translating between SQL dialects is notoriously hard and attempts to translate [1] are working in 95% of cases. But the last 5% would require 5x amount of work. That's because "SQL dialect" also includes weird edge cases of type inference of things like COALESCE(5, FALSE) and emulation of system catalogs (pg_catalog, information_schema).

[1] https://github.com/tobymao/sqlglot

• ibis

  23 4,304 10.0 Python

  the portable Python dataframe library

  

Project mention: Show HN: Hashquery, a Python library for defining reusable analysis | news.ycombinator.com | 2024-04-23

I really don't understand the appeal of dbt vs a proper programming language. The templating approach leads to massive spaghetti. I look forward to trying out something like Ibis [0]

0: https://ibis-project.org/

• ethereum-etl

  3 2,836 6.9 Python

  Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

  

Project mention: Blockchain transactions decoding: making wallet activity understandable | dev.to | 2023-10-27

Event is a log entity which EVM smart contracts can emit during transaction execution. Events are very good at signalling that an some action has taken place on-chain. Applications can subscribe and listen to events to trigger some off-chain logic or they can index, transform and store events in some off-chain storage (look at The Graph protocol or Ethereum ETL).

• professional-services

  8 2,738 9.1 Python

  Common solutions and tools developed by Google Cloud's Professional Services team. This repository and its contents are not an officially supported Google product.

  

• ingestr

  4 2,341 9.0 Python

  ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

  

Project mention: FLaNK 04 March 2024 | dev.to | 2024-03-04

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• swirl-search

  32 1,552 9.8 Python

  Swirl is an open-source search platform that uses AI to search multiple content and data sources simultaneously and return AI-ranked results. And provides summaries of your answers from searches using LLMs. It's a one-click, easy-to-use Retrieval Augmented Generation (RAG) Solution.

  

Project mention: GitHub - swirlai/swirl-search: Swirl is an open-source search platform that uses AI to search multiple content and data sources simultaneously, finds the best results using a reader LLM, then prompts Generative AI, enabling you to get answers based on your data. | /r/programming | 2023-12-05

• jupysql

  8 611 9.1 Python

  Better SQL in Jupyter. 📊

  

Project mention: Show HN: JupySQL – a SQL client for Jupyter (ipython-SQL successor) | news.ycombinator.com | 2023-12-06

Hey, HN community!

We're stoked to launch JupySQL today! JupySQL is an open-source library that brings a modern SQL experience to Jupyter. JupySQL is compatible with all major databases, such as Snowflake, Redshift, PostgreSQL, MySQL, MariaDB, DuckDB, SQL Server, Clickhouse, Trino, and more!

To get started, check out our tutorial: https://jupysql.ploomber.io/en/latest/quick-start.html

SQL is the defacto language for data analysis; however, analysis often requires a mix of SQL and Python. JupySQL bridges this gap, allowing users to execute SQL queries seamlessly in Jupyter and continue their analysis in Python. Add %%sql to the top of your cell and start writing SQL.

Here are some of JupySQL's main features:

- Syntax highlighting

• BigQuery-Python

  1 451 1.8 Python

  Simple Python client for interacting with Google BigQuery.

• python-bigquery-pandas

  1 422 8.1 Python

  Google BigQuery connector for pandas

  

• pypinfo

  1 394 5.4 Python

  Easily view PyPI download statistics via Google's BigQuery.

• astro-sdk

  7 323 8.5 Python

  Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

  

Project mention: Orchestration: Thoughts on Dagster, Airflow and Prefect? | /r/dataengineering | 2023-06-01

Have you tried the Astro SDK? https://github.com/astronomer/astro-sdk

• bigquery-schema-generator

  1 232 6.3 Python

  Generates the BigQuery schema from newline-delimited JSON or CSV data records.

• dbt-coves

  1 209 9.2 Python

  CLI tool for dbt users to simplify creation of staging models (yml and sql) files

  

• CueObserve

  6 208 0.0 Python

  Timeseries Anomaly detection and Root Cause Analysis on data in SQL data warehouses and databases

  

• dbt-ml-preprocessing

  2 176 3.7 Python

  A SQL port of python's scikit-learn preprocessing module, provided as cross-database dbt macros.

  

• premier-league

  8 154 9.4 Python

  A Data Engineering project. Repository for backend infrastructure and Streamlit app files for a Premier League Dashboard.

Project mention: Google Cloud Portfolio Projects? | /r/googlecloud | 2023-12-09

I have a data engineering project that uses BigQuery, Cloud Run, Compute Engine, Cloud SQL, Artifact Registry, Firestore, and Datastream.

• dataproc-templates

  1 112 8.7 Python

  Dataproc templates and pipelines for solving simple in-cloud data tasks

  

• bigquery_fdw

  2 89 3.9 Python

  BigQuery Foreign Data Wrapper for PostgreSQL

• prism

  7 79 8.9 Python

  Prism is the easiest way to develop, orchestrate, and execute data pipelines in Python. (by runprism)

  

Project mention: Prism: the easiest way to create robust data workflows. Accessible via CLI | /r/coolgithubprojects | 2023-09-21

• iris3

  2 68 8.5 Python

  An upgraded and improved version of the Iris automatic GCP-labeling project

  

• dbd

  4 56 0.0 Python

  dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Python Bigquery related posts

• This Week In Python

  5 projects | dev.to | 17 Mar 2024

• Show HN: I built an open-source data copy tool called ingestr

  3 projects | news.ycombinator.com | 27 Feb 2024

• Ingestr: CLI tool to copy data between any databases with a single command

  1 project | news.ycombinator.com | 27 Feb 2024

• JupySQL: Connecting to a SQL database from Jupyter

  1 project | /r/SQL | 9 Sep 2023

• GitHub - ploomber/jupysql: Better SQL in Jupyter. 📊

  1 project | /r/coolgithubprojects | 6 Sep 2023

• SQL CTE's in Jupyter notebooks, DuckDB integration and more

  1 project | /r/Jupyter | 2 Aug 2023

• TL;DR incorporate SQL functionality within Jupyter, access to modern data processing DBs (like DuckDB), polars and data exploration through plotting easier with JupySQL.

  1 project | /r/coolgithubprojects | 2 Aug 2023
• A note from our sponsor - InfluxDB
  www.influxdata.com | 20 May 2024

  Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com

Do not miss the trending Python projects with our weekly report!