Python data-quality

Open-source Python projects categorized as data-quality
Edit details
Topics: Python Data Science data-profiling data-engineering Data Validation
Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured

Top 14 Python data-quality Projects

  • ydata-profiling

    1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.

    • Project mention: FLaNK 25 December 2023 | dev.to | 2023-12-26

  • great_expectations

    Always know what to expect from your data.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • cleanlab

    The standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.

    • Project mention: [Research] Detecting Annotation Errors in Semantic Segmentation Data | /r/MachineLearning | 2023-11-05

    We have feely open-sourced our new method for improving segmentation data, published a paper on the research behind it, and released a 5-min code tutorial. You can also read more in the blog if you'd like.

  • feast

    The Open Source Feature Store for Machine Learning

    • Project mention: What's Happening with Feast? | news.ycombinator.com | 2023-12-07

  • data-diff

    Discontinued Compare tables within or across databases

    • Project mention: How to Check 2 SQL Tables Are the Same | news.ycombinator.com | 2023-07-26

    If the issue happen a lot, there is also: https://github.com/datafold/data-diff

    That is a nice tool to do it cross database as well.

    I think it's based on checksum method.

  • soda-core

    :zap: Data quality testing for the modern data stack (SQL, Spark, and Pandas) https://www.soda.io

  • cleanvision

    Automatically find issues in image datasets and practice data-centric computer vision.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo

  • piperider

    Code review for data in dbt

    • Project mention: Show HN: PipeRider – open-source Data Impact Analysis for dbt changes | news.ycombinator.com | 2023-09-06

  • Encord Active

    Open source active learning toolkit to find failure modes in your computer vision models, prioritize data to label next, and drive data curation to improve model performance.

    • Project mention: Launch HN: Encord (YC W21) – Unit testing for computer vision models | news.ycombinator.com | 2024-01-31

    We base our pricing on your user and consumption scale and would be happy to discuss this with you directly. Please feel free to explore the OS version of Active at https://github.com/encord-team/encord-active. Note that some features, such as natural language search using GPU accelerated APIs, are not included in the cloud version.

  • cuallee

    Possibly the fastest DataFrame-agnostic quality check library in town.

    • Project mention: Show HN: Snowflake Data Quality Checks in Python | news.ycombinator.com | 2024-02-11

  • swiple

    Swiple enables you to easily observe, understand, validate and improve the quality of your data

  • soda-spark

    Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes

  • panda_patrol

    • Project mention: Show HN: Data monitoring and profiling with 1 function call | news.ycombinator.com | 2023-12-13

  • data_check

    data and pipeline testing with and for SQL

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Python data-quality related posts

  • Show HN: Snowflake Data Quality Checks in Python

    1 project | news.ycombinator.com | 11 Feb 2024

  • Show HN: Data monitoring and profiling with 1 function call

    1 project | news.ycombinator.com | 13 Dec 2023

  • [Research] Detecting Annotation Errors in Semantic Segmentation Data

    1 project | /r/MachineLearning | 5 Nov 2023

  • [R] Automated Quality Assurance for Object Detection Datasets

    1 project | /r/computervision | 28 Sep 2023

  • Show HN: PipeRider – open-source Data Impact Analysis for dbt changes

    3 projects | news.ycombinator.com | 6 Sep 2023

  • [D] Is accurately estimating image quality even possible?

    3 projects | /r/MachineLearning | 22 Apr 2023

  • Looking for Unit Testing framework in Database Migration Process

    3 projects | /r/dataengineering | 23 Mar 2023
  • A note from our sponsor - InfluxDB
    www.influxdata.com | 24 May 2024
    Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Index

What are some of the best open-source data-quality projects in Python? This list will help you:

Project Stars
1 ydata-profiling 12,101
2 great_expectations 9,526
3 cleanlab 8,858
4 feast 5,303
5 data-diff 2,899
6 soda-core 1,780
7 cleanvision 931
8 piperider 469
9 Encord Active 423
10 cuallee 111
11 swiple 78
12 soda-spark 60
13 panda_patrol 21
14 data_check 4

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com