Stream Processing

Open-source projects categorized as Stream Processing
Edit details
Language: + Rust + C++ + Go + Python + Java + C + TypeScript
Topics: Kafka Streaming streaming-data Python Go
Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured

Top 23 Stream Processing Open-Source Projects

  • mediapipe

    Cross-platform, customizable ML solutions for live and streaming media.

    • Project mention: Mediapipe openpose Controlnet model for SD | /r/localdiffusion | 2023-11-15

    mediapipe/docs/solutions/pose.md at master · google/mediapipe · GitHub

  • vector

    A high-performance observability data pipeline.

    • Project mention: What is a low/reasonable cost solution for service log storage and querying? | news.ycombinator.com | 2024-05-05

    I am thinking about using https://vector.dev/ but would also love opinions on the best deal for lower or reasonable cost storage/querying of logs. Thanks!

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • awesome-bigdata

    A curated list of awesome big data frameworks, ressources and other awesomeness.

    • Project mention: Good coding groups for black women? | news.ycombinator.com | 2024-01-13

  • redpanda

    Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!

    • Project mention: Using Redpanda with OpenTelemetry and Grafana for real-time event monitoring | dev.to | 2024-05-04

    To learn more about Redpanda and stay up-to-date, see Redpanda's source codes available on GitHub and join the Redpanda Community on Slack with fellow developers and data engineers.

  • awesome-system-design

    A curated list of awesome System Design (A.K.A. Distributed Systems) resources.

    • Project mention: Ask HN: Resources to learn boring architecture for a small startup? | news.ycombinator.com | 2023-12-25

    https://github.com/madd86/awesome-system-design

  • Benthos

    Fancy stream processing made operationally mundane

    • Project mention: Benthos – Fancy stream processing made operationally mundane | news.ycombinator.com | 2024-05-15

  • watermill

    Building event-driven applications the easy way in Go.

    • Project mention: Microservices communication | /r/golang | 2023-12-09

    I’ve successfully worked on projects using an asynchronous event-driven way of connecting services. I really like the decoupling of business logic and the events triggering it. I highly recommend https://github.com/ThreeDotsLabs/watermill to be more flexible when it comes to choosing the actual technology driving the async patter. It might be NATS today but requirements might change and you need to change. Watermill prepares you for this.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo

  • Faust

    Python Stream Processing

    • Project mention: Faust VS quix-streams - a user suggested alternative | libhunt.com/r/faust | 2023-12-07

  • risingwave

    SQL stream processing, analytics, and management. We decouple storage and compute to offer instant failover, dynamic scaling, speedy bootstrapping, and efficient joins.

    • Project mention: Proton, a fast and lightweight alternative to Apache Flink | news.ycombinator.com | 2024-01-30

    How does this compare to RisingWave and Materialize?

    https://github.com/risingwavelabs/risingwave

  • Hazelcast

    Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.

    • Project mention: Does anyone know any good java implementations for distributed key-value store? | /r/ExperiencedDevs | 2023-06-08

    You're probably looking for Hazelcast here. Note that it does much more than just a distributed k/v, but it will get you where you need to go.

  • ksql

    The database purpose-built for stream processing applications.

  • materialize

    The data warehouse for operational workloads. (by MaterializeInc)

    • Project mention: The Notifier Pattern for Applications That Use Postgres | news.ycombinator.com | 2024-05-14

    Those updates are not retroactive. They apply on a go forward basis. Each day's changes become Apache 2.0 licensed on that day four years in the future.

    For example, v0.28 was released on October 18, 2022, and becomes Apache 2.0 licensed four years after that date (i.e., 2.5 years from today), on October 18, 2026.

    [0]: https://github.com/MaterializeInc/materialize/blob/76cb6647d...

  • fluent-bit

    Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows

    • Project mention: Fluent Bit – An End to End Observability Pipeline | news.ycombinator.com | 2024-05-21

  • hudi

    Upserts, Deletes And Incremental Processing on Big Data.

    • Project mention: Getting Started with Flink SQL, Apache Iceberg and DynamoDB Catalog | dev.to | 2023-12-18

    Apache Iceberg is one of the three types of lakehouse, the other two are Apache Hudi and Delta Lake.

  • river

    🌊 Online machine learning in Python

    • Project mention: River: Online Machine Learning in Python | news.ycombinator.com | 2024-05-12

  • danfojs

    Danfo.js is an open source, JavaScript library providing high performance, intuitive, and easy to use data structures for manipulating and processing structured data.

  • arroyo

    Distributed stream processing engine in Rust

    • Project mention: FLaNK AI Weekly 18 March 2024 | dev.to | 2024-03-18

  • dpark

    Python clone of Spark, a MapReduce alike framework in Python

  • fluvio

    Lean and mean distributed stream processing system written in rust and web assembly.

    • Project mention: Ask HN: WebSocket Relay? | news.ycombinator.com | 2024-02-27

  • PipelineDB

    High-performance time-series aggregation for PostgreSQL

    • Project mention: PostgreSQL Is Enough | news.ycombinator.com | 2024-02-06

  • awesome-streaming

    a curated list of awesome streaming frameworks, applications, etc

  • Memgraph

    Open-source graph database, tuned for dynamic analytics environments. Easy to adopt, scale and own.

    • Project mention: Ask HN: Who is hiring? (March 2024) | news.ycombinator.com | 2024-03-01

    Memgraph | Staff C++ Database Engineer | REMOTE (Central/Western Europe, LatAm, or North America) https://memgraph.com/

    Memgraph is a Seed stage, open source graph database vendor. Graph DBs are a great solution for GenAI, logistics, cybersecurity and fintech so we are looking to grow aggressively this year.

    We're looking for a staff-level engineer to set technical direction, mentor junior team members, and solve some very difficult problems.

    Either DM me (the hiring manager) or apply here: https://join.com/companies/memgraph/10684850-staff-software-...

  • peerdb

    Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

    • Project mention: PeerDB Streams – Simple, Native Postgres Change Data Capture | news.ycombinator.com | 2024-05-06

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Stream Processing related posts

  • Benthos – Fancy stream processing made operationally mundane

    1 project | news.ycombinator.com | 15 May 2024

  • Show HN: Streaming DataFrames–a Pandas-like syntax for real-time data

    1 project | news.ycombinator.com | 23 Apr 2024

  • Building a streaming SQL engine with Arrow and DataFusion

    1 project | news.ycombinator.com | 18 Mar 2024

  • FLaNK AI Weekly 18 March 2024

    39 projects | dev.to | 18 Mar 2024

  • Proton, a fast and lightweight alternative to Apache Flink

    7 projects | news.ycombinator.com | 30 Jan 2024

  • Airflow VS quix-streams - a user suggested alternative

    2 projects | 7 Dec 2023

  • Apache Pulsar VS quix-streams - a user suggested alternative

    2 projects | 7 Dec 2023
  • A note from our sponsor - SaaSHub
    www.saashub.com | 22 May 2024
    SaaSHub helps you find the best software and product alternatives Learn more →

Index

What are some of the best open-source Stream Processing projects? This list will help you:

Project Stars
1 mediapipe 25,688
2 vector 16,672
3 awesome-bigdata 12,845
4 redpanda 8,939
5 awesome-system-design 8,524
6 Benthos 7,686
7 watermill 6,800
8 Faust 6,675
9 risingwave 6,394
10 Hazelcast 5,920
11 ksql 5,830
12 materialize 5,598
13 fluent-bit 5,390
14 hudi 5,114
15 river 4,810
16 danfojs 4,667
17 arroyo 3,340
18 dpark 2,691
19 fluvio 2,683
20 PipelineDB 2,615
21 awesome-streaming 2,572
22 Memgraph 2,116
23 peerdb 1,842

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com