PeerDB Streams – Simple, Native Postgres Change Data Capture

• peerdb

  7 1,842 9.9 Go

  Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

  

Hello HN, I am Sai Srirampur, one of the Co-founders of PeerDB. (https://github.com/PeerDB-io/peerdb)

We spent the past 7 months building a solid experience to replicate data from Postgres to Data Warehouses. Now, we want to expand to Queues. We are excited to announce PeerDB Streams to provide a simple and native way to replicate changes as they happen in Postgres to Queues (Kafka, Redpanda, Google PubSub, etc). We use Postgres logical decoding to enable Change Data Capture (CDC).

We chose queues as many users found that existing tools are complex. Debezium is the most used tool for this use-case. It has large production usage. However, a common pain point among our users is that it has a significant learning curve taking months to productionize.

A few issues are: a)Interacting through a command line interface, understanding the various settings, and learning best practices for running it in production is not trivial. Debezium UI, released to address usability concerns [1], is still in an incubating state [2]. Additionally, reading Debezium resources to get started can be overwhelming [3].

• examples

  2 3 5.2 Lua

  Example scripts for PeerDB (by PeerDB-io)

  

Yes, that is doable through a very simple Lua-based row-level transformation. :) Here is an example transformation that adds an additional field: https://github.com/PeerDB-io/examples/blob/main/usdcad.lua . More details on row-level transformations are planned to be released later this week. :)

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project