Change data capture (CDC) is a data integration process that tracks row-level inserts, updates, and deletes in a source database and streams those changes to target systems in real time.
Imagine a database table with 50 million customer records, and only 200 of them changed in the last hour. Without CDC, a naive pipeline would re-extract all 50 million rows every time it runs, just to find those 200 changes. That’s slow, expensive, and puts unnecessary load on the source database – the same one prod live application is trying to serve customers from.
CDC solves this by watching for changes directly, so you move only what actually changed. This single idea underpins a huge amount of modern data engineering:
If you’re lead/architect and someone proposes “let’s just re-sync the whole table every hour”, CDC is very often the answer to “there’s a much cheaper and faster way to do this”.
What this looks like on AWS, concretely:
AWS DMS does the log-based CDC work directly against RDS/Aurora, streams changes into Kinesis, and from there the changes fan out to whichever downstream systems need to stay in sync — a data lake in S3, a warehouse in Redshift, or a live search/dashboard layer in OpenSearch.
1. Log-based CDC (the modern standard)
Reads the database’s own internal transaction log e.g., PostgreSQL’s WAL, MySQL’s binlog – the same log the database uses internally for its own recovery and replication.
This is the least invasive approach as it doesn’t touch application tables or add load to normal queries, because it’s reading a log the database already maintains for itself.
Commonly used with: PostgreSQL (WAL / logical replication), MySQL and MariaDB (binlog), Oracle (LogMiner/GoldenGate), Microsoft SQL Server (native CDC feature), MongoDB (oplog), and managed cloud variants like Amazon Aurora and Amazon RDS.
Example: A banking app running PostgreSQL enables logical replication on its WAL so Debezium can stream every committed transaction, the instant it happens, into a real-time fraud-detection pipeline with zero added query load on the live database processing customer transactions.
2. Query-based CDC
Periodically queries the source table for rows that changed since the last check, usually using a last_updated timestamp column.
Simple to implement, but has real limitations as it can’t reliably detect deletes (a deleted row just isn’t there to query anymore), and it adds repeated query load to the source database.
Commonly used with: older/legacy versions of Oracle and SQL Server, Teradata, IBM Db2, and many SaaS platforms accessed only through a REST API (e.g., Salesforce, HubSpot) where there’s no transaction log to read in the first place.
Example: A small internal reporting tool queries a legacy on-prem SQL Server table every 15 minutes for rows where last_updated > last_check_time, since the older database version doesn’t support log-based CDC. It works fine for new and updated records but if a customer record is deleted outright, that deletion is silently missed, since there’s no row left to query.
3. Trigger-based CDC
Uses database triggers that fire on insert/update/delete and write the change to a separate tracking table.
Works on databases without log-based CDC support, but adds write overhead to every transaction on the source table – a real performance cost on high-traffic systems.
Commonly used with: older MySQL deployments without accessible binlogs, IBM Db2, Sybase, and enterprise Oracle environments where DBAs restrict direct log access for security reasons, forcing teams to fall back to triggers.
Example: An e-commerce platform on an older MySQL setup without accessible binlog-based CDC adds triggers on its orders table that copy every insert, update, and delete into an orders_audit table. A nightly job reads that audit table to sync changes into the warehouse. It’s reliable and captures deletes correctly, but every single order write now does extra work — acceptable at moderate order volume, but a real concern if that table sees Amazon-scale transaction throughput.
In practice, log-based CDC is the default choice for any modern pipeline unless the source database doesn’t support it. it’s the most reliable and has the lowest impact on the source system.
The CDC tool never queries the application tables directly – it reads the transaction log the database already maintains, then fans the changes out to whatever downstream systems need to stay in sync.
| Company / Scenario | How CDC is used | Why not just re-copy everything? |
|---|---|---|
| Swiggy/Zomato — order status sync | The moment an order status changes in the operational database (placed → preparing → out for delivery → delivered), CDC streams that single row change to the systems powering live order tracking and analytics dashboards. | Millions of orders exist historically; re-scanning all of them every few seconds to find the handful that changed would overwhelm the database. |
| Amazon — inventory sync across systems | When a warehouse updates a product’s stock count, CDC propagates that single change to the website’s “in stock” display, the recommendation engine, and the analytics warehouse simultaneously. | Inventory changes constantly and in huge volume; only propagating actual changes keeps every downstream system accurate without re-syncing the entire catalog repeatedly. |
| Banking — transaction replication for fraud detection | Every new transaction row is captured the instant it’s committed and streamed to a real-time fraud-scoring system. | Fraud detection needs the transaction the moment it happens – batch-querying “what changed in the last hour” is both too slow and could miss the fraud detection window entirely. |
| Uber/Ola — keeping a search/cache layer fresh | Driver availability and location changes are captured via CDC and pushed into a fast in-memory cache used for matching, instead of that cache repeatedly querying the main database. | Protects the primary operational database from being hammered by constant read queries from a completely separate system. |