One of the most difficult aspects to building enterprise-ready data integration pipelines is capturing just the latest changed data from disparate sources quickly. Change Data Capture (CDC) is a set of technologies and techniques that enables data engineers to discover changed records as they occur at the source.
Azure Data Factory (ADF) has recently added many new CDC-enabled connectors to process change data from SQL, Storage, Cosmos DB, and many other sources. Much of the feedback that we received from our users about this has been centered around making it easy to configure and to continuously detect changes at the source.
We heard your feedback and are super excited to announce the public preview and immediate release of a new top-level ADF resource, found in your ADF resource explorer, that will automatically detect changes in your data source(s).
NOTE: This feature is being rolled out to Azure regions throughout this week.
With this new feature, you can quickly configure a continuous process to automatically read changes from your data source(s) without needing to design pipelines or data flows and triggers. Just point to your sources, tell ADF where you want to land the data, and click start. It’s that easy!
With the CDC resource, there is no need to learn ADF concepts like triggers, schedules, integration runtimes, and more. Simply follow a guided process to configure your sources, targets, and any required data transformation options. ADF will then auto-map your columns based on fuzzy matching.
The new CDC resource will also offer continuous data integration as well as frequency-based latencies. This makes budgeting super-easy as the cost of using a CDC resource is a 4-core General Purpose dataflow cluster that is only billed while your data is being processed based on the latencies you select.
Get started now with data processing by trying the new Change Data Capture resource in Azure Data Factory, now available in Public Preview. Find more information in our documentation and follow this guide to set up your new CDC resource.
We are always open for feedback so please let us know your thoughts in the comments below or add to our Ideas forum.