New Log-Based CDC Database Connectors

We're excited to announced that our new log-based connectors for MySQL, PostgreSQL and Oracle databases have just been released into a public beta.

These connectors use the logical replication methods of database systems and read all events (transactions) directly from the binary (or WAL) log. This minimizes the load on the source system and also enables us to capture every single event (INSERTs, UPDATEs, and DELETEs) that happens in the database.

How do they work?

• You select the tables and columns you want to sync. The connector supports regex patterns so that you can filter replicated database objects in bulk.
• The connector performs an initial sync of the tables.
  • This step can be skipped or modified by custom snapshot queries for each table.
• The connector starts reading events directly from the replication log. Hence, it avoids sending direct queries to the database. Events are processed only once, thus ensuring the consistency and efficiency of the load.
• The schema changes are handled automatically without interruption and without needing a re-sync.
• You can choose to keep all changes in the resulting tables or deduplicate the batch result so that only the latest states of each record are reflected in the result table.
• The connector uses micro-batches, so you can schedule them as often as needed, whether every 5 minutes or twice a day.
  • This approach gives you more control over the resulting cost of the solution and lets you leverage the power of log-based replication even for smaller use cases if needed.
  • If your use case requires short interval syncs, reach out to us for information about a cost-efficient CDC add-on license.

Our connectors are based on the widely adopted Debezium framework. They use its powerful features while providing a convenient UI interface.

Supported Databases

• MySQL, MariaDB
• PostgreSQL
• Oracle (currently incubating in private beta)
• More to be added soon

What's next?

Apart from adding new functionalities and supported databases, we plan to introduce a fully streaming CDC solution this year, which will be available out of the box in our platform.

Learn more about the functionalities and the setup process of these log-based CDC connectors in our documentation. If you have any queries or feedback, please do not hesitate to contact us.