Software engineer explains: Real-time Data Integration using HVR

Each month we interview a Sofico colleague about an application or tool Sofico uses daily. This month, Bram Wallach (Product Manager) talks to us about High Volume Replicator (HVR).

What is HVR?

“High Volume Replicator or HVR (www.hvr-software.com) is a powerful software product that enables real-time homogeneous and heterogeneous data replication. HVR uses Change Data Capture (CDC) methods to replicate changes between databases, directories (file locations), as well as between databases and directories, that HVR calls 'locations'. Locations can either be a source or a target.”

“Each change in a source location is captured by HVR, transmitted, and then applied to a target location. Database CDC technology is also referred to as a 'log mining process' that reads a database transaction log for relevant transactions. HVR uses its own internal log mining technology along with certain database vendor APIs.”

Why do we use HVR?

Why do we need to replicate data elsewhere?

“Our Miles core application plays a key role in the transaction processing that underpins our customers’ business, from sales quotations and contract management to accounting processes. Its relational database is normalized to reduce data redundancy and improve data integrity.”

“When looking at shifting workloads, making data available online or supporting downstream reporting and analytics systems, one typically needs some sort of data ingestion layer to transport these data from the OLTP source system to one or more destinations; another database, data marts or a data warehouse.”

Why replicate data in real-time?

​​​​​​​“Our customers have typically relied on batch ETL processing to periodically extract (=E) data sets from the source database, transforming (=T) data for storage in the target location and loading (=L) the data into a target database.”

“While such traditional batch ETL approach may be just fine to support use cases which can be built on a snapshot of yesterday’s data (think quarterly reporting for instance), it falls short of satisfying the requirements in many of today’s use cases such as ‘live’ dashboards (minimum latency), high available data consultation or decision intelligence based on advanced analytics and perhaps even big data. So, we decided to extend our platform capabilities with an option to perform real-time processing (also called stream processing or streaming), ingesting data directly ‘as it happens’.”

“This is also called a ‘data pipeline’; the fluid, seamless movement of multiple data streams from one system to another. Pipelining is about putting the pieces together. As businesses (and software applications) evolve into more loosely coupled components and hybrid architectures combining on-premise platforms with cloud solutions, there has to be orchestration of the movement of data between these systems. It’s not just a slight evolution of the ETL model; it’s a different paradigm."

And that’s where HVR comes in.

How do we use HVR? 

“The Miles Core database (source) runs on Oracle or MS SQL Server. HVR performs log-based CDC from these platforms, directly reading/parsing the binary transaction logs on the database server, and feeds all data changes into an Apache Kafka cluster (target), where they can then be consumed further by other (loosely coupled) services and downstream applications. As illustrated below:”

“So, we basically use HVR for the change data capture and real-time data ingestion as the first steps in our data pipeline architecture.”

“What happens afterward (data transformations, high available data stores, advanced analytics, decision intelligence, etc.) we'll have to explain in a future blog post…”