België
Nederland
France
Deutschland
United Kingdom
Australia
日本
International
Mexico
Remote working: that's how it's done here
In these exceptional times of seclusion and care, all businesses are looking for new ways to keep in touch.
Software engineer explains: Kafka
Each month we interview a Sofico colleague about an application or tool Sofico uses daily. This month, Petar Pilipovic (Lead Developer) and Wim Van Den Eynde (Technical Consultant) talk to us about Apache Kafka.
It had to be fast - almost without latency -, able to process high amount of data and easy to scale up. We found such a solution in Apache Kafka.
What is Kafka?
Apache Kafka is a distributed streaming platform. It has three key capabilities:
- It lets you publish and subscribe to streams of records. In this respect, it is similar to a message queue or enterprise messaging system
- It lets you store streams of records in a fault-tolerant way
- It lets you process streams of records as they occur.
Kafka is an open-source project, developed by LinkedIn and it is part of the Apache Software Foundation since 2012. One of the founders of Kafka, Jay Kreps, chose to name the software after the author Franz Kafka because it is "a system optimized for writing".
Why do we use it at Sofico?
“Everything started a couple of years ago, when new requirements were popping up to get almost real-time analytics and reports. With the setup and architecture, we had at that time, the fulfilment of such requirements could have an impact on the performance of our core system, so we looked for a solution.”
“It had to be fast almost without latency, able to process high amount of data and easy to scale up. We found such a solution in Apache Kafka.”
Strimzi provides container images and operators for running Kafka on Kubernetes.
How do we use Kafka at Sofico?
“We are using Kafka as a streaming platform. We stream data from our Miles Core system to our Miles Microservice Platform. Data from Miles Core is streamed to topics, where they are going to be consumed. The data is then transformed to a canonical domain model and published to domain topics.”
We are deploying Kafka as part of our Miles Microservice Platform on Kubernetes. We use Strimzi to run Kafka clusters on Kubernetes in various deployment configurations.
Are there dependencies and specific work methods?
“A typical, Kafka setup contains multiple brokers and Zookeeper instances, which act as a service discovery. In the future, Zookeeper will be removed from Kafka setup, but the current versions still need it.”
“We are deploying Kafka as part of our Miles Microservice Platform on Kubernetes. We use Strimzi to run Kafka clusters on Kubernetes in various deployment configurations.”
“Strimzi provides container images and operators for running Kafka on Kubernetes. Operators simplify the process of deploying and running Kafka clusters and components. Additionally, it allows us to configure access to Kafka, upgrade Kafka, and manage brokers, such as the creation and management of topics and users.”
Do you have any tips and tricks that might be useful to others?
“Kafka is highly configurable. You can configure it to achieve a higher data throughput or to gain higher data consistency. We configure Kafka for high data throughput when you want to stream less important data (which will be aggregated in the end) but we need the data as a real-time feed."
"We configure Kafka for high data consistency when we want to stream very important data (such as customer transactions) and we don't want to lose any data."
---
Depicted: Franz Kafka
Are there alternatives to Apache Kafka?
“Yes, they were other alternatives such as IBM MQ or Rabbit MQ at that moment, but since Kafka can handle big amounts of data in a very performant way as well as some other capabilities that are of interest to us, we decided on using Apache Kafka.”
“We use the software or frameworks which best suit our needs and give us advantages for our projects. In the end, this is what is most important to our customers.”
Tags
More life at Sofico
Sofico gains Investors in People Silver accreditation
Sofico has been awarded Investors in People Silver level accreditation, thanks to initiatives rolled out as part of the companies’ ‘Learning Organization’...
How's the weather?
Since a couple of months, we started at Sofico with our ‘Sofico - How is the weather', a weekly meetup between colleagues.
"I SWOT my new job”
After 21 years working for BMW Financial Services, Bert Vanden Bergh decided it was time for a new challenge. “I had to step outside my comfort zone in...