Confluent Helps Power a Diverse and Scalable Online Shopping Experience for OTTO

After running Apache Kafka^® on-prem for a few years, OTTO found marketplace growth outpaced internal resources, so self-management was no longer an option. To fill the gaps and ease the management burden, OTTO chose Confluent Cloud as the basis for its newly expanded event exchange platform.

“Running Kafka ourselves was a bit of a no-go given our team size and requirements. As a team we decided we wanted to focus on the technical functions of the platform. We wanted to spend more energy on these functionalities, and leave other things to [Confluent] who can do it better!” says Fabian Schmidt, senior expert product data technology at OTTO.

As a result, OTTO has standardized all data streams, improved overall accessibility, and made it easy to onboard new platform partners with fewer operational headaches.

Untangling a complex network of disparate data sources

As a leading e-commerce platform, everything OTTO does revolves around its customers. And that involves offering the biggest possible range of products to meet and fulfill a diverse range of needs. However, the event streaming solution OTTO had built internally relied on monolithic data banks and used replay mechanics to reproduce events and send them to different teams and different internal or external clients.

“We relied on an event streaming solution we had created ourselves, which meant when new clients wanted to get data from us, each needed a separate implementation,” says Schmidt. 

For the best experience, product data needed to be available as a service, but this was stunted by a complex system that attempted to read lots of different and disparate data amendments at once. “We wanted all of our clients to be in a position to consume our data as easily as possible – and also allow them to process any changes and deletions quickly and efficiently,” says Schmidt.

Building a future-ready event exchange platform

After assessing options, OTTO decided Confluent was the platform of choice to form the basis for its core systems, taking a 10-yearold data infrastructure and bringing it into a new age.

When a new platform partner requests access to the OTTO e-commerce platform, OTTO receives source data, bringing it together centrally and distributing it to the right locations at speed. Instead of each client needing a separate implementation, OTTO has a system optimized for efficiency. “Thanks to Kafka and Confluent we now only have one protocol, so new teams can get hold of our data quickly,” says Schmidt.

“We actually have our own microservices in place that pulls the relevant product data, validates it, processes, and then creates new data streams,” he says. “Confluent has helped us chain up a lot of our data processes to ensure when we’re done with processing, our clients get good, healthy product data.”

The initial implementation went off without a hitch, owing to OTTO’s existing experience with Kafka and ongoing support from Confluent, who were on hand to troubleshoot and plug any knowledge gaps.
“We were able to use the platform pretty easily ourselves, thanks to the information provided by Confluent. The support was really good, and during the pilot phase we were very well taken care of,” says Schmidt.

Enjoying the results of a modern data streaming platform

With a future-proof event exchange platform powered by Confluent, OTTO has achieved some impressive results:

Product data processed in as little as 15 minutes

With Confluent, event streaming is quick and easy, which has cut out the need for monolithic data banks and helped OTTO’s clients and partners access business-critical data much faster. “In 15 minutes, we aim to take product data all the way through the process. That’s an important KPI for us, and it massively increases the currentness of our data, which is a big win for our clients,” says Schmidt.

A platform built to scale

With low-maintenance software that’s both easily adoptable and agile enough to react quickly to business requirements, OTTO now has a purpose-built platform for growth, and can quickly scale up, onboard new teams, and add new functionality based on demand. “Kafka scales way better than most other data-exchange platforms, as it doesn’t use HTTP for data exchange and offers blazingly fast throughput out-of-the-box,” says Torsten Rascher, team lead, integration services at OTTO.

Standardized data for faster client onboarding

Before, data accessibility was poor, as there were many communication islands between teams. Now, data is available at the push of a button, which helps expedite onboarding and speed up the development of new features. 

“We’re now able to standardize not just the whole data stream but many individual smaller data streams. Starting new topics is really intuitive,” says Schmidt. “Our data gets into live scenarios much faster, which makes it easier to onboard new clients and implement productive new cases.”

Technical Solution

• Schema Registry (Stream Governance)
• CSID Accelerators (E2E encryption)
• SSO with Keycloak using Azure AD
• Galapagos as a self-service portal for all teams

What’s next for OTTO?

OTTO is now well-equipped with the tools needed to continue growing its e-commerce business in Germany and beyond. “We’re constantly adding teams and new use cases, and we think that will continue as we grow as a company. We’re currently in the process of migrating all of those who use our data to the new method of distribution – and are looking at our internal analytics infrastructure to consider how we accelerate the process of data distribution further,” says Schmidt.

For now, OTTO has a platform that can handle new teams, exciting use cases, diverse product lines, and more without breaking a sweat. “It’s good to know that if we needed more bandwidth, it’s there – but for what we do now, the system is very well-scaled,” says Schmidt.