How Confluent Elevates Data Streaming: Insights from CEO Jay Kreps

 Confluent's Current 2025, Bengaluru was all about data. It is an event that brings professionals involved in managing data. Its focus is on the importance of data streaming and real-time processing.  The event acts as a platform to advance real life and use cases, and insights from across a spectrum of personas - from tech leaders, industry giants, and startups leveraging its platforms like Apache Kafka and Apache Flink.

Before we deep dive into the key insights from the event, let's look at what Confluent does. In the company’s own words: “It’s a data streaming platform that is pioneering a fundamentally new category of data infrastructure that sets data in motion. Its cloud-native offering is the foundational platform for data in motion—designed to be the intelligent connective tissue enabling real-time data from multiple sources to constantly stream across an organization.”

Essentially, what it means is, that enterprises can meet the new business imperative of delivering rich, digital front-end customer experiences and transition to a more sophisticated, real-time, software-driven back-end operations.

The company is also going aggressive in India and has forged a strategic partnership with Jio Cloud Services. Confluent Platform will be offered as a managed service in Jio Cloud Services for Indian consumers. 

Tableflow: Real-time Business Context to Analytical System

The company also announced the availability of Tableflow, which it claims to be the easiest way to access operational data from data lakes and warehouses. Moreover, all streaming data in Confluent Cloud can be accessed in popular open table formats, unlocking limitless possibilities for advanced analytics, real-time AI, and next-generation applications.

How AI and Data Streaming Are Changing the Game

Let's hear from Jay Kreps, the Co-founder, and CEO of Confluent, from his keynote address:

Kreps paints a compelling picture of where businesses are headed in the age of AI. The crux of his argument revolves around how companies today are no longer just businesses—they are becoming software-driven enterprises. AI is pushing this shift even faster, turning manual, human-driven decisions into automated, real-time processes powered by data streaming.

Giving a rationale to this thought, Kreps explains that in the past, businesses relied on batch processing and periodic reports to make decisions. But that approach is already becoming outdated. Now, AI enables instant real-time insights and continuous automation, changing the way companies operate at their core. A completely new way of working with data for actionable insights. Here are some key takeaways:

Software Runs the Show: More and more, businesses are moving away from manual workflows. Instead of waiting for humans to analyze data and make decisions, software is doing it on the fly. AI will only speed this up.

From Reports to Real-Time AI: Traditional Business Intelligence involved gathering data, cleaning it, and generating reports. That was slow and reactive. Now, companies are shifting to AI-driven automation, where decisions happen in real-time, thanks to streaming platforms like Kafka.

GenAI is the New Normal: Unlike traditional AI models, which rely on batch training, Generative AI (GenAI) can process and generate insights instantly. This changes how businesses use AI, moving from data lakes and warehouses to real-time AI applications.

AI Agents Are the Future of Automation: Imagine a system where AI automatically manages inventory, classifies products, and updates records – without human intervention. That’s where AI agents come in. These are built using Kafka-based microservices, allowing businesses to automate key processes while staying agile.

Real-time data is the Backbone of AI: Today’s businesses need to integrate operational data (what’s happening right now) with analytics (understanding patterns and making predictions).

The Need for Data Streaming Platforms

These platforms bridge this gap, helping companies move from static, stale reports to real-time insights. Let's look at how these platforms are demolishing the legacy ways of slicing and dicing the data.

Good Data Starts at the Source:The old way? Gather messy unstructured data and clean it at every step. The new way? Process and structure data at the source, so that everything downstream works smoothly.

AI is Now a Core Job Skill: AI isn’t just for data scientists anymore. Every engineer will need to understand how to build AI-powered systems as part of their work.

The Big Picture: Data on IMAX: One can say that with real-time data streaming, you get to see more, just like a movie on an IMAX screen. Businesses that embrace real-time data streaming and AI-driven automation will outperform their competitors. The companies that  fence sit? They risk falling behind. AI is no longer just an extra tool—it’s becoming the foundation of how modern businesses run.

Apache and Kafka: A Storytelling Approach to Managing Data

The Apache Software Foundation got its name after the Apache Native American tribe, symbolizing its community-driven collaboration and resilience. The parallel here is that, in a similar vein, open-source projects rely on contributors worldwide.

Jay Kreps, one of Kafka’s creators, named it after Franz Kafka, the Czech writer known for his complex, interconnected storytelling. The logic here is, Kafka effectively handles distributed, interconnected data streams. It’s a storytelling approach to managing data.

Apache Kafka, was originally developed at LinkedIn. It was  open-sourced in 2011. Jay Kreps, with Neha Narkhede and Jun Rao, were the key engineers behind it. In 2014, they founded Confluent, which commercialized and pivoted Kafka for enterprise use.