Confluent sells hosted Kafka: companies use it to move data between their systems as things happen instead of overnight. Public customer stories like Victoria’s Secret, BMW Group, and King (maker of Candy Crush) give a sense of what that looks like in practice.
I work on the web app customers use to run all of it. A Senior Software Engineer II here does more than write code: leading projects that cross several teams, mentoring other engineers, and helping decide what gets built next as much as how it gets built. My time generally splits about evenly between the two, roughly half writing code and half on team and project leadership.
I’ve led four different parts of the product, one after another.
Connect
First dedicated UI engineer on Kafka Connect, Confluent’s fully managed product for moving data in and out of Kafka using “connectors,” prebuilt plugs to outside systems. There are over 100 of them.
Most of the actual work was improving the UI for observability, debugging, settings, and provisioning, the day-to-day surface customers use to run connectors that are already moving their data. Getting that wrong isn’t a cosmetic problem: the wrong change can take down a production pipeline, and now there’s a customer complaining about it on Twitter.
- Built the proxy that made connector UIs fast to build: each connector’s actual settings lived as JSON inside that connector’s own backend repo, bundled into a JAR, and only became real once it shipped and ran on a Connect cluster, which exposed it through the runtime’s own config API. I put a proxy in front of that API so the frontend could read a connector’s real configuration directly and build a UI around it, rather than working blind against a spec that lived somewhere else. The settings screens still need to be built by hand, but the backend repos stayed the actual source of truth, and cut onboarding time for a new fully-managed connector by as much as 90% in some cases.
- The hard part wasn’t the proxying itself: the raw config JSON coming out of a connector runtime wasn’t reliably well-formed, and the validation logic that was supposed to run against it didn’t actually execute outside the runtime. So on top of the normal middleware and CORS handling needed to let a browser talk to it at all, the proxy also had to reproduce what the runtime does in practice, not just read its output.
- Ran a long series of experiments to improve connector provisioning: Confluent’s revenue is consumption-based, and the main way customers consume is by moving data in or out through a connector, so a connector that gets configured but never finishes provisioning is revenue that never happens. Every percentage point of improvement there has immediate, downstream revenue impact. Some individual connectors saw conversion rates improve more than 40% over the three years I owned Connect.
- What that’s actually for: a gaming company might stream player telemetry into an analytics platform so designers can see how people play and adjust difficulty in near real time, instead of waiting on an analyst to write a query and build a dashboard. A medical device manufacturer might stream mechanical telemetry off equipment in the field, so a potential failure gets caught the moment it starts, not after the fact, when seconds matter.
Custom Compute
Founding UI engineer on a new product line letting customers run their own code inside Confluent, instead of choosing only from prebuilt connectors.
- Built the interface: the UI customers use to upload, configure, and manage their own code, running on an execution environment the backend team built to invoke it as a connector in its own right.
- Why that matters: if Confluent doesn’t have a connector for something a customer needs, they can write one and run it themselves. A real example of what that unlocks: pulling every new post from r/news the moment it’s posted, generating an embedding for it with OpenAI’s embedding API, and writing that into a vector database like Pinecone, all as part of the same stream. The customer ends up with semantic search over their own live feed of data, updated in real time instead of batched in overnight.
Information Architecture
The product’s navigation and app chrome had barely changed in almost ten years, while the number of things built on top of it kept growing. The structure was still organized around the old pitch, Kafka as a way to move data between systems, while marketing had moved on to a bigger one: not logistics, a full platform for processing data as it happens, the “data streaming platform.”
- The mismatch: new products kept getting bolted onto navigation that was never designed to hold them, because it still reflected the old mental model instead of the one the company was now selling.
- Why it matters right now: this isn’t just a rebrand. An LLM is only as useful as the context it’s working from, and a growing share of that context needs to be real time rather than a nightly batch job. A platform built for that has to actually be organized like one, not like a place that moves files around overnight.
Side bets
Noticed customers were struggling to provision connectors inside private networks. Getting something like AWS PrivateLink configured alongside Confluent’s own private networking is genuinely hard to set up correctly.
- Ran an experiment: partnered with the networking team on the idea that marketing the feature and walking customers through setup, not just documenting it, would raise adoption. Sounds obvious in hindsight; it wasn’t obvious going in.
- Result: connector adoption inside private-link clusters rose 70%, and the feature became permanent.
The other one I surfaced myself: nobody could actually answer whether UI changes affected how long a connector kept running, not just whether it got created. Making a connector easier to set up can look like a win in the revenue numbers that week or month, but if that same connector gets abandoned in six months instead of running for years, that’s a worse outcome wearing a good one’s clothes, and there was no data to tell the difference.
- Built end-to-end lifecycle tracking: from the moment the UI loads and what the customer actually sees, all the way through to the connector eventually getting deleted, sometimes years later, sometimes by an automated bot calling the API rather than a person. Before this, that whole path was invisible.
- Needed buy-in across the org to build it: two backend teams, data science, and sign-off from each of those teams’ managers, since the data needed for this didn’t live under any one team’s ownership.
- Built it as a generic solution, not just for my own team: wrote up the pattern internally so other engineers could instrument their own funnels the same way, rather than it staying a one-off for Connect.