Full talk · edited transcript

Ultra Low Latency Streaming for Real-Time Feature Engineering

Abhay Bothra, Jerry Peng17 Jun 2026

Abhay Bothra and Jerry Peng (Software Engineers, Databricks) · Databricks.

Feature freshness is critical for fraud detection, ad tech, real-time recommendations, dynamic pricing and similar use cases. Historically, ML engineers have had to maintain two separate systems for feature engineering: Spark and Spark Structured Streaming for batch and near-real-time features, and bespoke pipelines when sub-second feature freshness was needed. This talk covers Real-Time Mode (RTM) in Apache Spark Structured Streaming, the engine behind the Databricks Feature Store, and how companies such as Coinbase and MakeMyTrip use it to get real-time features with millisecond-level freshness. It also shows how to pair RTM with Lakebase to build an online feature store, giving a blueprint for a unified feature engineering platform that eliminates logic spread across different systems and shifts the focus from infrastructure orchestration to semantic feature modeling.

Agenda: feature engineering overview (why feature freshness matters, the "two systems" problem); Real-Time Mode in Structured Streaming and in SDP (Spark Declarative Pipelines); do-it-yourself real-time feature pipelines on Databricks; the Databricks Feature Store; customer stories; what is next.

Continue reading — it's free

Google or a one-time email code. No password.