Problem
A retail lakehouse platform organized as data-mesh "hubs" ran into chaos in access control. It had 15+ hubs, about 60,000 authorization requests per minute, hundreds of schemas, thousands of tables and S3 buckets (the slide; the speaker says hundreds of thousands), and almost 5,000 users. The engines were mixed (Trino, Kafka, Flink, Spark) and ran both in Kubernetes and on bare metal, some started on demand. Each tool had its own authorization, so there was no single control point, change history or audit.
Example from the article: a new Trino login policy took 25.1 s to evaluate because of a large input, and engineers could not tell why connecting was slow.
Solution in brief
X5 centralized authorization on Open Policy Agent (OPA) with OPAL, which delivers Rego policies from Git to OPA sidecars in real time. Apache Ranger stays as the UI for access lists. The talk is mostly lessons from running this in production.
Questions this case answers
- How to build one auditable access-control layer for many heterogeneous query engines that keeps working when the central policy service is unreachable
- How to keep policy decisions fast and detect stale policies when rules are distributed from version control to many local agents
- How to build a single, auditable access-control layer for a multi-hub lakehouse with many query engines
- How to keep OPA policy evaluation fast at tens of thousands of requests per minute
Tools and methods
- OPA (Open Policy Agent) — policy engine written in Go; the authorization point for each tool
- OPAL (Open Policy Administration Layer) — pushes policies and data to OPA agents over WebSocket
- Rego — declarative policy language
- Apache Ranger — UI and source of access control lists (ACLs)
- Trino, MinIO — enforcement targets, each with its own OPA sidecar or pods
- GitLab, Grafana, Regal — policy storage, decision-log dashboards, Rego linter and debugger
Google or a one-time email code. No password.
