Problem
HubSpot runs semantic search for agents, RAG and contact deduplication, used by 38+ teams. A proof of concept in 2023 had about a dozen Qdrant clusters, managed with Helm and manual tuning of replicas, disk, RAM and CPU. That stopped working as the fleet grew to 140+ clusters. Helm cannot call APIs, autoscale on external metrics or run state-aware lifecycle logic. Open-source Qdrant leaves shard transfers and peer removal to the user. Scaling down was a four-step manual job, and uneven shards forced premature scale-ups.
Example from the article: a BM42 sparse-vector collection with over 3 billion points and shards unevenly spread across pods, so the heaviest pod set the memory limit for the whole cluster.
Solution in brief
HubSpot wraps Qdrant in a Kubernetes operator. Each cluster is one custom resource, and "translators" re-check the cluster every 60 seconds. The same loop adds scale-down with safe shard eviction, greedy shard balancing and replica recovery, so a small team can run hundreds of clusters.
Questions this case answers
- How to operate a fleet of stateful distributed search clusters when the open-source engine has no automated scaling, rebalancing or replica recovery
- How to keep per-node memory even in a sharded in-memory index so that one skewed node does not force an early scale-up
- How to run a self-hosted vector database for dozens of teams without operations work growing with every new cluster?
- How to scale a Qdrant cluster down without losing data or breaking the replication factor?
Tools and methods
- Qdrant — open-source vector database with HNSW index, named vectors, hybrid search, quantization
- Kubernetes operator pattern — keeps actual cluster state equal to the desired state in a custom resource
- Kafka — asynchronous write path; indexers consume topics created by the operator
- Vector as a Service (VaaS) — in-house API layer: access control, embeddings, versioning, feedback
Google or a one-time email code. No password.