Problem
eBay's risk decision engine checks rules on each user action, such as listing, checkout or payment, to catch account takeovers, collusion and money laundering. Its data access was Java code inside the engine, one class per data source. Every new variable (the data unit a rule reads) needed a full engine release, which took a few weeks to a few months, so analysts could not deploy new rules in time. A leak in one integration could also halt rule execution for the whole engine.
Example from the article: one risk evaluation fires hundreds of rules, each reading 5 to about 20 variables. After removing duplicates it still needs more than 100 variables, within a few hundred milliseconds.
Solution in brief
Move data access into a separate service, Variable Hub, and write each variable as a script in a small sandboxed in-house language instead of Java. Link engine and hub with gRPC and a client-side router that sends all calls of one risk evaluation to one hub instance, so shared source responses can be cached in memory.
Questions this case answers
- How to add new data inputs to a latency-bound rule engine without a full release cycle of the engine itself
- How to keep per-request in-memory caching when a stateless service sits behind a load balancer that spreads one request's calls over many instances
- How to add a new data variable to a risk rule engine in days instead of weeks or months?
- How to stop a broken data integration from taking down the whole decision engine?
Tools and methods
- Variable Hub — a data-access service between the decision engine and data sources
- In-house DSL — a restricted scripting language for variable logic, hot-loaded without compilation
- gRPC with client-side endpoint selection — a fast link without a hardware load balancer
- Sticky session by request ID — one evaluation's variables go to one instance
Google or a one-time email code. No password.