Inside LinkedIn's data context layer: curating trustworthy context for humans and agents

Ilho Ye, Wei Li, Jianlong Zhong9 Oct 2026

Problem

Writing the query for a data question is quick. Someone first has to find the right dataset, apply the required filters and learn what one row means. LinkedIn's catalog is very large, and similarly named datasets or versions differ in small ways. Which one to use is known to a few people or lost in chat threads.

AI agents write valid SQL in seconds but can pick the wrong dataset or miss a required filter. The query runs and returns a wrong number.

Example from the article: "how many members engaged with a new feature last week" takes a minute to write as SQL and up to a day to prepare.

Solution in brief

One context layer holds two parts under the same asset ID. An LLM pipeline generates baseline dataset profiles from schemas, query logs, code and samples. Teams add business rules and corrections as Markdown in Git, reviewed through pull requests. Agents read both through MCP tools and find files by walking a tree of summary indexes.

Questions this case answers
  • How to supply an LLM agent with trustworthy knowledge about thousands of data assets when manual documentation covers only a fraction and the full text does not fit in the context window
  • How to combine automatically generated descriptions with owner-reviewed corrections so that regeneration never destroys the human knowledge
  • How to help a data agent pick the right dataset, filters and joins in a catalog with millions of similarly named tables?
  • How to document a huge dataset catalog when owners will not write the docs?

Tools and methods

  • AI enrichment pipeline — nominate, collect, generate, persist profiles
  • Query-log canonicalization — finds the joins and filters people really use
  • Curated context as code — Markdown plus YAML in Git, owner-approved pull requests
  • Index tree with summaries — progressive disclosure, so the agent loads only relevant files
  • Embedding search and MCP tools — discovery, retrieval and index navigation
  • DataHub — metadata catalog: nominations, owners, reviewer routing
Continue reading — it's free

Google or a one-time email code. No password.