Problem
A SingularityNET team wanted business users to ask questions in plain language about financial time series, such as prices, trading volumes and volatility of crypto pairs. The data updates every second, so the LLM cannot be trained on it and answers must follow the data. The team also had no way to compare self-hosted and paid models, and any change of model, prompt or tool list could silently break answers.
Example from the article: the question "btc price?" must return the number 101000 in the answer text; the same question in another phrasing ("bitcoin price?") must give the same result.
Solution in brief
The LLM never sees the time series. It only picks one of the pre-written Python functions, fills in its parameters and turns the returned value into text. A benchmark of 100 question and answer pairs, run against stubbed functions, measures how often this works and lets the team compare models on the same task.
Questions this case answers
- How to let an LLM answer natural-language questions about live numeric series by choosing among prewritten functions, so that the numbers never pass through the model
- How to benchmark a tool-calling LLM pipeline against stubbed data and choose between a self-hosted small model and a paid model
- How to let an LLM answer natural-language questions about live time-series data
- How to choose an LLM for a tool-calling pipeline with a task-specific benchmark
Tools and methods
- Tool-augmented generation — LangChain parses the question, calls a hardcoded function over the time-series DB and phrases the result.
- Benchmark-driven development — stub functions return fixed values, so answers can be checked without live data.
- DeepEval — LLM-as-a-judge accuracy and hallucination metrics.
- Models compared — Llama 3.1/3.2, Qwen2 and Qwen2.5 via Ollama, GPT-4o, GPT-4o-mini, DeepSeek-V3.
Google or a one-time email code. No password.