Problem
A team building a foreign-language tutoring assistant (AppFox, for a customer with Arabic and English courses) runs an open-source LLM in production, because the customer wants open models. The customer's methodologists, the language experts who could grade answers and write test questions, are expensive and slow to respond.
Example from the article: to improve Arabic grammar answers, the team needed a RAG corpus and training data. Arabic sources are slow to collect and parse by hand.
Solution in brief
Use ChatGPT and the OpenAI API as a helper around the local model, not as the production model. They convert data, draft prompts, generate synthetic question-answer sets, and grade the local model's answers by grammar, consistency and knowledge. Experiments run as Python scripts with a config and saved prompt artifacts. Methodologists review only the best variants.
Questions this case answers
- How to evaluate the answers of a local open-source generative model cheaply when expert reviewers are too slow and expensive
- How to generate a synthetic question-answer set for testing or fine-tuning with a stronger hosted model
- How to make prompt experiments reproducible from versioned configs and prompt artifacts
- How to evaluate answers of a local open-source LLM when expert reviewers are too expensive and slow
Tools and methods
- OpenAI API (gpt-4o) — bulk, async, JSON-mode calls for data conversion, synthetic data and answer grading.
- ChatGPT — prototyping prompts and writing parsers, before moving to the API.
- LLM-as-judge — scores each local-model answer from 1 to 10 on grammar, consistency and knowledge.
- Config plus executor scripts — one Python config per experiment and one script that runs it.
- RAG and QLoRA — consumers of the synthetic corpus and question-answer pairs.
- Hugging Face, Transformers, JupyterLab, Gradio, DVC, MLflow — environment and tracking stack.
Google or a one-time email code. No password.
