Problem
A bank's data science team ships an LLM feature that turns transcripts of recorded meetings into minutes: type, title, summary, hashtags, conclusions, and a version with a short summary per timestamp. It ran on Qwen2.5 32B, which needs an 80 GB card (H100) and handles contexts of about 30 thousand tokens. The available hardware was A100 cards with 40 GB, so the feature had to fit into half the memory without a visible drop in quality.
Example from the article: a transcript of a working meeting about a network tunnel and traffic rules becomes a JSON with a type, title, summary, hashtags and 00:00:30-style timestamp entries.
Solution in brief
Run all available transcripts through the 32B model to get reference minutes, then fine-tune cheaper candidates on these pairs: a 14B model with LoRA, and 32B models quantized to INT4 (AWQ) and INT8 (SmoothQuant) with QLoRA. Compare them by embedding cosine similarity to the reference, then by manual review. The 14B model won.
Questions this case answers
- How to cut the serving cost of a large language model on smaller hardware while keeping its output quality on one structured generation task
- How to train a smaller model without human-written targets by using the large model as the labeler and choosing between a small model and a quantized large one
- How to cut the GPU cost of an LLM feature that runs on a 32B model without losing output quality
- How to distill a large open LLM into a smaller one using the large model's own outputs as training data
Tools and methods
- Qwen2.5-Instruct (14B, 32B) — teacher and student models
- LoRA / QLoRA — fine-tuning of the 14B model and of the quantized 32B models
- AWQ (INT4), SmoothQuant (INT8) — post-training quantization of the 32B model
- Universal Sentence Encoder — embeddings of the "summary" section for cosine similarity to the reference
- Manual review by annotators — section-by-section comparison
Google or a one-time email code. No password.