Problem
A team that wants its own ChatGPT-like assistant has an open base model but no instruction data. OpenAI got ChatGPT by fine-tuning GPT-3 on detailed, step-by-step answers from many annotators and on real user queries from its own platform. Almost no other company has either. The question of the talk is whether LLaMA, which beats GPT-3 at a smaller size, can be fine-tuned on a small budget to compete.
Example from the article: Stanford Alpaca grew 175 hand-written seed tasks into 52 thousand training examples and fine-tuned LLaMA 7B on them in 3 hours on A100 cards.
Solution in brief
Replace human annotation with generation. A strong model produces instruction data from a handful of seed tasks (self-instruct), another pass makes the instructions harder (Evol-Instruct), and ratings from an open Q&A site replace paid preference labels for the reward model. LoRA, 8-bit quantization and sequence packing cut the training time.
Questions this case answers
- How to build instruction-tuning data for an open language model without human annotators, starting from only a few hundred seed tasks
- How to train a reward model and run policy optimization without paid preference labels, while keeping the policy from exploiting the reward
- How to fine-tune an open LLaMA model on instructions without a team of annotators
- How to generate a synthetic instruction dataset from a few hundred seed tasks
Tools and methods
- Self-instruct — bootstrap an instruction dataset from 175 seed tasks
- Evol-Instruct — rewrite simple instructions into harder ones (WizardLM)
- Supervised fine-tuning (SFT) — Alpaca, Vicuna, Koala, WizardLM
- Reward model and PPO with KL penalty — the RLHF stages of StackLLaMA
- LoRA, 8-bit quantization, packing — cheaper training
Google or a one-time email code. No password.