Full talk · edited transcript

LLaMA and a Zillion Ways to Fine-Tune It

Karina Romanova27 May 2023

Author: Karina Romanova, NLP engineer (previously SberDevices, development of generative chatbots)

The speaker began studying the application of LLaMA, its pitfalls, and the tasks the model is worth applying to; the topic is close to her previous work with generative networks. She runs a Telegram channel where she analyzes new articles and shares experience, and often publishes her work in open source.

The talk is an overview of how LLaMA is structured and how it was fine-tuned on instructions and dialogues: Alpaca, Vicuna, Koala, WizardLM, StackLLaMA. Throughout the talk, the speaker discusses methods for collecting instructional datasets (self-instruct, Evol-Instruct) and techniques that speed up fine-tuning (LoRA, 8-bit quantization, packing).

Continue reading — it's free

Google or a one-time email code. No password.