Fine-tuning & LoRA

Teaching an old model new tricks

Pretraining a large model costs millions. Adapting one doesn’t have to. Fine-tuning means continuing to train a pretrained model on a small, specific dataset: medical notes, your company’s support tickets, a particular style of writing. Everything the model already learned about language carries over. This is transfer learning, and it’s why one pretrained model can become thousands of specialised ones.

• Supervised fine-tuning (SFT): train on example conversations written the way you want the model to answer. It’s the first step in turning a pretrained model into an assistant.
• Preference tuning (RLHF and its relatives): train on which answers people prefer. You met that in the previous lesson.