Fine-tuning Llama 3 with SFT to create a psychotherapy chatbot

[Colab link] [Huggingface Model]

This post explores how Supervised Fine-Tuning (SFT) can be used to fine-tune Llama 3 into a psychotherapy chatbot, addressing the widespread mental health gap. By training the large language model (LLM) on curated therapeutic conversations and leveraging Parameter-Efficient Fine-Tuning (QLoRA), the chatbot delivers more empathetic and therapeutically sound responses. Evaluation using the LLM-as-a-judge method confirmed that the fine-tuned model’s responses were preferred over the pre-trained version.

Improving psychotherapy chatbot responses in Llama 3 using DPO

[Colab link] [Huggingface Model]

This post builds on previous post on Supervised Fine-Tuning (SFT) for developing a psychotherapy chatbot by enhancing response quality using Direct Preference Optimization (DPO), an efficient Reinforcement Learning from Human Feedback (RLHF) technique. While SFT improved the chatbot’s empathetic and professional responses, it occasionally generated irrelevant or unhelpful answers. To address this, DPO refines the model by directly optimizing response preferences without requiring a separate reward model, making it computationally efficient.