Comparing the Performance of LLMs: A Deep Dive into Roberta, Llama 2, and Mistral for Disaster Tweets Analysis with Lora

• Comparing the Performance of LLMs: A Deep Dive into Roberta, Llama 2, and Mistral for Disaster Tweets Analysis with LoRA
  • Introduction
  • Hardware Used
  • Goals
  • Dependencies
  • Pre-trained Models
    • RoBERTa
    • Llama 2
    • Mistral 7B
  • LoRA
  • Setup
  • Data preparation
    • Data loading
    • Data Processing
  • Models
    • RoBERTa
      • Load RoBERTA Checkpoints for the Classification Task
      • LoRA setup for RoBERTa classifier
    • Mistral
      • Load checkpoints for the classification model
      • LoRA setup for Mistral 7B classifier
    • Llama 2
      • Load checkpoints for the classification mode
      • LoRA setup for Llama 2 classifier
  • Setup the trainer
    • Evaluation Metrics
    • Custom Trainer for Weighted Loss
    • Trainer Setup
      • RoBERTa
      • Mistral-7B
      • Llama 2
  • Hyperparameter Tuning
  • Results
  • Conclusion
  • Resources

Introduction

In the fast-moving world of Natural Language Processing (NLP), we often find ourselves comparing different language models to see which one works best for specific tasks. This blog post is all about comparing three models: RoBERTa, Mistral-7b, and Llama-2-7b. We used them to tackle a common problem - classifying tweets about disasters. It is important to note that Mistral and Llama 2 are large models with 7 billion parameters. In contrast, RoBERTa-large (355M parameters) is a relatively smaller model used as a baseline for the comparison study.

In this blog, we used PEFT (Parameter-Efficient Fine-Tuning) technique: LoRA (Low-Rank Adaptation of Large Language Models) for fine-tuning the pre-trained model on the sequence classification task. LoRa is designed to significantly reduce the number of trainable parameters while maintaining strong downstream task performance.

The main objective of this blog post is to implement LoRA fine-tuning for sequence classification tasks using three pre-trained models from Hugging Face: meta-llama/Llama-2-7b-hf, mistralai/Mistral-7B-v0.1, and roberta-large

Hardware Used

• Number of nodes: 1
• Number of GPUs per node: 1
• GPU type: A6000
• GPU memory: 48GB