Tutorial for fine tuning of Mistral 7B with Qlora using Axolotl for Effective LLM training

In this tutorial we demonstrate the workflow of fine -tuning Mistral 7B using Qlora with Axolotlshowing how to manage limited GPU resources while adjusting the model to new tasks. We install Axolotl, create a small sample data set, configure the LORA-specific hyperparameters, run the fine-tuning process and test the resulting model performance.

Step 1: Prepare the environment and install Axolotl

# 1. Check GPU availability
!nvidia-smi


# 2. Install git-lfs (for handling large model files)
!sudo apt-get -y install git-lfs
!git lfs install


# 3. Clone Axolotl and install from source
!git clone https://github.com/OpenAccess-AI-Collective/axolotl.git
%cd axolotl
!pip install -e .


# (Optional) If you need a specific PyTorch version, install it BEFORE Axolotl:
# !pip install torch==2.0.1+cu118 --extra-index-url https://download.pytorch.org/whl/cu118


# Return to /content directory
%cd /content

First we check which GPU is there and how much memory is there. We then install Git LFS so that large model files (which Mistral 7b) can be handled properly. Then we clon the Axolotl archive from GitHub and install it in “Editable” mode, which allows us to call its commands anywhere. An optional section allows you to install a specific Pytorch version if necessary. Finally, we navigate back to /content folder to organize subsequent files and trails nicely.

Step 2: Create a small examples of data set and Qlora Config to Mistral 7B

import os


# Create a small JSONL dataset
os.makedirs("data", exist_ok=True)
with open("data/sample_instructions.jsonl", "w") as f:
    f.write('{"instruction": "Explain quantum computing in simple terms.", "input": "", "output": "Quantum computing uses qubits..."}\n')
    f.write('{"instruction": "What is the capital of France?", "input": "", "output": "The capital of France is Paris."}\n')


# Write a QLoRA config for Mistral 7B
config_text = """\
base_model: mistralai/mistral-7b-v0.1
tokenizer: mistralai/mistral-7b-v0.1


# We'll use QLoRA to minimize memory usage
train_type: qlora
bits: 4
double_quant: true
quant_type: nf4


lora_r: 8
lora_alpha: 16
lora_dropout: 0.05
target_modules:
  - q_proj
  - k_proj
  - v_proj


data:
  datasets:
    - path: /content/data/sample_instructions.jsonl
  val_set_size: 0
  max_seq_length: 512
  cutoff_len: 512


training_arguments:
  output_dir: /content/mistral-7b-qlora-output
  num_train_epochs: 1
  per_device_train_batch_size: 1
  gradient_accumulation_steps: 4
  learning_rate: 0.0002
  fp16: true
  logging_steps: 10
  save_strategy: "epoch"
  evaluation_strategy: "no"


wandb:
  enabled: false
"""


with open("qlora_mistral_7b.yml", "w") as f:
    f.write(config_text)


print("Dataset and QLoRA config created.")

Here we build a minimal JSONL Data set with two instructional responses, which gives us a toy example of training. We then construct a YAML configuration pointing to the Mistral 7B base model, set Qlora parameters to memory-efficient fine-tuning and define the training of hyperparameters such as batch size, learning speed and sequence length. We also specify Lora settings such as dropout and rank and eventually save this configuration as Qlora_mistral_7b.yml.

Step 3: Finch with Axolotl

# This will download Mistral 7B (~13 GB) and start fine-tuning with QLoRA.
# If you encounter OOM (Out Of Memory) errors, reduce max_seq_length or LoRA rank.


!axolotl --config /content/qlora_mistral_7b.yml

Here, Axolotl automatically picks up and downloads and downloads Mistral 7B weights (a large file) and then initiates a Qlora-based fine-tuning procedure. The model is quantized for 4-bit precision, which helps reduce GPU memory consumption. You can see training logs that show the progress, including the loss of exercise, step by step.

Step 4: Test the fine -tuned model

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer


# Load the base Mistral 7B model
base_model_path = "mistralai/mistral-7b-v0.1"   #First establish access using your user account on HF then run this part
output_dir = "/content/mistral-7b-qlora-output"


print("\nLoading base model and tokenizer...")
tokenizer = AutoTokenizer.from_pretrained(
    base_model_path,
    trust_remote_code=True
)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_path,
    device_map="auto",
    torch_dtype=torch.float16,
    trust_remote_code=True
)


print("\nLoading QLoRA adapter...")
model = PeftModel.from_pretrained(
    base_model,
    output_dir,
    device_map="auto",
    torch_dtype=torch.float16
)
model.eval()


# Example prompt
prompt = "What are the main differences between classical and quantum computing?"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")


print("\nGenerating response...")
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=128)


response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print("\n=== Model Output ===")
print(response)

Finally, we load the Base Mistral 7B model again and then apply the newly trained Lora weights. We create a quick prompt about the differences between classic and quantum computing, convert it to tokens and generate an answer using the fine -tuned model. This confirms that our Qlora training has taken effect and that we can successfully run inference on the updated model.

Snapshot of supported models with Axolotl

Finally, the above steps have shown you how to prepare the environment, create a small dataset, configure Lora-specific hyperparameters and run a Qlora final adjustment session on Mistral 7B with Axolotl. This approach shows a parameter -efficient training process suitable for resource -limited environments. You can now expand the data set, change hyperparameters or experiment with various open source LLMs to further refine and optimize your fine -tuning pipeline.

Download Colab Notebook here. All credit for this research goes to the researchers in this project. Nor do not forget to follow us on Twitter and join in our Telegram Channel and LinkedIn GrOUP. Don’t forget to take part in our 75k+ ml subbreddit.

🚨 Marketchpost invites AI companies/startups/groups to collaborate with its upcoming AI magazines on ‘Open Source AI in Production’ and ‘Agentic AI’.

Asif Razzaq is CEO of Marketchpost Media Inc. His latest endeavor is the launch of an artificial intelligence media platform, market post that stands out for its in -depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts over 2 million monthly views and illustrates its popularity among the audience.

✅ [Recommended] Join our Telegram -Canal

Leave a Comment Cancel reply