1

I wish to fine-tune a base LLM model using LoRA with multiple datasets that are structured differently (different columns and data types). I have two questions:

Can I fine-tune the model with the first dataset, then add/fine-tune the generated LoRA bin with the subsequent datasets? I want to end up with one LoRA bin trained on all of my different datasets. Is that possible?

If it's not, I will create a separate LoRA bin for each dataset. Can I load the multiple generated LoRA bins on top of the base model in the same session and use them simultaneously?

I can see on this page (https://github.com/huggingface/peft) that multi-adapter support is now available, but the code isn't clear on how to use it for fine-tuning and inferencing:

compute_environment: LOCAL_MACHINE
deepspeed_config:
  gradient_accumulation_steps: 1
  gradient_clipping: 1.0
  offload_optimizer_device: cpu
  offload_param_device: cpu
  zero3_init_flag: true
  zero3_save_16bit_model: true
  zero_stage: 3
distributed_type: DEEPSPEED
downcast_bf16: 'no'
dynamo_backend: 'NO'
fsdp_config: {}
machine_rank: 0
main_training_function: main
megatron_lm_config: {}
mixed_precision: 'no'
num_machines: 1
num_processes: 1
rdzv_backend: static
same_network: true
use_cpu: false

The adapter-transformers library can "stack" or "fuse" different adapters as long as they are done through LoRA, unfortunately. You can see more here: https://docs.adapterhub.ml/adapter_composition.html#stack

karim1104
  • 13
  • 4

1 Answers1

1

I guess both the approach that you have mentioned, works.

Incase you want to use multiple lora adapters to fine-tune, you can fine-tune each adapters on your different datasets and store separately. During inference, you can use them as below

##Load your base model
model = AutoModelForCausalLM(path_to_model)

##Loading the peft model with first adapter
model = PeftModel.from_pretrained(model, adapter1_path, adapter_name="adapter1_name")

##Add another adapter
model.load_adapter(adapter2_path, adapter_name="adapter2_name")

Now your base model is loaded with 2 adapters.

While inferencing if you want to use the second adapter, you can use

model.set_adapter('adapter2_name')

To check the active adapter :

print(model.active_adapter)
gag123
  • 357
  • 1
  • 2
  • 8