LoRA Training

[Galarzaa90]

Hello! Thanks for releasing GLiClass.

I’m fine-tuning GLiClass with LoRA and started from train.py. Do you have an official LoRA/QLoRA script or recommended configuration? If not, any quick pointers on:

• Target modules to adapt
• Typical LoRA parameters
• Precision/memory tips

Thanks!

PS: Is this the right channel for this? Please let me know if there's a better place for these kind of inquiries.

[BioMikeUkr]

Hello, you could start with the parameters we used:

	gliclass‑modern‑base‑v3.0	gliclass‑modern‑large‑v3.0	gliclass‑base‑v3.0	gliclass‑large‑v3.0
LoRa r	512	768	384	384
LoRa α	1024	1536	768	768
focal loss α	0.7	0.7	0.7	0.7
Target modules	"Wqkv", "Wo", "Wi", "linear_1", "linear_2"	"Wqkv", "Wo", "Wi", "linear_1", "linear_2"	"query_proj", "key_proj", "value_proj", "dense", "linear_1", "linear_2", mlp.0", "mlp.2", "mlp.4"	"query_proj", "key_proj", "value_proj", "dense", "linear_1", "linear_2", mlp.0", "mlp.2", "mlp.4"

The key idea is that the larger the LoRa r, the fewer data and epochs you need to fine-tune the model. Try LoRa a between 256-1024. Probably we'll prepare a script for fine-tuning with Peft.

[dcrey7]

a finetuning script with peft would be really helpful, could you also mention how much time it token for finetuning with the gpu and LoRA of such high ranks? if the ranks are so high why not use the traditional finetuning approach?

[Galarzaa90]

Thank you @BioMikeUkr ! This is very helpful!

Now I'm more curious about your fine-tuning script with Peft, the parameters I was using (by experimentation) were way different than yours.

[BioMikeUkr]

a finetuning script with peft would be really helpful, could you also mention how much time it token for finetuning with the gpu and LoRA of such high ranks? if the ranks are so high why not use the traditional finetuning approach?

Here you can find some useful calculations for knowledgator/gliclass-edge-v3.0 for different LoRa a (32-2048, r=0 - without LoRa). Fine-tuning with LoRa could be 30-40% more time-efficient (medium-large models). However, the main reason is that there is a crossover, where training time with LoRA is lower than without it, and eval accuracy is higher than without LoRA. With LoRa, models are fine-tuned more stably.

[dcrey7]

a finetuning script with peft would be really helpful, could you also mention how much time it token for finetuning with the gpu and LoRA of such high ranks? if the ranks are so high why not use the traditional finetuning approach?

Here you can find some useful calculations for knowledgator/gliclass-edge-v3.0 for different LoRa a (32-2048, r=0 - without LoRa). Fine-tuning with LoRa could be 30-40% more time-efficient (medium-large models). However, the main reason is that there is a crossover, where training time with LoRA is lower than without it, and eval accuracy is higher than without LoRA. With LoRa, models are fine-tuned more stably.

thank you! really helpful