Add LORA support to TRLX

[ethankim00]

🚀 The feature, motivation, and pitch

LORA and other parameter-efficient methods can provide a number of advantages when finetuning large language models. These methods typically only update a small fraction of the model parameters during finetuning. For example, LORA only trains low-rank reparameterizations of weight matrices during training reducing parameter cost up to 10000 times.

Key Advantages:

• Reduced storage cost and ability to switch out different adapters for different tasks
• Increased training efficiency and reduced memory requirements since fewer optimizer states need to be tracked

Alternatives

No response

Additional context

The OpenDelta library provides support for LORA and other "delta methods"

model =  AutoModelForCausalLM.from_pretrained(model_base)
from opendelta import LoraModel
delta_model = LoraModel(backbone_model=model, modified_modules=['fc2'])
delta_model.freeze_module(exclude=["deltas", "layernorm_embedding"], set_state_dict=True)
# save only  trained parameters
delta_model.save_finetuned(save_path)

[LouisCastricato]

cc @Dahoas

[Dahoas]

Closing this.