Parameter-efficient Lora Replacement In Attention Layers

Research code for CURLoRA, a LoRA variant that uses CUR matrix decomposition to fine-tune LLMs with fewer trainable parameters and less catastrophic forgetting. Includes PyTorch/Transformers implementations and notebooks reproducing experiments on Mistral 7B and GPT2-Large.

AI Tools | Jupyter Notebook · lora · peft