Vibe Coding Discovercurlora
View on GitHubThe code repository for the CURLoRA research paper. Stable LLM continual fine-tuning and catastrophic forgetting mitigation.
Research code for CURLoRA, a LoRA variant that uses CUR matrix decomposition to fine-tune LLMs with fewer trainable parameters and less catastrophic forgetting. Includes PyTorch/Transformers implementations and notebooks reproducing experiments on Mistral 7B and GPT2-Large.
Use Cases
Continual fine-tuning of LLMs without catastrophic forgettingParameter-efficient LoRA replacement in attention layersSentiment classification fine-tuning on MRPC, SST-2, Sentiment140Q&A fine-tuning on SQuAD with SFTTrainerReproducing CURLoRA vs LoRA research benchmarksReducing trainable parameters during LLM adaptation
Built With
- Language
- Jupyter Notebook
- Frameworks
- PyTorch · Hugging Face Transformers · TRL · cuda
Tags
lora · peft · fine-tuning · continual-learning · catastrophic-forgetting · cur-decomposition · llm · parameter-efficient · matrix-decomposition · research-code · sft · quantization-planned · transformers · pytorch