wuxianchao/lazylora-7bhf

Lazy LoRA

Benefits

using the updated Meta's LLaMA-2 models.
support 4-bit qlora, extreme GPU memory and inference time saving;
comparable MMLU evaluation dataset results:

	eval	test	comp-eval	comp-test
llama2-7b	46.68%	46.82%
ckpt-200	44.28%	46.03%	-2.40%	-0.79%
ckpt-600	45.26%	45.61%	-1.42%	-1.21%

llama2-7b: "4e4d531bcab430a66c4d562b7e89e21c0fa235ea"

Introduction

Determine the rank of LoRA layers by the singular values of pretrained weight matrices. Also, combines:

LoRA: LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS
Prefix Tuning: [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://aclanthology.org/2021.acl-long.3 53/), [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://arxiv.or g/pdf/2110.07602.pdf)
Prompt Tuning: The Power of Scale for Parameter-Efficient Prompt Tuning
LLaMA adapter: [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention] (https://arxiv.org/abs/2303.16199) in one model.

This allows you to perform LoRA (additional low rank adapters inserted to each linear layer), and prompt learning (additional virtual tokens attached to the input and to the attention layers acting as past_key_values)

Usage:

import sys
sys.path.insert(1, '/workspace/asr/peft/src')
# TODO set this path to the lazy-lora source code path, 
# or you can install it from source code:
# TODO, please install lazylora for usage:
# git clone [email protected]:Xianchao-Wu/peft.git
# cd peft
# python setup.py install

from transformers import (AutoTokenizer, 
    AutoModelForCausalLM, BitsAndBytesConfig)
from peft import PeftModel, PeftConfig
import os
import torch

#import ipdb; ipdb.set_trace()
cache_dir="/workspace/asr/peft/qlora"
# TODO set this cache_dir to the path where you 
# stored (or, want to store) llama2-7bhf model

lazylora_dir=os.getcwd() 
# the path that contains 'adapter_config.json' 
#     and 'adapter_model.bin'

config = PeftConfig.from_pretrained(lazylora_dir)

tokenizer = AutoTokenizer.from_pretrained(
    config.base_model_name_or_path,
    cache_dir=cache_dir,
    use_auth_token=True
)

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type='nf4',
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
    config.base_model_name_or_path,
    quantization_config=bnb_config,
    device_map="auto",
    cache_dir=cache_dir,
    use_auth_token=True
)
#model.print_trainable_parameters()
print(sum(p.numel() for p in model.parameters()))
# 3,500,412,928 -> half-size of 7B due to 4-bit loading

model = PeftModel.from_pretrained(model, lazylora_dir)
print('after adding lazy lora parameters:')
model.print_trainable_parameters()
# trainable params: 0 || all params: 3,660,359,168 || trainable%: 0.0

MMLU result:

MMLU eval result:

{"mmlu_loss": 1.9065961667247102,
 "mmlu_eval_accuracy_professional_medicine": 0.3870967741935484,
 "mmlu_eval_accuracy_college_physics": 0.45454545454545453,
 "mmlu_eval_accuracy_conceptual_physics": 0.34615384615384615,
 "mmlu_eval_accuracy_econometrics": 0.3333333333333333,
 "mmlu_eval_accuracy_high_school_chemistry": 0.45454545454545453,
 "mmlu_eval_accuracy_nutrition": 0.5151515151515151,
 "mmlu_eval_accuracy_high_school_computer_science": 0.5555555555555556,
 "mmlu_eval_accuracy_security_studies": 0.4444444444444444,
 "mmlu_eval_accuracy_world_religions": 0.6842105263157895,
 "mmlu_eval_accuracy_anatomy": 0.5,
 "mmlu_eval_accuracy_prehistory": 0.42857142857142855,
 "mmlu_eval_accuracy_high_school_government_and_politics": 0.6666666666666666,
 "mmlu_eval_accuracy_professional_accounting": 0.3225806451612903,
 "mmlu_eval_accuracy_philosophy": 0.4411764705882353,
 "mmlu_eval_accuracy_astronomy": 0.3125,
 "mmlu_eval_accuracy_medical_genetics": 0.8181818181818182,
 "mmlu_eval_accuracy_jurisprudence": 0.5454545454545454,
 "mmlu_eval_accuracy_professional_law": 0.38235294117647056,
 "mmlu_eval_accuracy_college_chemistry": 0.125,
 "mmlu_eval_accuracy_moral_disputes": 0.4473684210526316,
 "mmlu_eval_accuracy_abstract_algebra": 0.36363636363636365,
 "mmlu_eval_accuracy_computer_security": 0.5454545454545454,
 "mmlu_eval_accuracy_business_ethics": 0.5454545454545454,
 "mmlu_eval_accuracy_virology": 0.5,
 "mmlu_eval_accuracy_electrical_engineering": 0.375,
 "mmlu_eval_accuracy_high_school_biology": 0.34375,
 "mmlu_eval_accuracy_public_relations": 0.3333333333333333,
 "mmlu_eval_accuracy_high_school_physics": 0.35294117647058826,
 "mmlu_eval_accuracy_high_school_psychology": 0.65,
 "mmlu_eval_accuracy_college_computer_science": 0.5454545454545454,
 "mmlu_eval_accuracy_high_school_european_history": 0.7222222222222222,
 "mmlu_eval_accuracy_international_law": 0.8461538461538461,
 "mmlu_eval_accuracy_high_school_microeconomics": 0.2692307692307692,
 "mmlu_eval_accuracy_college_biology": 0.25,
 "mmlu_eval_accuracy_formal_logic": 0.14285714285714285,
 "mmlu_eval_accuracy_machine_learning": 0.18181818181818182,
 "mmlu_eval_accuracy_human_aging": 0.6956521739130435,
 "mmlu_eval_accuracy_logical_fallacies": 0.5555555555555556,
 "mmlu_eval_accuracy_clinical_knowledge": 0.41379310344827586,
 "mmlu_eval_accuracy_high_school_macroeconomics": 0.3488372093023256,
 "mmlu_eval_accuracy_miscellaneous": 0.5930232558139535,
 "mmlu_eval_accuracy_sociology": 0.7272727272727273,
 "mmlu_eval_accuracy_high_school_us_history": 0.6363636363636364,
 "mmlu_eval_accuracy_college_medicine": 0.4090909090909091,
 "mmlu_eval_accuracy_high_school_world_history": 0.5,
 "mmlu_eval_accuracy_marketing": 0.8,
 "mmlu_eval_accuracy_human_sexuality": 0.4166666666666667,
 "mmlu_eval_accuracy_professional_psychology": 0.36231884057971014,
 "mmlu_eval_accuracy_moral_scenarios": 0.24,
 "mmlu_eval_accuracy_college_mathematics": 0.18181818181818182,
 "mmlu_eval_accuracy_us_foreign_policy": 0.6363636363636364,
 "mmlu_eval_accuracy_high_school_geography": 0.6818181818181818,
 "mmlu_eval_accuracy_high_school_statistics": 0.34782608695652173,
 "mmlu_eval_accuracy_high_school_mathematics": 0.2413793103448276,
 "mmlu_eval_accuracy_elementary_mathematics": 0.3170731707317073,
 "mmlu_eval_accuracy_management": 0.36363636363636365,
 "mmlu_eval_accuracy_global_facts": 0.2,
 "mmlu_eval_accuracy": 0.4526436056641111}

MMLU test result:

{"mmlu_loss": 1.925738222594615,
 "mmlu_test_accuracy_business_ethics": 0.53,
 "mmlu_test_accuracy_medical_genetics": 0.53,
 "mmlu_test_accuracy_international_law": 0.628099173553719,
 "mmlu_test_accuracy_professional_law": 0.3363754889178618,
 "mmlu_test_accuracy_econometrics": 0.32456140350877194,
 "mmlu_test_accuracy_high_school_biology": 0.4806451612903226,
 "mmlu_test_accuracy_computer_security": 0.57,
 "mmlu_test_accuracy_global_facts": 0.34,
 "mmlu_test_accuracy_clinical_knowledge": 0.46037735849056605,
 "mmlu_test_accuracy_miscellaneous": 0.6347381864623244,
 "mmlu_test_accuracy_high_school_microeconomics": 0.39915966386554624,
 "mmlu_test_accuracy_public_relations": 0.5636363636363636,
 "mmlu_test_accuracy_high_school_computer_science": 0.45,
 "mmlu_test_accuracy_human_sexuality": 0.5572519083969466,
 "mmlu_test_accuracy_virology": 0.43373493975903615,
 "mmlu_test_accuracy_human_aging": 0.5695067264573991,
 "mmlu_test_accuracy_high_school_world_history": 0.6371308016877637,
 "mmlu_test_accuracy_college_medicine": 0.3699421965317919,
 "mmlu_test_accuracy_marketing": 0.6923076923076923,
 "mmlu_test_accuracy_world_religions": 0.6783625730994152,
 "mmlu_test_accuracy_college_physics": 0.23529411764705882,
 "mmlu_test_accuracy_high_school_chemistry": 0.33004926108374383,
 "mmlu_test_accuracy_elementary_mathematics": 0.2751322751322751,
 "mmlu_test_accuracy_high_school_psychology": 0.6018348623853211,
 "mmlu_test_accuracy_sociology": 0.5920398009950248,
 "mmlu_test_accuracy_astronomy": 0.4342105263157895,
 "mmlu_test_accuracy_high_school_mathematics": 0.27037037037037037,
 "mmlu_test_accuracy_high_school_us_history": 0.5343137254901961,
 "mmlu_test_accuracy_logical_fallacies": 0.49693251533742333,
 "mmlu_test_accuracy_high_school_statistics": 0.19907407407407407,
 "mmlu_test_accuracy_management": 0.5825242718446602,
 "mmlu_test_accuracy_moral_disputes": 0.5057803468208093,
 "mmlu_test_accuracy_formal_logic": 0.24603174603174602,
 "mmlu_test_accuracy_college_chemistry": 0.25,
 "mmlu_test_accuracy_college_mathematics": 0.3,
 "mmlu_test_accuracy_high_school_geography": 0.5050505050505051,
 "mmlu_test_accuracy_machine_learning": 0.35714285714285715,
 "mmlu_test_accuracy_philosophy": 0.5787781350482315,
 "mmlu_test_accuracy_college_computer_science": 0.32,
 "mmlu_test_accuracy_security_studies": 0.46938775510204084,
 "mmlu_test_accuracy_abstract_algebra": 0.27,
 "mmlu_test_accuracy_professional_psychology": 0.4526143790849673,
 "mmlu_test_accuracy_college_biology": 0.4444444444444444,
 "mmlu_test_accuracy_us_foreign_policy": 0.68,
 "mmlu_test_accuracy_professional_medicine": 0.4522058823529412,
 "mmlu_test_accuracy_prehistory": 0.48148148148148145,
 "mmlu_test_accuracy_anatomy": 0.45925925925925926,
 "mmlu_test_accuracy_moral_scenarios": 0.2346368715083799,
 "mmlu_test_accuracy_nutrition": 0.4738562091503268,
 "mmlu_test_accuracy_high_school_macroeconomics": 0.4461538461538462,
 "mmlu_test_accuracy_high_school_european_history": 0.6181818181818182,
 "mmlu_test_accuracy_jurisprudence": 0.5370370370370371,
 "mmlu_test_accuracy_professional_accounting": 0.35815602836879434,
 "mmlu_test_accuracy_high_school_government_and_politics": 0.6321243523316062,
 "mmlu_test_accuracy_high_school_physics": 0.32450331125827814,
 "mmlu_test_accuracy_electrical_engineering": 0.47586206896551725,
 "mmlu_test_accuracy_conceptual_physics": 0.3872340425531915,
 "mmlu_test_accuracy": 0.4560969792275357}

License and intended use

This lazy-lora adapter is based on Meta's LLaMA-2-7b-hf, and using the oasst1 dataset, following Guanaco.

lazy lora adapter weights are available under LLAMA-2 license. Note the use of the lazy lora adapter weights, requires access to the LLaMA model weighs. Lazy lora is based on LLaMA and therefore should be used according to the LLaMA license.

Risks and Biases

The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. The model was trained on various public datasets; it is possible that this model could generate lewd, biased, or otherwise offensive outputs.