phi-1_5-finetuned-SQL-2

This model is a fine-tuned version of microsoft/phi-1_5 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 2.1014

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• training_steps: 20000

Training results

Training Loss	Epoch	Step	Validation Loss
2.2954	3.2	1000	1.8424
1.747	6.4	2000	1.7355
1.6037	9.6	3000	1.6513
1.4616	12.8	4000	1.6125
1.3183	16.0	5000	1.6114
1.1743	19.2	6000	1.6670
1.0421	22.4	7000	1.7143
0.9229	25.6	8000	1.7434
0.8166	28.8	9000	1.8055
0.7256	32.0	10000	1.8523
0.6435	35.2	11000	1.9160
0.5779	38.4	12000	1.9352
0.5209	41.6	13000	1.9740
0.4739	44.8	14000	2.0014
0.4353	48.0	15000	2.0192
0.4036	51.2	16000	2.0506
0.3814	54.4	17000	2.0678
0.3626	57.6	18000	2.0861
0.3504	60.8	19000	2.0940
0.3391	64.0	20000	2.1014

Framework versions

• Transformers 4.34.1
• Pytorch 2.1.0+cu118
• Datasets 2.14.6
• Tokenizers 0.14.1