See axolotl config

axolotl version: 0.4.1

adapter: lora
base_model: fxmarty/really-tiny-falcon-testing
bf16: auto
chat_template: llama3
dataset_prepared_path: null
datasets:
- data_files:
  - e8ef6edb66e20da7_train_data.json
  ds_type: json
  format: custom
  path: /workspace/input_data/e8ef6edb66e20da7_train_data.json
  type:
    field_input: input
    field_instruction: instruction
    field_output: output
    format: '{instruction} {input}'
    no_input_format: '{instruction}'
    system_format: '{system}'
    system_prompt: ''
debug: null
deepspeed: null
do_eval: true
early_stopping_patience: 3
eval_batch_size: 4
eval_max_new_tokens: 128
eval_steps: 500
evals_per_epoch: null
flash_attention: false
fp16: false
fsdp: null
fsdp_config: null
gradient_accumulation_steps: 8
gradient_checkpointing: true
group_by_length: true
hub_model_id: lesso09/7f4d0272-839f-452d-bd43-dabda63396ab
hub_repo: null
hub_strategy: checkpoint
hub_token: null
learning_rate: 0.000209
load_in_4bit: false
load_in_8bit: false
local_rank: null
logging_steps: 50
lora_alpha: 128
lora_dropout: 0.15
lora_fan_in_fan_out: null
lora_model_dir: null
lora_r: 64
lora_target_linear: true
lr_scheduler: cosine
max_grad_norm: 1.0
max_steps: 25000
micro_batch_size: 4
mlflow_experiment_name: /tmp/e8ef6edb66e20da7_train_data.json
model_type: AutoModelForCausalLM
num_epochs: 10
optimizer: adamw_torch_fused
output_dir: miner_id_24
pad_to_sequence_len: true
resume_from_checkpoint: null
s2_attention: null
sample_packing: false
save_steps: 500
saves_per_epoch: null
seed: 90
sequence_len: 1024
strict: false
tf32: true
tokenizer_type: AutoTokenizer
train_on_inputs: false
trust_remote_code: true
val_set_size: 0.05
wandb_entity: null
wandb_mode: online
wandb_name: 6f911363-8c0f-4331-9742-a2fb57ee53b7
wandb_project: 09a
wandb_run: your_name
wandb_runid: 6f911363-8c0f-4331-9742-a2fb57ee53b7
warmup_steps: 100
weight_decay: 0.0
xformers_attention: null

7f4d0272-839f-452d-bd43-dabda63396ab

This model is a fine-tuned version of fxmarty/really-tiny-falcon-testing on the None dataset. It achieves the following results on the evaluation set:

• Loss: 10.6611

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.000209
• train_batch_size: 4
• eval_batch_size: 4
• seed: 90
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• training_steps: 19868

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0005	1	11.0932
86.0282	0.2517	500	10.7382
85.8172	0.5033	1000	10.7085
85.7517	0.7550	1500	10.6967
86.061	1.0067	2000	10.6913
85.6738	1.2583	2500	10.6858
85.6973	1.5100	3000	10.6809
85.606	1.7617	3500	10.6779
85.6544	2.0133	4000	10.6755
85.5822	2.2650	4500	10.6731
85.5585	2.5167	5000	10.6715
85.4748	2.7683	5500	10.6698
85.595	3.0200	6000	10.6687
85.5199	3.2717	6500	10.6679
85.4948	3.5233	7000	10.6668
85.4707	3.7750	7500	10.6663
85.4624	4.0267	8000	10.6661
85.5253	4.2783	8500	10.6653
85.4998	4.5300	9000	10.6647
85.4906	4.7817	9500	10.6645
85.452	5.0333	10000	10.6634
85.4633	5.2850	10500	10.6634
85.4769	5.5367	11000	10.6628
85.4446	5.7883	11500	10.6628
85.5044	6.0400	12000	10.6627
85.512	6.2917	12500	10.6621
85.452	6.5433	13000	10.6623
85.4906	6.7950	13500	10.6617
85.4964	7.0467	14000	10.6618
85.4633	7.2984	14500	10.6616
85.4613	7.5500	15000	10.6616
85.4375	7.8017	15500	10.6616
85.4846	8.0534	16000	10.6613
85.449	8.3050	16500	10.6614
85.4237	8.5567	17000	10.6613
85.4712	8.8084	17500	10.6611
85.4472	9.0600	18000	10.6611
85.4579	9.3117	18500	10.6610
85.4644	9.5634	19000	10.6611
85.4548	9.8150	19500	10.6611

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1