Quantized version, credits to afrideva https://huggingface.co./afrideva/phi-1_5_dolly_instruction_polish-GGUF

microsoft/phi_1.5 finetuned on s3nh/dolly_instruction_polish.

Finetuned with QLora, provided version is adapter merged with base model. Load in 4 bit, sequence length set to 1024.

axolotl config

base_model: microsoft/phi-2
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
is_llama_derived_model: false
trust_remote_code: true

load_in_8bit: false
load_in_4bit: true
strict: false

datasets:
  - path: s3nh/alpaca-dolly-instruction-only-polish
    type: alpaca

dataset_prepared_path:
val_set_size: 0.05
output_dir: ./phi-2-sft-out

sequence_len: 1024
sample_packing: false  # not CURRENTLY compatible with LoRAs
pad_to_sequence_len:

adapter: qlora
lora_model_dir:
lora_r: 64
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 1
micro_batch_size: 1
num_epochs: 4
optimizer: adamw_torch
adam_beta2: 0.95
adam_epsilon: 0.00001
max_grad_norm: 1.0
lr_scheduler: cosine
learning_rate: 0.000003

train_on_inputs: false
group_by_length: true
bf16: true
fp16: false
tf32: true

gradient_checkpointing:
early_stopping_patience:
resume_from_checkpoint: false
local_rank:
logging_steps: 100
xformers_attention:
flash_attention: true

warmup_steps: 10
evals_per_epoch: 4
saves_per_epoch:
save_strategy: steps
save_steps: 5000
debug:
deepspeed:
weight_decay: 0.1
fsdp:
fsdp_config:
resize_token_embeddings_to_32x: true
special_tokens:
  bos_token: "<|endoftext|>"
  eos_token: "<|endoftext|>"
  unk_token: "<|endoftext|>"
  pad_token: "<|endoftext|>"
Downloads last month
17
Safetensors
Model size
1.42B params
Tensor type
FP16
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for s3nh/phi-1_5_dolly_instruction_polish

Quantizations
1 model

Collection including s3nh/phi-1_5_dolly_instruction_polish