Model Information
Note: This model is a Schnell based model, but it requires CFG scale 3.5 or higher and 20 steps or more. It needs to be used with clip_l_sumeshi_f1s.
My English is terrible, so I use translation tools.
Description
Sumeshi flux.1 S is an experimental anime model to verify if de-distilling and enabling CFG will function. You can use a negative prompt which works to some extent. Since this model uses CFG, it takes about twice as long to generate compared to a regular FLUX model, even with the same number of steps. The output is blurred and the style varies depending on the prompt, perhaps because the model has not been fully trained.
Usage
- Resolution: Like other Flux models
- CFG Scale: 3.5 ~ 7 ( Scale1 does not generate decent outputs. )
- Steps: 20 ~ 60 (Not around 4 steps)
- (Distilled) Guidance Scale: 0 (Does not work due to Schnell-based model)
- sampler: Euler
- scheduler: Simple, Beta
Prompt Format (from Kohaku-XL-Epsilon)
<1girl/1boy/1other/...>, <character>, <series>, <artists>, <general tags>, <quality tags>, <year tags>, <meta tags>, <rating tags>
Due to the small amount of training, the <character><series><artists> tags are almost non-functional. As training is focused on girl characters, it may not generate boy or other non-persons well. Since the dataset was created using hakubooru, the prompt format will be the same as the KohakuXL format. However, based on experiments, it is not strictly necessary to follow this format, as it interprets meaning to some extent even in natural language.
Special Tags
- Quality Tags: masterpiece, best quality, great quality, good quality, normal quality, low quality, worst quality
- Rating Tags: safe, sensitive, nsfw, explicit
- Date Tags: newest, recent, mid, early, old
Training
Dataset Preparation
I used hakubooru-based custom scripts.
- Exclude Tags:
traditional_media, photo_(medium), scan, animated, animated_gif, lowres, non-web_source, variant_set, tall image, duplicate, pixel-perfect_duplicate - Minimum Post ID: 1,000,000
Key Addition
I added tensors filled with zeros with the guidance_in key to the Schnell model. This tensor is adjusted to the shape of the corresponding key in Dev, as inferred from flux/src/flux/model.py. This is because the trainer did not work properly when these keys were missing if the model name did not include 'schnell'. Since it is filled with zeros, I understand that guidance, like in the Schnell model, will not function. Due to my limited skills and the forceful addition, I'm not sure if this was the correct approach.
Training Details
Basically, the assumption is that the more we learn, the more the network will be reconstructed, the more the distillation will be lifted, and the more CFGs will be available.
- Training Hardware: A single RTX 4090
- Method: LoRA training and merging the results
- Training Script: sd-scripts
- Basic Settings:
accelerate launch --num_cpu_threads_per_process 4 flux_train_network.py --network_module networks.lora_flux --sdpa --gradient_checkpointing --cache_latents --cache_latents_to_disk --cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk --max_data_loader_n_workers 1 --save_model_as "safetensors" --mixed_precision "bf16" --fp8_base --save_precision "bf16" --full_bf16 --min_bucket_reso 320 --max_bucket_reso 1536 --seed 1 --max_train_epochs 1 --keep_tokens_separator "|||" --network_dim 32 --network_alpha 32 --unet_lr 1e-4 --text_encoder_lr 5e-5 --train_batch_size 3 --gradient_accumulation_steps 2 --optimizer_type adamw8bit --lr_scheduler="constant_with_warmup" --lr_warmup_steps 100 --vae_batch_size 8 --cache_info --guidance_scale 7 --timestep_sampling shift --model_prediction_type raw --discrete_flow_shift 3.2 --loss_type l2 --highvram
3,893images (res512 bs4 / res768 bs2 / res1024 bs1, acc4) 1epoch
60,000images (res768 bs3 acc2) 1epoch
36,000images (res1024 bs1 acc3) 1epoch
3,000images (res1024 bs1 acc1) 1epoch
18,000images (res1024 bs1 acc3) 1epoch
merged into model and CLIP_L
693images (res1024 bs1 acc3) 1epoch
693images (res1024 bs1 acc3 warmup50) 1epoch
693images (res1024 bs1 acc3 warmup50) 10ecpohs
693images (res1024 bs1 acc3 warmup50) 15ecpohs
merged into model and CLIP_L
543images (res1024 bs1 acc3 warmup50 --optimizer_args "betas=0.9,0.95" "eps=1e-06" "weight_decay=0.1" --caption_dropout_rate 0.1 --shuffle_caption --network_train_unet_only) 20epochs
merged into model and CLIP_L
Resources (License)
- FLUX.1-schnell (Apache2.0)
- danbooru2023-webp-4Mpixel (MIT)
- danbooru2023-metadata-database (MIT)
Acknowledgements
- black-forest-labs: Thanks for publishing a great open source model.
- kohya-ss: Thanks for publishing the essential training scripts and for the quick updates.
- Kohaku-Blueleaf: Thanks for the extensive publication of the scripts for the dataset and the various training conditions.