|
--- |
|
library_name: diffusers |
|
license: other |
|
license_name: flux-1-dev-non-commercial-license |
|
license_link: LICENSE.md |
|
--- |
|
|
|
## Running Flux.1-dev under 12GBs |
|
|
|
This repository contains the mixed int8 params for the T5 and transformer of Flux.1-Dev. |
|
|
|
This is how the checkpoints were obtained: |
|
|
|
```python |
|
import torch |
|
from diffusers import BitsAndBytesConfig, FluxTransformer2DModel |
|
from transformers import T5EncoderModel, BitsAndBytesConfig |
|
from huggingface_hub import create_repo, upload_folder |
|
import tempfile |
|
|
|
model_id = "black-forest-labs/FLUX.1-dev" |
|
|
|
config = BitsAndBytesConfig(load_in_8bit=True) |
|
transformer = FluxTransformer2DModel.from_pretrained( |
|
model_id, subfolder="transformer", quantization_config=config, torch_dtype=torch.float16 |
|
) |
|
|
|
config = BitsAndBytesConfig(load_in_8bit=True) |
|
t5 = T5EncoderModel.from_pretrained( |
|
model_id, subfolder="text_encoder_2", quantization_config=config, torch_dtype=torch.float16 |
|
) |
|
|
|
repo_id = create_repo("sayakpaul/flux.1-dev-int8-pkg", exist_ok=True).repo_id |
|
|
|
with tempfile.TemporaryDirectory() as tmpdirname: |
|
transformer.save_pretrained(tmpdirname) |
|
upload_folder(repo_id=repo_id, folder_path=tmpdirname, path_in_repo="transformer") |
|
|
|
with tempfile.TemporaryDirectory() as tmpdirname: |
|
t5.save_pretrained(tmpdirname) |
|
upload_folder(repo_id=repo_id, folder_path=tmpdirname, path_in_repo="text_encoder_2") |
|
``` |
|
|
|
Respective `diffusers` PR: https://github.com/huggingface/diffusers/pull/9213/. |
|
|
|
> [!NOTE] |
|
> The checkpoints of this repository were optimized to run on a T4 notebook. More specifically, the compute datatype of the quantized checkpoints was kept to FP16. In practice, if you have a GPU card that supports BF16, you should change the compute datatype to BF16 (`bnb_4bit_compute_dtype`). |
