Textual Inversion
Textual Inversion is a training method for personalizing models by learning new text embeddings from a few example images. The file produced from training is extremely small (a few KBs) and the new embeddings can be loaded into the text encoder.
TextualInversionLoaderMixin provides a function for loading Textual Inversion embeddings from Diffusers and Automatic1111 into the text encoder and loading a special token to activate the embeddings.
To learn more about how to load Textual Inversion embeddings, see the Textual Inversion loading guide.
TextualInversionLoaderMixin
Load Textual Inversion tokens and embeddings to the tokenizer and text encoder.
load_textual_inversion
< source >( pretrained_model_name_or_path: typing.Union[str, typing.List[str], typing.Dict[str, torch.Tensor], typing.List[typing.Dict[str, torch.Tensor]]] token: typing.Union[str, typing.List[str], NoneType] = None tokenizer: typing.Optional[ForwardRef('PreTrainedTokenizer')] = None text_encoder: typing.Optional[ForwardRef('PreTrainedModel')] = None **kwargs )
Parameters
- pretrained_model_name_or_path (
stroros.PathLikeorList[str or os.PathLike]orDictorList[Dict]) — Can be either one of the following or a list of them:- A string, the model id (for example
sd-concepts-library/low-poly-hd-logos-icons) of a pretrained model hosted on the Hub. - A path to a directory (for example
./my_text_inversion_directory/) containing the textual inversion weights. - A path to a file (for example
./my_text_inversions.pt) containing textual inversion weights. - A torch state dict.
- A string, the model id (for example
- token (
strorList[str], optional) — Override the token to use for the textual inversion weights. Ifpretrained_model_name_or_pathis a list, thentokenmust also be a list of equal length. - text_encoder (CLIPTextModel, optional) — Frozen text-encoder (clip-vit-large-patch14). If not specified, function will take self.tokenizer.
- tokenizer (CLIPTokenizer, optional) —
A
CLIPTokenizerto tokenize text. If not specified, function will take self.tokenizer. - weight_name (
str, optional) — Name of a custom weight file. This should be used when:- The saved textual inversion file is in 🤗 Diffusers format, but was saved under a specific weight
name such as
text_inv.bin. - The saved textual inversion file is in the Automatic1111 format.
- The saved textual inversion file is in 🤗 Diffusers format, but was saved under a specific weight
name such as
- cache_dir (
Union[str, os.PathLike], optional) — Path to a directory where a downloaded pretrained model configuration is cached if the standard cache is not used. - force_download (
bool, optional, defaults toFalse) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. - proxies (
Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, for example,{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request. - local_files_only (
bool, optional, defaults toFalse) — Whether to only load local model weights and configuration files or not. If set toTrue, the model won’t be downloaded from the Hub. - token (
stror bool, optional) — The token to use as HTTP bearer authorization for remote files. IfTrue, the token generated fromdiffusers-cli login(stored in~/.huggingface) is used. - revision (
str, optional, defaults to"main") — The specific model version to use. It can be a branch name, a tag name, a commit id, or any identifier allowed by Git. - subfolder (
str, optional, defaults to"") — The subfolder location of a model file within a larger model repository on the Hub or locally. - mirror (
str, optional) — Mirror source to resolve accessibility issues if you’re downloading a model in China. We do not guarantee the timeliness or safety of the source, and you should refer to the mirror site for more information.
Load Textual Inversion embeddings into the text encoder of StableDiffusionPipeline (both 🤗 Diffusers and Automatic1111 formats are supported).
Example:
To load a Textual Inversion embedding vector in 🤗 Diffusers format:
from diffusers import StableDiffusionPipeline
import torch
model_id = "runwayml/stable-diffusion-v1-5"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16).to("cuda")
pipe.load_textual_inversion("sd-concepts-library/cat-toy")
prompt = "A <cat-toy> backpack"
image = pipe(prompt, num_inference_steps=50).images[0]
image.save("cat-backpack.png")To load a Textual Inversion embedding vector in Automatic1111 format, make sure to download the vector first (for example from civitAI) and then load the vector
locally:
from diffusers import StableDiffusionPipeline
import torch
model_id = "runwayml/stable-diffusion-v1-5"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16).to("cuda")
pipe.load_textual_inversion("./charturnerv2.pt", token="charturnerv2")
prompt = "charturnerv2, multiple views of the same character in the same outfit, a character turnaround of a woman wearing a black jacket and red shirt, best quality, intricate details."
image = pipe(prompt, num_inference_steps=50).images[0]
image.save("character.png")maybe_convert_prompt
< source >( prompt: typing.Union[str, typing.List[str]] tokenizer: PreTrainedTokenizer ) → str or list of str
Processes prompts that include a special token corresponding to a multi-vector textual inversion embedding to be replaced with multiple special tokens each corresponding to one of the vectors. If the prompt has no textual inversion token or if the textual inversion token is a single vector, the input prompt is returned.
unload_textual_inversion
< source >( tokens: typing.Union[str, typing.List[str], NoneType] = None tokenizer: typing.Optional[ForwardRef('PreTrainedTokenizer')] = None text_encoder: typing.Optional[ForwardRef('PreTrainedModel')] = None )
Unload Textual Inversion embeddings from the text encoder of StableDiffusionPipeline
Example:
from diffusers import AutoPipelineForText2Image
import torch
pipeline = AutoPipelineForText2Image.from_pretrained("runwayml/stable-diffusion-v1-5")
# Example 1
pipeline.load_textual_inversion("sd-concepts-library/gta5-artwork")
pipeline.load_textual_inversion("sd-concepts-library/moeb-style")
# Remove all token embeddings
pipeline.unload_textual_inversion()
# Example 2
pipeline.load_textual_inversion("sd-concepts-library/moeb-style")
pipeline.load_textual_inversion("sd-concepts-library/gta5-artwork")
# Remove just one token
pipeline.unload_textual_inversion("<moe-bius>")
# Example 3: unload from SDXL
pipeline = AutoPipelineForText2Image.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0")
embedding_path = hf_hub_download(
repo_id="linoyts/web_y2k", filename="web_y2k_emb.safetensors", repo_type="model"
)
# load embeddings to the text encoders
state_dict = load_file(embedding_path)
# load embeddings of text_encoder 1 (CLIP ViT-L/14)
pipeline.load_textual_inversion(
state_dict["clip_l"],
tokens=["<s0>", "<s1>"],
text_encoder=pipeline.text_encoder,
tokenizer=pipeline.tokenizer,
)
# load embeddings of text_encoder 2 (CLIP ViT-G/14)
pipeline.load_textual_inversion(
state_dict["clip_g"],
tokens=["<s0>", "<s1>"],
text_encoder=pipeline.text_encoder_2,
tokenizer=pipeline.tokenizer_2,
)
# Unload explicitly from both text encoders and tokenizers
pipeline.unload_textual_inversion(
tokens=["<s0>", "<s1>"], text_encoder=pipeline.text_encoder, tokenizer=pipeline.tokenizer
)
pipeline.unload_textual_inversion(
tokens=["<s0>", "<s1>"], text_encoder=pipeline.text_encoder_2, tokenizer=pipeline.tokenizer_2
)