Transformers documentation

Image Processor

Transformers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v4.48.2).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

Image Processor

An image processor is in charge of preparing input features for vision models and post processing their outputs. This includes transformations such as resizing, normalization, and conversion to PyTorch, TensorFlow, Flax and Numpy tensors. It may also include model specific post-processing such as converting logits to segmentation masks.

Fast image processors are available for a few models and more will be added in the future. They are based on the torchvision library and provide a significant speed-up, especially when processing on GPU. They have the same API as the base image processors and can be used as drop-in replacements. To use a fast image processor, you need to install the torchvision library, and set the use_fast argument to True when instantiating the image processor:

from transformers import AutoImageProcessor

processor = AutoImageProcessor.from_pretrained("facebook/detr-resnet-50", use_fast=True)

Note that use_fast will be set to True by default in a future release.

When using a fast image processor, you can also set the device argument to specify the device on which the processing should be done. By default, the processing is done on the same device as the inputs if the inputs are tensors, or on the CPU otherwise.

from torchvision.io import read_image
from transformers import DetrImageProcessorFast

images = read_image("image.jpg")
processor = DetrImageProcessorFast.from_pretrained("facebook/detr-resnet-50")
images_processed = processor(images, return_tensors="pt", device="cuda")

Here are some speed comparisons between the base and fast image processors for the DETR and RT-DETR models, and how they impact overall inference time:

These benchmarks were run on an AWS EC2 g5.2xlarge instance, utilizing an NVIDIA A10G Tensor Core GPU.

ImageProcessingMixin

class transformers.ImageProcessingMixin

( **kwargs )

This is an image processor mixin used to provide saving/loading functionality for sequential and image feature extractors.

from_pretrained

( pretrained_model_name_or_path: typing.Union[str, os.PathLike] cache_dir: typing.Union[str, os.PathLike, NoneType] = None force_download: bool = False local_files_only: bool = False token: typing.Union[bool, str, NoneType] = None revision: str = 'main' **kwargs )

Parameters

pretrained_model_name_or_path (str or os.PathLike) — This can be either:
- a string, the model id of a pretrained image_processor hosted inside a model repo on huggingface.co.
- a path to a directory containing a image processor file saved using the save_pretrained() method, e.g., ./my_model_directory/.
- a path or url to a saved image processor JSON file, e.g., ./my_model_directory/preprocessor_config.json.
cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model image processor should be cached if the standard cache should not be used.
force_download (bool, optional, defaults to False) — Whether or not to force to (re-)download the image processor files and override the cached versions if they exist.
resume_download — Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers.
proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
token (str or bool, optional) — The token to use as HTTP bearer authorization for remote files. If True, or not specified, will use the token generated when running huggingface-cli login (stored in ~/.huggingface).
revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.

Instantiate a type of ImageProcessingMixin from an image processor.

Examples:

# We can't instantiate directly the base class *ImageProcessingMixin* so let's show the examples on a
# derived class: *CLIPImageProcessor*
image_processor = CLIPImageProcessor.from_pretrained(
    "openai/clip-vit-base-patch32"
)  # Download image_processing_config from huggingface.co and cache.
image_processor = CLIPImageProcessor.from_pretrained(
    "./test/saved_model/"
)  # E.g. image processor (or model) was saved using *save_pretrained('./test/saved_model/')*
image_processor = CLIPImageProcessor.from_pretrained("./test/saved_model/preprocessor_config.json")
image_processor = CLIPImageProcessor.from_pretrained(
    "openai/clip-vit-base-patch32", do_normalize=False, foo=False
)
assert image_processor.do_normalize is False
image_processor, unused_kwargs = CLIPImageProcessor.from_pretrained(
    "openai/clip-vit-base-patch32", do_normalize=False, foo=False, return_unused_kwargs=True
)
assert image_processor.do_normalize is False
assert unused_kwargs == {"foo": False}

save_pretrained

( save_directory: typing.Union[str, os.PathLike] push_to_hub: bool = False **kwargs )

Parameters

save_directory (str or os.PathLike) — Directory where the image processor JSON file will be saved (will be created if it does not exist).
push_to_hub (bool, optional, defaults to False) — Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with repo_id (will default to the name of save_directory in your namespace).
kwargs (Dict[str, Any], optional) — Additional key word arguments passed along to the push_to_hub() method.

Save an image processor object to the directory save_directory, so that it can be re-loaded using the from_pretrained() class method.

BatchFeature

class transformers.BatchFeature

( data: typing.Optional[typing.Dict[str, typing.Any]] = None tensor_type: typing.Union[NoneType, str, transformers.utils.generic.TensorType] = None )

Parameters

data (dict, optional) — Dictionary of lists/arrays/tensors returned by the call/pad methods (‘input_values’, ‘attention_mask’, etc.).
tensor_type (Union[None, str, TensorType], optional) — You can give a tensor_type here to convert the lists of integers in PyTorch/TensorFlow/Numpy Tensors at initialization.

Holds the output of the pad() and feature extractor specific __call__ methods.

This class is derived from a python dictionary and can be used as a dictionary.

convert_to_tensors

( tensor_type: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None )

Parameters

tensor_type (str or TensorType, optional) — The type of tensors to use. If str, should be one of the values of the enum TensorType. If None, no modification is done.

Convert the inner content to tensors.

to

( *args **kwargs ) → BatchFeature

Parameters

args (Tuple) — Will be passed to the to(...) function of the tensors.
kwargs (Dict, optional) — Will be passed to the to(...) function of the tensors. To enable asynchronous data transfer, set the non_blocking flag in kwargs (defaults to False).

Returns

The same instance after modification.

Send all values to device by calling v.to(*args, **kwargs) (PyTorch only). This should support casting in different dtypes and sending the BatchFeature to a different device.

BaseImageProcessor

class transformers.BaseImageProcessor

( **kwargs )

center_crop

( image: ndarray size: typing.Dict[str, int] data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None **kwargs )

Parameters

image (np.ndarray) — Image to center crop.
size (Dict[str, int]) — Size of the output image.
data_format (str or ChannelDimension, optional) — The channel dimension format for the output image. If unset, the channel dimension format of the input image is used. Can be one of:
- "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
- "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
input_data_format (ChannelDimension or str, optional) — The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of:
- "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
- "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.

Center crop an image to (size["height"], size["width"]). If the input size is smaller than crop_size along any edge, the image is padded with 0’s and then center cropped.

normalize

( image: ndarray mean: typing.Union[float, typing.Iterable[float]] std: typing.Union[float, typing.Iterable[float]] data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None **kwargs ) → np.ndarray

Parameters

image (np.ndarray) — Image to normalize.
mean (float or Iterable[float]) — Image mean to use for normalization.
std (float or Iterable[float]) — Image standard deviation to use for normalization.
data_format (str or ChannelDimension, optional) — The channel dimension format for the output image. If unset, the channel dimension format of the input image is used. Can be one of:
- "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
- "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
input_data_format (ChannelDimension or str, optional) — The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of:
- "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
- "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.

Returns

np.ndarray

The normalized image.

Normalize an image. image = (image - image_mean) / image_std.

rescale

( image: ndarray scale: float data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None **kwargs ) → np.ndarray

Parameters

image (np.ndarray) — Image to rescale.
scale (float) — The scaling factor to rescale pixel values by.
data_format (str or ChannelDimension, optional) — The channel dimension format for the output image. If unset, the channel dimension format of the input image is used. Can be one of:
- "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
- "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
input_data_format (ChannelDimension or str, optional) — The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of:
- "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
- "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.

Returns

np.ndarray

The rescaled image.

Rescale an image by a scale factor. image = image * scale.

BaseImageProcessorFast

class transformers.BaseImageProcessorFast

( **kwargs: typing_extensions.Unpack[transformers.image_processing_utils_fast.DefaultFastImageProcessorInitKwargs] )

Parameters

do_resize (bool, optional, defaults to self.do_resize) — Whether to resize the image’s (height, width) dimensions to the specified size. Can be overridden by the do_resize parameter in the preprocess method.
size (dict, optional, defaults to self.size) — Size of the output image after resizing. Can be overridden by the size parameter in the preprocess method.
default_to_square (bool, optional, defaults to self.default_to_square) — Whether to default to a square image when resizing, if size is an int.
resample (PILImageResampling, optional, defaults to self.resample) — Resampling filter to use if resizing the image. Only has an effect if do_resize is set to True. Can be overridden by the resample parameter in the preprocess method.
do_center_crop (bool, optional, defaults to self.do_center_crop) — Whether to center crop the image to the specified crop_size. Can be overridden by do_center_crop in the preprocess method.
crop_size (Dict[str, int] optional, defaults to self.crop_size) — Size of the output image after applying center_crop. Can be overridden by crop_size in the preprocess method.
do_rescale (bool, optional, defaults to self.do_rescale) — Whether to rescale the image by the specified scale rescale_factor. Can be overridden by the do_rescale parameter in the preprocess method.
rescale_factor (int or float, optional, defaults to self.rescale_factor) — Scale factor to use if rescaling the image. Only has an effect if do_rescale is set to True. Can be overridden by the rescale_factor parameter in the preprocess method.
do_normalize (bool, optional, defaults to self.do_normalize) — Whether to normalize the image. Can be overridden by the do_normalize parameter in the preprocess method. Can be overridden by the do_normalize parameter in the preprocess method.
image_mean (float or List[float], optional, defaults to self.image_mean) — Mean to use if normalizing the image. This is a float or list of floats the length of the number of channels in the image. Can be overridden by the image_mean parameter in the preprocess method. Can be overridden by the image_mean parameter in the preprocess method.
image_std (float or List[float], optional, defaults to self.image_std) — Standard deviation to use if normalizing the image. This is a float or list of floats the length of the number of channels in the image. Can be overridden by the image_std parameter in the preprocess method. Can be overridden by the image_std parameter in the preprocess method.
do_convert_rgb (bool, optional, defaults to self.image_std) — Whether to convert the image to RGB.

Constructs a fast base image processor.

center_crop

( image: torch.Tensor size: typing.Dict[str, int] **kwargs ) → torch.Tensor

Parameters

image ("torch.Tensor") — Image to center crop.
size (Dict[str, int]) — Size of the output image.

Returns

torch.Tensor

The center cropped image.

Center crop an image to (size["height"], size["width"]). If the input size is smaller than crop_size along any edge, the image is padded with 0’s and then center cropped.

convert_to_rgb

( image: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]] ) → ImageInput

Parameters

image (ImageInput) — The image to convert.

Returns

ImageInput

The converted image.

Converts an image to RGB format. Only converts if the image is of type PIL.Image.Image, otherwise returns the image as is.

normalize

( image: torch.Tensor mean: typing.Union[float, typing.Iterable[float]] std: typing.Union[float, typing.Iterable[float]] **kwargs ) → torch.Tensor

Parameters

image (torch.Tensor) — Image to normalize.
mean (torch.Tensor, float or Iterable[float]) — Image mean to use for normalization.
std (torch.Tensor, float or Iterable[float]) — Image standard deviation to use for normalization.

Returns

torch.Tensor

The normalized image.

Normalize an image. image = (image - image_mean) / image_std.

preprocess

( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]] **kwargs: typing_extensions.Unpack[transformers.image_processing_utils_fast.DefaultFastImageProcessorPreprocessKwargs] )

Parameters

images (ImageInput) — Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set do_rescale=False.
do_resize (bool, optional, defaults to self.do_resize) — Whether to resize the image.
size (Dict[str, int], optional, defaults to self.size) — Describes the maximum input dimensions to the model.
resample (PILImageResampling or InterpolationMode, optional, defaults to self.resample) — Resampling filter to use if resizing the image. This can be one of the enum PILImageResampling. Only has an effect if do_resize is set to True.
do_center_crop (bool, optional, defaults to self.do_center_crop) — Whether to center crop the image.
crop_size (Dict[str, int], optional, defaults to self.crop_size) — Size of the output image after applying center_crop.
do_rescale (bool, optional, defaults to self.do_rescale) — Whether to rescale the image.
rescale_factor (float, optional, defaults to self.rescale_factor) — Rescale factor to rescale the image by if do_rescale is set to True.
do_normalize (bool, optional, defaults to self.do_normalize) — Whether to normalize the image.
image_mean (float or List[float], optional, defaults to self.image_mean) — Image mean to use for normalization. Only has an effect if do_normalize is set to True.
image_std (float or List[float], optional, defaults to self.image_std) — Image standard deviation to use for normalization. Only has an effect if do_normalize is set to True.
do_convert_rgb (bool, optional, defaults to self.do_convert_rgb) — Whether to convert the image to RGB.
return_tensors (str or TensorType, optional) — Returns stacked tensors if set to `pt, otherwise returns a list of tensors.
data_format (ChannelDimension or str, optional, defaults to ChannelDimension.FIRST) — The channel dimension format for the output image. Can be one of:
- "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
- "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
- Unset: Use the channel dimension format of the input image.
input_data_format (ChannelDimension or str, optional) — The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of:
- "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
- "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
- "none" or ChannelDimension.NONE: image in (height, width) format.
device (torch.device, optional) — The device to process the images on. If unset, the device is inferred from the input images.

Preprocess an image or batch of images.

rescale

( image: torch.Tensor scale: float **kwargs ) → torch.Tensor

Parameters

image (torch.Tensor) — Image to rescale.
scale (float) — The scaling factor to rescale pixel values by.

Returns

torch.Tensor

The rescaled image.

Rescale an image by a scale factor. image = image * scale.

rescale_and_normalize

( images: torch.Tensor do_rescale: bool rescale_factor: float do_normalize: bool image_mean: typing.Union[float, typing.List[float]] image_std: typing.Union[float, typing.List[float]] )

Rescale and normalize images.

resize

( image: torch.Tensor size: SizeDict interpolation: F.InterpolationMode = None **kwargs ) → torch.Tensor

Parameters

image (torch.Tensor) — Image to resize.
size (SizeDict) — Dictionary in the format {"height": int, "width": int} specifying the size of the output image.
resample (InterpolationMode, optional, defaults to InterpolationMode.BILINEAR) — InterpolationMode filter to use when resizing the image e.g. InterpolationMode.BICUBIC.

Returns

torch.Tensor

The resized image.

Resize an image to (size["height"], size["width"]).

< > Update on GitHub

←Feature Extractor ALBERT→