Parameters

• suffix (str, optional, defaults to </w>) — The suffix that was used to caracterize an end-of-word. This suffix will be replaced by whitespaces during the decoding

Parameters

• pad_token (str, optional, defaults to <pad>) — The pad token used by CTC to delimit a new token.
• word_delimiter_token (str, optional, defaults to |) — The word delimiter token. It will be replaced by a
• cleanup (bool, optional, defaults to True) — Whether to cleanup some tokenization artifacts. Mainly spaces before punctuation, and some abbreviated english forms.

Parameters

• replacement (str, optional, defaults to ▁) — The replacement character. Must be exactly one character. By default we use the ▁ (U+2581) meta symbol (Same as in SentencePiece).
• prepend_scheme (str, optional, defaults to "always") — Whether to add a space to the first word if there isn’t already one. This lets us treat hello exactly like say hello. Choices: “always”, “never”, “first”. First means the space is only added on the first token (relevant when special tokens are used or other pre_tokenizer are used).

Parameters

• prefix (str, optional, defaults to ##) — The prefix to use for subwords that are not a beginning-of-word
• cleanup (bool, optional, defaults to True) — Whether to cleanup some tokenization artifacts. Mainly spaces before punctuation, and some abbreviated english forms.