#
BARThez

##
Overview

BARThez モデルは、Moussa Kamal Eddine、Antoine J.-P によって BARThez: a Skilled Pretrained French Sequence-to-Sequence Model で提案されました。ティクシエ、ミカリス・ヴァジルジャンニス、10月23日、 2020年。

論文の要約:

帰納的転移学習は、自己教師あり学習によって可能になり、自然言語処理全体を実行します。 (NLP) 分野は、BERT や BART などのモデルにより、無数の自然言語に新たな最先端技術を確立し、嵐を巻き起こしています。 タスクを理解すること。いくつかの注目すべき例外はありますが、利用可能なモデルと研究のほとんどは、 英語を対象に実施されました。この作品では、フランス語用の最初の BART モデルである BARTez を紹介します。 （我々の知る限りに）。 BARThez は、過去の研究から得た非常に大規模な単一言語フランス語コーパスで事前トレーニングされました BART の摂動スキームに合わせて調整しました。既存の BERT ベースのフランス語モデルとは異なり、 CamemBERT と FlauBERT、BARThez は、エンコーダだけでなく、 そのデコーダは事前トレーニングされています。 FLUE ベンチマークからの識別タスクに加えて、BARThez を新しい評価に基づいて評価します。 この論文とともにリリースする要約データセット、OrangeSum。また、すでに行われている事前トレーニングも継続します。 BARTHez のコーパス上で多言語 BART を事前訓練し、結果として得られるモデル (mBARTHez と呼ぶ) が次のことを示します。 バニラの BARThez を大幅に強化し、CamemBERT や FlauBERT と同等かそれを上回ります。

このモデルは moussakam によって寄稿されました。著者のコードはここにあります。

BARThez の実装は、トークン化を除いて BART と同じです。詳細については、BART ドキュメント を参照してください。 構成クラスとそのパラメータ。 BARThez 固有のトークナイザーについては以下に記載されています。

###
Resources

- BARThez は、BART と同様の方法でシーケンス間のタスクを微調整できます。以下を確認してください。 examples/pytorch/summarization/。

##
BarthezTokenizer

###
class transformers.BarthezTokenizer

< source >

( vocab_file bos_token = '<s>' eos_token = '</s>' sep_token = '</s>' cls_token = '<s>' unk_token = '<unk>' pad_token = '<pad>' mask_token = '<mask>' sp_model_kwargs: typing.Optional[typing.Dict[str, typing.Any]] = None **kwargs )

Parameters

- vocab_file (`str`) — SentencePiece file (generally has a .spm extension) that contains the vocabulary necessary to instantiate a tokenizer.
- bos_token (`str`, optional, defaults to `"<s>"`) — The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.

  When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`.

- eos_token (`str`, optional, defaults to `"</s>"`) — The end of sequence token.

  When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`.

- sep_token (`str`, optional, defaults to `"</s>"`) — The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens.
- cls_token (`str`, optional, defaults to `"<s>"`) — The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens.
- unk_token (`str`, optional, defaults to `"<unk>"`) — The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead.
- pad_token (`str`, optional, defaults to `"<pad>"`) — The token used for padding, for example when batching sequences of different lengths.
- mask_token (`str`, optional, defaults to `"<mask>"`) — The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict.
- sp_model_kwargs (`dict`, optional) — Will be passed to the `SentencePieceProcessor.__init__()` method. The Python wrapper for SentencePiece can be used, among other things, to set:

  - `enable_sampling`: Enable subword regularization.

  - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.

    - `nbest_size = {0,1}`: No sampling is performed.
    - `nbest_size > 1`: samples from the nbest_size results.
    - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm.

  - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout.

- sp_model (`SentencePieceProcessor`) — The SentencePiece processor that is used for every conversion (string, tokens and IDs).

Adapted from CamembertTokenizer and BartTokenizer. Construct a BARThez tokenizer. Based on SentencePiece.

This tokenizer inherits from PreTrainedTokenizer which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

####
build_inputs_with_special_tokens

< source >

( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None ) →
`List[int]`

Parameters

- token_ids_0 (`List[int]`) — List of IDs to which the special tokens will be added.
- token_ids_1 (`List[int]`, optional) — Optional second list of IDs for sequence pairs.

Returns

`List[int]`

List of input IDs with the appropriate special tokens.

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BARThez sequence has the following format:

- single sequence: `<s> X </s>`
- pair of sequences: `<s> A </s></s> B </s>`

####
convert_tokens_to_string

< source >

( tokens )

Converts a sequence of tokens (string) in a single string.

####
create_token_type_ids_from_sequences

< source >

( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None ) →
`List[int]`

Parameters

- token_ids_0 (`List[int]`) — List of IDs.
- token_ids_1 (`List[int]`, optional) — Optional second list of IDs for sequence pairs.

Returns

`List[int]`

List of zeros.

Create a mask from the two sequences passed to be used in a sequence-pair classification task.

####
get_special_tokens_mask

< source >

( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None already_has_special_tokens: bool = False ) →
`List[int]`

Parameters

- token_ids_0 (`List[int]`) — List of IDs.
- token_ids_1 (`List[int]`, optional) — Optional second list of IDs for sequence pairs.
- already_has_special_tokens (`bool`, optional, defaults to `False`) — Whether or not the token list is already formatted with special tokens for the model.

Returns

`List[int]`

A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.

Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method.

##
BarthezTokenizerFast

###
class transformers.BarthezTokenizerFast

< source >

( vocab_file = None tokenizer_file = None bos_token = '<s>' eos_token = '</s>' sep_token = '</s>' cls_token = '<s>' unk_token = '<unk>' pad_token = '<pad>' mask_token = '<mask>' **kwargs )

Parameters

- vocab_file (`str`) — SentencePiece file (generally has a .spm extension) that contains the vocabulary necessary to instantiate a tokenizer.
- bos_token (`str`, optional, defaults to `"<s>"`) — The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.

  When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`.

- eos_token (`str`, optional, defaults to `"</s>"`) — The end of sequence token.

  When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`.

- sep_token (`str`, optional, defaults to `"</s>"`) — The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens.
- cls_token (`str`, optional, defaults to `"<s>"`) — The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens.
- unk_token (`str`, optional, defaults to `"<unk>"`) — The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead.
- pad_token (`str`, optional, defaults to `"<pad>"`) — The token used for padding, for example when batching sequences of different lengths.
- mask_token (`str`, optional, defaults to `"<mask>"`) — The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict.
- additional_special_tokens (`List[str]`, optional, defaults to `["<s>NOTUSED", "</s>NOTUSED"]`) — Additional special tokens used by the tokenizer.

Adapted from CamembertTokenizer and BartTokenizer. Construct a “fast” BARThez tokenizer. Based on SentencePiece.

This tokenizer inherits from PreTrainedTokenizerFast which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

####
build_inputs_with_special_tokens

< source >

( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None ) →
`List[int]`

Parameters

- token_ids_0 (`List[int]`) — List of IDs to which the special tokens will be added.
- token_ids_1 (`List[int]`, optional) — Optional second list of IDs for sequence pairs.

Returns

`List[int]`

List of input IDs with the appropriate special tokens.

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BARThez sequence has the following format:

- single sequence: `<s> X </s>`
- pair of sequences: `<s> A </s></s> B </s>`

####
create_token_type_ids_from_sequences

< source >

( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None ) →
`List[int]`

Parameters

- token_ids_0 (`List[int]`) — List of IDs.
- token_ids_1 (`List[int]`, optional) — Optional second list of IDs for sequence pairs.

Returns

`List[int]`

List of zeros.

Create a mask from the two sequences passed to be used in a sequence-pair classification task.

< > Update on GitHub