#
TimeSformer

##
개요

TimeSformer 모델은 Facebook Research에서 제안한 TimeSformer: Is Space-Time Attention All You Need for Video Understanding?에서 소개되었습니다. 이 연구는 첫 번째 비디오 Transformer로서, 행동 인식 분야에서 중요한 이정표가 되었습니다. 또한 Transformer 기반의 비디오 이해 및 분류 논문에 많은 영감을 주었습니다.

논문의 초록은 다음과 같습니다.

우리는 공간과 시간에 걸쳐 셀프 어텐션만을 사용하는 합성곱이 없는(convolution-free) 비디오 분류 방법을 제안합니다. 이 방법은 “TimeSformer”라고 불리며, 표준 Transformer 아키텍처를 비디오에 적용하여 프레임 수준 패치 시퀀스로부터 직접 시공간적 특징을 학습할 수 있게 합니다. 우리의 실험적 연구는 다양한 셀프 어텐션 방식을 비교하며, 시간적 어텐션과 공간적 어텐션을 각각의 블록 내에서 별도로 적용하는 “분할 어텐션” 방식이 고려된 설계 선택 중 가장 우수한 비디오 분류 정확도를 제공한다는 것을 시사합니다. 이 혁신적인 설계에도 불구하고, TimeSformer는 Kinetics-400 및 Kinetics-600을 포함한 여러 행동 인식 벤치마크에서 최첨단 결과를 달성했으며, 현재까지 보고된 가장 높은 정확도를 기록했습니다. 마지막으로, 3D 합성곱 네트워크와 비교했을 때, TimeSformer는 더 빠르게 학습할 수 있으며, 약간의 정확도 저하를 감수하면 테스트 효율성이 크게 향상되고, 1분 이상의 긴 비디오 클립에도 적용할 수 있습니다. 코드와 모델은 다음 링크에서 확인할 수 있습니다: https URL 링크.

이 모델은 fcakyon이 기여하였습니다. 원본 코드는 여기에서 확인할 수 있습니다.

##
사용 팁

다양한 사전 학습된 모델의 변형들이 있습니다. 사용하려는 데이터셋에 맞춰 사전 학습된 모델을 선택해야 합니다. 또한, 모델 크기에 따라 클립당 입력 프레임 수가 달라지므로, 사전 학습된 모델을 선택할 때 이 매개변수를 고려해야 합니다.

##
리소스

- Video classification task guide

##
TimesformerConfig

###
class transformers.TimesformerConfig

< source >

( image_size = 224 patch_size = 16 num_channels = 3 num_frames = 8 hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 intermediate_size = 3072 hidden_act = 'gelu' hidden_dropout_prob = 0.0 attention_probs_dropout_prob = 0.0 initializer_range = 0.02 layer_norm_eps = 1e-06 qkv_bias = True attention_type = 'divided_space_time' drop_path_rate = 0 **kwargs )

Parameters

- image_size (`int`, optional, defaults to 224) — The size (resolution) of each image.
- patch_size (`int`, optional, defaults to 16) — The size (resolution) of each patch.
- num_channels (`int`, optional, defaults to 3) — The number of input channels.
- num_frames (`int`, optional, defaults to 8) — The number of frames in each video.
- hidden_size (`int`, optional, defaults to 768) — Dimensionality of the encoder layers and the pooler layer.
- num_hidden_layers (`int`, optional, defaults to 12) — Number of hidden layers in the Transformer encoder.
- num_attention_heads (`int`, optional, defaults to 12) — Number of attention heads for each attention layer in the Transformer encoder.
- intermediate_size (`int`, optional, defaults to 3072) — Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.
- hidden_act (`str` or `function`, optional, defaults to `"gelu"`) — The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` are supported.
- hidden_dropout_prob (`float`, optional, defaults to 0.0) — The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
- attention_probs_dropout_prob (`float`, optional, defaults to 0.0) — The dropout ratio for the attention probabilities.
- initializer_range (`float`, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
- layer_norm_eps (`float`, optional, defaults to 1e-06) — The epsilon used by the layer normalization layers.
- qkv_bias (`bool`, optional, defaults to `True`) — Whether to add a bias to the queries, keys and values.
- attention_type (`str`, optional, defaults to `"divided_space_time"`) — The attention type to use. Must be one of `"divided_space_time"`, `"space_only"`, `"joint_space_time"`.
- drop_path_rate (`float`, optional, defaults to 0) — The dropout ratio for stochastic depth.

This is the configuration class to store the configuration of a TimesformerModel. It is used to instantiate a TimeSformer model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the TimeSformer facebook/timesformer-base-finetuned-k600 architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Example:

    >>> from transformers import TimesformerConfig, TimesformerModel

    >>> # Initializing a TimeSformer timesformer-base style configuration
    >>> configuration = TimesformerConfig()

    >>> # Initializing a model from the configuration
    >>> model = TimesformerModel(configuration)

    >>> # Accessing the model configuration
    >>> configuration = model.config

##
TimesformerModel

###
class transformers.TimesformerModel

< source >

( config )

Parameters

- config (TimesformerConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The bare TimeSformer Model transformer outputting raw hidden-states without any specific head on top. This model is a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( pixel_values: FloatTensor output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) →
transformers.modeling_outputs.BaseModelOutput or `tuple(torch.FloatTensor)`

Parameters

- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_frames, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using AutoImageProcessor. See `VideoMAEImageProcessor.preprocess()` for details.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_outputs.BaseModelOutput or `tuple(torch.FloatTensor)`

A transformers.modeling_outputs.BaseModelOutput or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (TimesformerConfig) and inputs.

- last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) — Sequence of hidden-states at the output of the last layer of the model.

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The TimesformerModel forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> import av
    >>> import numpy as np

    >>> from transformers import AutoImageProcessor, TimesformerModel
    >>> from huggingface_hub import hf_hub_download

    >>> np.random.seed(0)

    >>> def read_video_pyav(container, indices):
    ...     '''
    ...     Decode the video with PyAV decoder.
    ...     Args:
    ...         container (`av.container.input.InputContainer`): PyAV container.
    ...         indices (`List[int]`): List of frame indices to decode.
    ...     Returns:
    ...         result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).
    ...     '''
    ...     frames = []
    ...     container.seek(0)
    ...     start_index = indices[0]
    ...     end_index = indices[-1]
    ...     for i, frame in enumerate(container.decode(video=0)):
    ...         if i > end_index:
    ...             break
    ...         if i >= start_index and i in indices:
    ...             frames.append(frame)
    ...     return np.stack([x.to_ndarray(format="rgb24") for x in frames])

    >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len):
    ...     '''
    ...     Sample a given number of frame indices from the video.
    ...     Args:
    ...         clip_len (`int`): Total number of frames to sample.
    ...         frame_sample_rate (`int`): Sample every n-th frame.
    ...         seg_len (`int`): Maximum allowed index of sample's last frame.
    ...     Returns:
    ...         indices (`List[int]`): List of sampled frame indices
    ...     '''
    ...     converted_len = int(clip_len * frame_sample_rate)
    ...     end_idx = np.random.randint(converted_len, seg_len)
    ...     start_idx = end_idx - converted_len
    ...     indices = np.linspace(start_idx, end_idx, num=clip_len)
    ...     indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64)
    ...     return indices

    >>> # video clip consists of 300 frames (10 seconds at 30 FPS)
    >>> file_path = hf_hub_download(
    ...     repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset"
    ... )
    >>> container = av.open(file_path)

    >>> # sample 8 frames
    >>> indices = sample_frame_indices(clip_len=8, frame_sample_rate=4, seg_len=container.streams.video[0].frames)
    >>> video = read_video_pyav(container, indices)

    >>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base")
    >>> model = TimesformerModel.from_pretrained("facebook/timesformer-base-finetuned-k400")

    >>> # prepare video for the model
    >>> inputs = image_processor(list(video), return_tensors="pt")

    >>> # forward pass
    >>> outputs = model(**inputs)
    >>> last_hidden_states = outputs.last_hidden_state
    >>> list(last_hidden_states.shape)
    [1, 1569, 768]

##
TimesformerForVideoClassification

###
class transformers.TimesformerForVideoClassification

< source >

( config )

Parameters

- config (TimesformerConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

TimeSformer Model transformer with a video classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. This model is a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( pixel_values: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) →
transformers.modeling_outputs.ImageClassifierOutput or `tuple(torch.FloatTensor)`

Parameters

- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_frames, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using AutoImageProcessor. See `VideoMAEImageProcessor.preprocess()` for details.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
- labels (`torch.LongTensor` of shape `(batch_size,)`, optional) — Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy).

Returns

transformers.modeling_outputs.ImageClassifierOutput or `tuple(torch.FloatTensor)`

A transformers.modeling_outputs.ImageClassifierOutput or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (TimesformerConfig) and inputs.

- loss (`torch.FloatTensor` of shape `(1,)`, optional, returned when `labels` is provided) — Classification (or regression if config.num_labels==1) loss.

- logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`) — Classification (or regression if config.num_labels==1) scores (before SoftMax).

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states (also called feature maps) of the model at the output of each stage.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, patch_size, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The TimesformerForVideoClassification forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> import av
    >>> import torch
    >>> import numpy as np

    >>> from transformers import AutoImageProcessor, TimesformerForVideoClassification
    >>> from huggingface_hub import hf_hub_download

    >>> np.random.seed(0)

    >>> def read_video_pyav(container, indices):
    ...     '''
    ...     Decode the video with PyAV decoder.
    ...     Args:
    ...         container (`av.container.input.InputContainer`): PyAV container.
    ...         indices (`List[int]`): List of frame indices to decode.
    ...     Returns:
    ...         result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).
    ...     '''
    ...     frames = []
    ...     container.seek(0)
    ...     start_index = indices[0]
    ...     end_index = indices[-1]
    ...     for i, frame in enumerate(container.decode(video=0)):
    ...         if i > end_index:
    ...             break
    ...         if i >= start_index and i in indices:
    ...             frames.append(frame)
    ...     return np.stack([x.to_ndarray(format="rgb24") for x in frames])

    >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len):
    ...     '''
    ...     Sample a given number of frame indices from the video.
    ...     Args:
    ...         clip_len (`int`): Total number of frames to sample.
    ...         frame_sample_rate (`int`): Sample every n-th frame.
    ...         seg_len (`int`): Maximum allowed index of sample's last frame.
    ...     Returns:
    ...         indices (`List[int]`): List of sampled frame indices
    ...     '''
    ...     converted_len = int(clip_len * frame_sample_rate)
    ...     end_idx = np.random.randint(converted_len, seg_len)
    ...     start_idx = end_idx - converted_len
    ...     indices = np.linspace(start_idx, end_idx, num=clip_len)
    ...     indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64)
    ...     return indices

    >>> # video clip consists of 300 frames (10 seconds at 30 FPS)
    >>> file_path = hf_hub_download(
    ...     repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset"
    ... )
    >>> container = av.open(file_path)

    >>> # sample 8 frames
    >>> indices = sample_frame_indices(clip_len=8, frame_sample_rate=1, seg_len=container.streams.video[0].frames)
    >>> video = read_video_pyav(container, indices)

    >>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics")
    >>> model = TimesformerForVideoClassification.from_pretrained("facebook/timesformer-base-finetuned-k400")

    >>> inputs = image_processor(list(video), return_tensors="pt")

    >>> with torch.no_grad():
    ...     outputs = model(**inputs)
    ...     logits = outputs.logits

    >>> # model predicts one of the 400 Kinetics-400 classes
    >>> predicted_label = logits.argmax(-1).item()
    >>> print(model.config.id2label[predicted_label])
    eating spaghetti

< > Update on GitHub