<p>If the frame-rate is not too high and you can associate a frame with a precise timestamp then the easiest would be to add the metadata as textual subtitles.</p>

<p>The other option is to mux the metadata as standard headers in <code>.mp4</code> or similar container format. <code>.mp4</code> can be broken down into multiple fragments, each one with its own header, and theoretically this can be done per frame. However, doing that for more than a few frames is extremely inefficient in terms of file size and performance. A variation of this method would be to extract every frame as an image (e.g. JEPG), and set its metadata.</p>

<p>Yet another option is to use stenography without any hacking, you just encode the metadata into a separate stream which is timestamp-synced with the main video stream. Most modern container formats, including <code>.mp4</code>, can accommodate multiple video streams in the same container (e.g., you can label the encoded metadata stream as the same content but in another language).</p>

CC BY-SA 3.0
2015-06-05T20:07:33.660Z
Subtitle streams is what I was thinking of, too.  Make sure you flag it as non-default, or add an empty subtitle stream as the first sub stream, so players that aren't aware of your custom sub format don't display it as text.
Peter Cordes
CC BY-SA 3.0
2015-06-05T20:09:27.553Z
ffmpeg distinguishes between subtitle and other data streams.  e.g. there's stuff like RTP metadata in some video files, and it's a separate metadata stream in the mp4 container.  So depending on what you need, you might be able to just use a custom data stream muxed in mp4 or mkv.
Peter Cordes
CC BY-SA 3.0
2015-05-28T10:55:36.063Z
2015-05-28T10:55:36.063Z
avnr
<p>I'm not sure how this fits into a streaming strategy, but the DPX file format includes per-frame metadata (potentially a great deal of it). This <a href="http://en.wikipedia.org/wiki/Digital_Picture_Exchange" rel="nofollow">Wikipedia</a> article has the basic info.</p>

<p>I'm also not clear on what you mean by 'bound permanently'. In the AVI file container the "I" stands for "Interleaved", where video and audio packets (roughly) alternate in the file. This is true for almost all container formats. The streams can be separated, but they natively co-exist in one meta-stream.</p>

CC BY-SA 3.0
2015-05-24T13:09:46.863Z
2015-05-24T13:09:46.863Z
Jim Mack
<p>I wish to stream a video with dynamic metadata. I know I could just steam the video and metadata separately, but ideally I'd like them to be bound permanently. </p>

<p>Is there an existing protocol / encoding / format that does this? I could store the data in each frame using steganography, but this is not an ideal solution and is rather hacky.</p>

<streaming><encoding><formats>
Is there an existing video format that records per-frame metadata?
enigma
# Is there an existing video format that records per-frame metadata?

(Asked by: username_0 on 2015-05-24)

I wish to stream a video with dynamic metadata. I know I could just steam the video and metadata separately, but ideally I'd like them to be bound permanently.

Is there an existing protocol / encoding / format that does this? I could store the data in each frame using steganography, but this is not an ideal solution and is rather hacky.

**Tags:** streaming, encoding, formats


---

## 2 answers

### Answer by username_1 on 2015-05-28. Score: 7

If the frame-rate is not too high and you can associate a frame with a precise timestamp then the easiest would be to add the metadata as textual subtitles.

The other option is to mux the metadata as standard headers in `.mp4` or similar container format. `.mp4` can be broken down into multiple fragments, each one with its own header, and theoretically this can be done per frame. However, doing that for more than a few frames is extremely inefficient in terms of file size and performance. A variation of this method would be to extract every frame as an image (e.g. JEPG), and set its metadata.

Yet another option is to use stenography without any hacking, you just encode the metadata into a separate stream which is timestamp-synced with the main video stream. Most modern container formats, including `.mp4`, can accommodate multiple video streams in the same container (e.g., you can label the encoded metadata stream as the same content but in another language).

> **Comments:**
> **username_2**: Subtitle streams is what I was thinking of, too.  Make sure you flag it as non-default, or add an empty subtitle stream as the first sub stream, so players that aren't aware of your custom sub format don't display it as text.  
> **username_2**: ffmpeg distinguishes between subtitle and other data streams.  e.g. there's stuff like RTP metadata in some video files, and it's a separate metadata stream in the mp4 container.  So depending on what you need, you might be able to just use a custom data stream muxed in mp4 or mkv.  

---

### Answer by username_3 on 2015-05-24. Score: 2

I'm not sure how this fits into a streaming strategy, but the DPX file format includes per-frame metadata (potentially a great deal of it). This [Wikipedia](http://en.wikipedia.org/wiki/Digital_Picture_Exchange) article has the basic info.

I'm also not clear on what you mean by 'bound permanently'. In the AVI file container the "I" stands for "Interleaved", where video and audio packets (roughly) alternate in the file. This is true for almost all container formats. The streams can be separated, but they natively co-exist in one meta-stream.


---

