<p>To extract the moving things, I'd try a difference matte. <a href="http://motionworks.net/after-effects-difference-matte/" rel="nofollow">There's a tutorial on doing it in After Effects here</a> and <a href="http://www.motion-graphics-exchange.com/after-effects/Difference-Matte-Effect-Tutorial/474f283e66ac2" rel="nofollow">another one here</a>. You'll need to somehow extract the static background, probably by averaging a few (or several) frames of it together. Then use the techniques described above to subtract out the static stuff from the moving stuff. (Maybe a combo of garbage mattes and difference matte could do it? Depends on the footage.)</p>

<p>Once you've done that, combining the moving matted footage over the hi-res image (probably using a "Normal" blend mode) should get you what you want. I'd be interested to know how it looks in the end. I've seen composites that looked bad because the composited object had a different amount or type of grain from the background. I suspect being at a different resolution may be just as bad, given that the grain will be huge compared with the background, but since I've never done it, I can't say for sure. Good luck!</p>

CC BY-SA 3.0
2015-03-11T16:39:09.810Z
Here are a couple of results using your proposed method. This was using a difference matte, and then putting the moving matted footage on-top using a "Normal" blend mode. I kept the settings for the Difference Matte pretty low, so almost every change in pixel values was made visible.  
  
[Example 1: a crop from a 1080p video scaled up to 4k using a photo and a difference matte](http://sukzessiv.net/~gohai/superresolution/test1_composite4k.mp4)  
  
[Example 1: the same, simply scaled up to 4k](http://sukzessiv.net/~gohai/superresolution/test1_upscale4k.mp4)
gohai
CC BY-SA 3.0
2015-03-11T16:42:11.970Z
[Example 2: a crop from a 1080p video scaled up to 4k using a photo and a difference matte](http://sukzessiv.net/~gohai/superresolution/test2_composite4k.mp4)  -  
[Example 2: the same, simply scaled up to 4k](http://sukzessiv.net/~gohai/superresolution/test2_upscale4k.mp4)  -  
[Example 2: a crop from the original 1080p video, but using a photo and a difference matte](http://sukzessiv.net/~gohai/superresolution/test2_composite.mp4)  -  
[Example 2: a crop from the original 1080p video, no changes](http://sukzessiv.net/~gohai/superresolution/test2_original.mp4)
gohai
CC BY-SA 3.0
2015-03-11T16:42:31.147Z
I'd be interested to hear your opinion. The lack in detail of the moving object is quite noticeable to my eye, but I am not sure if it's better or worse than e.g. upscaling. I was surprised to see the composed clip look better than the original even at the original video resolution of 1080p, even though this was shot as RAW video (there is probably an easy explanation for this). Any suggestions on how to make the matte method more seamless/intelligent?
gohai
CC BY-SA 3.0
2015-03-12T03:34:55.103Z
My opinion is that the difference matted versions are much better, though I don't necessarily know the subject matter in the video you posted, so I may not be the best judge. The issues I was concerned with (grain matching, etc.) are probably more noticeable in typical narrative scenes. One thing that did stand out, though, was a difference in luminance and white balance. The difference matted versions seemed a little darker and cooler than the scaled up versions. Other than that, I think they look better, though.
user1118321
CC BY-SA 3.0
2015-03-09T00:33:12.893Z
2015-03-09T00:33:12.893Z
user1118321
<p>It's an interesting idea, and I'd also like to see the results.</p>

<p>My first thought is that it would probably look very odd.  My feeling is that the eye is drawn to movement first, and detail second.  </p>

<p>For instance, if your example is a locked-off interview shot, where everything is 4K resolution, except for the interesting part - e.g. the talking head, the viewer is barely going to notice that the background is higher quality, because all their attention is going to be drawn to the part of the image which is lower resolution.  Does that make any kind of sense?</p>

CC BY-SA 3.0
2015-03-09T14:26:44.123Z
2015-03-09T14:26:44.123Z
tomh
<p>Suppose I have a 1080p video of a static shot with minimal motion and little change in lighting, and a higher res (~ 18 megapixels) still photograph of the same setup.</p>

<p>How would I go best about using the extra resolution of the photo if I wanted an ever higher res video output (say, 4K)?</p>

<p>My naive idea would be to scale up the 1080p video in an AF composition and use a blending mode for it to influence the still photograph below. I'd have to do rotoscoping to isolate the moving parts and put them on top (upscaled)..</p>

<p>Is there a better way?</p>

CC BY-SA 3.0
2015-03-10T02:52:03.973Z
Welcome gohai :) Please start with the tour http://video.stackexchange.com/tour and add 2-3 images or frames to the question to get a solid answer, otherwise it's only guessing. Anyway, enjoy the site!
p2or
<after-effects><raw>
Combining high res photo with low res video in a static shot
gohai
# Combining high res photo with low res video in a static shot

(Asked by: username_0 on 2015-03-08)

Suppose I have a 1080p video of a static shot with minimal motion and little change in lighting, and a higher res (~ 18 megapixels) still photograph of the same setup.

How would I go best about using the extra resolution of the photo if I wanted an ever higher res video output (say, 4K)?

My naive idea would be to scale up the 1080p video in an AF composition and use a blending mode for it to influence the still photograph below. I'd have to do rotoscoping to isolate the moving parts and put them on top (upscaled)..

Is there a better way?

**Tags:** after-effects, raw

> **Comments:**
> **username_1**: Welcome username_0 :) Please start with the tour http://video.stackexchange.com/tour and add 2-3 images or frames to the question to get a solid answer, otherwise it's only guessing. Anyway, enjoy the site!  

---

## 2 answers

### Answer by username_2 on 2015-03-09. Score: 1 | Accepted answer

To extract the moving things, I'd try a difference matte. [There's a tutorial on doing it in After Effects here](http://motionworks.net/after-effects-difference-matte/) and [another one here](http://www.motion-graphics-exchange.com/after-effects/Difference-Matte-Effect-Tutorial/474f283e66ac2). You'll need to somehow extract the static background, probably by averaging a few (or several) frames of it together. Then use the techniques described above to subtract out the static stuff from the moving stuff. (Maybe a combo of garbage mattes and difference matte could do it? Depends on the footage.)

Once you've done that, combining the moving matted footage over the hi-res image (probably using a "Normal" blend mode) should get you what you want. I'd be interested to know how it looks in the end. I've seen composites that looked bad because the composited object had a different amount or type of grain from the background. I suspect being at a different resolution may be just as bad, given that the grain will be huge compared with the background, but since I've never done it, I can't say for sure. Good luck!

> **Comments:**
> **username_0**: Here are a couple of results using your proposed method. This was using a difference matte, and then putting the moving matted footage on-top using a "Normal" blend mode. I kept the settings for the Difference Matte pretty low, so almost every change in pixel values was made visible.      [Example 1: a crop from a 1080p video scaled up to 4k using a photo and a difference matte](http://sukzessiv.net/~username_0/superresolution/test1_composite4k.mp4)      [Example 1: the same, simply scaled up to 4k](http://sukzessiv.net/~username_0/superresolution/test1_upscale4k.mp4)  
> **username_0**: [Example 2: a crop from a 1080p video scaled up to 4k using a photo and a difference matte](http://sukzessiv.net/~username_0/superresolution/test2_composite4k.mp4)  -   [Example 2: the same, simply scaled up to 4k](http://sukzessiv.net/~username_0/superresolution/test2_upscale4k.mp4)  -   [Example 2: a crop from the original 1080p video, but using a photo and a difference matte](http://sukzessiv.net/~username_0/superresolution/test2_composite.mp4)  -   [Example 2: a crop from the original 1080p video, no changes](http://sukzessiv.net/~username_0/superresolution/test2_original.mp4)  
> **username_0**: I'd be interested to hear your opinion. The lack in detail of the moving object is quite noticeable to my eye, but I am not sure if it's better or worse than e.g. upscaling. I was surprised to see the composed clip look better than the original even at the original video resolution of 1080p, even though this was shot as RAW video (there is probably an easy explanation for this). Any suggestions on how to make the matte method more seamless/intelligent?  
> **username_2**: My opinion is that the difference matted versions are much better, though I don't necessarily know the subject matter in the video you posted, so I may not be the best judge. The issues I was concerned with (grain matching, etc.) are probably more noticeable in typical narrative scenes. One thing that did stand out, though, was a difference in luminance and white balance. The difference matted versions seemed a little darker and cooler than the scaled up versions. Other than that, I think they look better, though.  

---

### Answer by username_3 on 2015-03-09. Score: 0

It's an interesting idea, and I'd also like to see the results.

My first thought is that it would probably look very odd. My feeling is that the eye is drawn to movement first, and detail second.

For instance, if your example is a locked-off interview shot, where everything is 4K resolution, except for the interesting part - e.g. the talking head, the viewer is barely going to notice that the background is higher quality, because all their attention is going to be drawn to the part of the image which is lower resolution. Does that make any kind of sense?


---

