<p>This seems related to : <a href="https://video.stackexchange.com/a/33521/32436">https://video.stackexchange.com/a/33521/32436</a></p>
<blockquote>
<p>You could check out
<a href="https://theophanemayaud.github.io/video-simili-duplicate-cleaner/" rel="nofollow noreferrer">https://theophanemayaud.github.io/video-simili-duplicate-cleaner/</a>
(macOS and Windows binaries available) It's open source, I created it with QT and it relies on ffmpeg.</p>
<p>The idea is to extract a few (eg 2, the first and the last) frames,
and compare them. If the duration is close, and the frames look
similar : you've got a duplicate.</p>
</blockquote>
<p>You can &quot;manually compare the extracted frames&quot;, or use thresholds to configure some automatic behaviors, and trash the duplicates.</p>
<blockquote>
<p>It uses opencv for frame comparison, and ffmpeg for frame extraction
(ffmpeg is also for video metadata which is very useful when trying to
tell which video to keep !). Since it's open source, you can improve
it to fit your needs !</p>
</blockquote>
<p>I've found it to be really helpful in a situation like yours, where I had about 15 000 videos from google photos which compresses them, Picasa, handbrake compressed, original copies, and other changed versions of the same videos !</p>

CC BY-SA 4.0
2021-03-29T09:58:54.750Z
2022-03-16T13:25:00.893Z
2022-03-16T13:25:00.893Z
Théophane
<p>You can try Video Comparer (shareware).</p>

<p><a href="http://www.video-comparer.com" rel="nofollow">http://www.video-comparer.com</a></p>

<p>It quickly detects similar videos with image transformations and supports split videos into multiple CDs.</p>

CC BY-SA 3.0
2018-04-25T23:02:43.883Z
This software did not detect *similar* videos (same scene/person taken a few minutes apart).
Clay Nichols
CC BY-SA 3.0
2013-03-13T12:30:59.123Z
2013-03-13T12:30:59.123Z
Rigop
<p>I had the same problem recently and I found the program that helped me. It doesn't compare file names or calculate hashes. It compare real video content. It is not free, but if you haven't many duplicates, the trial version is enough. The link to the program is <a href="http://duplicatevideosearch.com" rel="nofollow">http://duplicatevideosearch.com</a></p>

CC BY-SA 3.0
2013-11-12T08:27:30.407Z
2013-11-12T08:27:30.407Z
Max Smirnoff
<p>Similar images - <a href="https://tn123.org/simimages/" rel="nofollow">https://tn123.org/simimages/</a> is a bit old, but still functional.</p>

CC BY-SA 3.0
2015-03-19T00:23:12.770Z
2015-03-19T00:23:12.770Z
Lion
<p>I'd recommend a hybrid approach using both computers and people.</p>

<ol>
<li>Bucket the videos by their length (round to the nearest second)</li>
<li>For each bucket, use ffmpeg to <a href="http://blog.prashanthellina.com/2008/03/29/creating-video-thumbnails-using-ffmpeg/">generate thumbnails</a> at a predictable and uniform point in the videos (ex: a frame from 10 seconds into the video)</li>
<li>Look at the generated thumbnails in a grid (most OS's provide a nice thumbnail view) and scan for duplicates to remove.</li>
</ol>

<p>You shouldn't have to do any programming to perform these steps, though creating the thumbnails on the command line with ffmpeg may take some finesse.  Good luck!</p>

CC BY-SA 3.0
2012-11-13T15:30:50.820Z
Thumbnails are a great idea! The new videos came in as FLV, which I can't seem to thumbnail no matter what I try. This could work quite well
Zelda
CC BY-SA 3.0
2012-11-13T19:05:26.637Z
ffmpeg should be able to convert the flv files into something else (avi, mp4, etc).
jimbo
CC BY-SA 3.0
2012-11-13T19:11:58.903Z
Yeah, I'll eventually convert them to MP4 but I want to make sure I lose as little quality as possible when I do that, so I've put it off until I can take the time
Zelda
CC BY-SA 3.0
2013-06-18T20:16:45.503Z
If you wanted to, you could find a program that would analyze the thumbnails to find duplicates. However, the time spent finding such a program then configuring it would not be worth it for a small project.
Cole Tobin
CC BY-SA 4.0
2019-09-14T04:23:52.867Z
@BenBrocka Another solution is you make an unique hash string (exampley 8 digits character) from your file in uploading time  and store it on your db, and add a condition on stored unique string for checking duplicate videos in upload time.
Parsa Saei
CC BY-SA 3.0
2012-11-13T15:16:55.360Z
2012-11-13T15:16:55.360Z
jimbo
<p>After losing a backup device with a bunch of my original videos I've had to download a large number of them (~300) back from Youtube. The new downloads are a mix of MP4 and FLV, and the originals are a mix of almost any video format due to lots of experimenting. </p>

<p>To make matters worse there's no clear cutoff for when my local copies disappeared; I've found a point in my youtube videos chonologically where I have some of the videos and I don't have others. I've been trying to manually weed out videos where I already have a copy, but with this many videos I'm sure I've missed some.</p>

<p>How can I effectively go through several hundred videos of different formats and weed out duplicates? The names will be similar but not identical. The formats may be different, and the file size may be different due to that (and or encoding differences). The file lengths should be the same, and visually the videos should be similar enough, but I'm not sure there's any tool to sort that out visually. </p>

<p>Am I doomed to go through the list based on running time and manually look for duplicates or is there a better way?</p>

CC BY-SA 3.0
2012-11-12T10:40:50.110Z
Very interesting question!
Dr Mayhem
<video>
How can I find duplicate videos from a large set of videos of varying formats?
Zelda
# How can I find duplicate videos from a large set of videos of varying formats?

(Asked by: username_0 on 2012-11-11)

After losing a backup device with a bunch of my original videos I've had to download a large number of them (~300) back from Youtube. The new downloads are a mix of MP4 and FLV, and the originals are a mix of almost any video format due to lots of experimenting.

To make matters worse there's no clear cutoff for when my local copies disappeared; I've found a point in my youtube videos chonologically where I have some of the videos and I don't have others. I've been trying to manually weed out videos where I already have a copy, but with this many videos I'm sure I've missed some.

How can I effectively go through several hundred videos of different formats and weed out duplicates? The names will be similar but not identical. The formats may be different, and the file size may be different due to that (and or encoding differences). The file lengths should be the same, and visually the videos should be similar enough, but I'm not sure there's any tool to sort that out visually.

Am I doomed to go through the list based on running time and manually look for duplicates or is there a better way?

**Tags:** video

> **Comments:**
> **username_1**: Very interesting question!  

---

## 5 answers

### Answer by username_7 on 2012-11-13. Score: 13 | Accepted answer

I'd recommend a hybrid approach using both computers and people.

1. Bucket the videos by their length (round to the nearest second)
2. For each bucket, use ffmpeg to [generate thumbnails](http://blog.prashanthellina.com/2008/03/29/creating-video-thumbnails-using-ffmpeg/) at a predictable and uniform point in the videos (ex: a frame from 10 seconds into the video)
3. Look at the generated thumbnails in a grid (most OS's provide a nice thumbnail view) and scan for duplicates to remove.

You shouldn't have to do any programming to perform these steps, though creating the thumbnails on the command line with ffmpeg may take some finesse. Good luck!

> **Comments:**
> **username_0**: Thumbnails are a great idea! The new videos came in as FLV, which I can't seem to thumbnail no matter what I try. This could work quite well  
> **username_7**: ffmpeg should be able to convert the flv files into something else (avi, mp4, etc).  
> **username_0**: Yeah, I'll eventually convert them to MP4 but I want to make sure I lose as little quality as possible when I do that, so I've put it off until I can take the time  
> **username_8**: If you wanted to, you could find a program that would analyze the thumbnails to find duplicates. However, the time spent finding such a program then configuring it would not be worth it for a small project.  
> **username_9**: @BenBrocka Another solution is you make an unique hash string (exampley 8 digits character) from your file in uploading time  and store it on your db, and add a condition on stored unique string for checking duplicate videos in upload time.  

---

### Answer by username_2 on 2021-03-29. Score: 2

This seems related to : <https://video.stackexchange.com/a/33521/32436>

> You could check out
> <https://theophanemayaud.github.io/video-simili-duplicate-cleaner/>
> (macOS and Windows binaries available) It's open source, I created it with QT and it relies on ffmpeg.
>
> The idea is to extract a few (eg 2, the first and the last) frames,
> and compare them. If the duration is close, and the frames look
> similar : you've got a duplicate.

You can "manually compare the extracted frames", or use thresholds to configure some automatic behaviors, and trash the duplicates.

> It uses opencv for frame comparison, and ffmpeg for frame extraction
> (ffmpeg is also for video metadata which is very useful when trying to
> tell which video to keep !). Since it's open source, you can improve
> it to fit your needs !

I've found it to be really helpful in a situation like yours, where I had about 15 000 videos from google photos which compresses them, Picasa, handbrake compressed, original copies, and other changed versions of the same videos !


---

### Answer by username_3 on 2013-03-13. Score: 1

You can try Video Comparer (shareware).

<http://www.video-comparer.com>

It quickly detects similar videos with image transformations and supports split videos into multiple CDs.

> **Comments:**
> **username_4**: This software did not detect *similar* videos (same scene/person taken a few minutes apart).  

---

### Answer by username_5 on 2013-11-12. Score: 1

I had the same problem recently and I found the program that helped me. It doesn't compare file names or calculate hashes. It compare real video content. It is not free, but if you haven't many duplicates, the trial version is enough. The link to the program is <http://duplicatevideosearch.com>


---

### Answer by username_6 on 2015-03-19. Score: -1

Similar images - <https://tn123.org/simimages/> is a bit old, but still functional.


---

