<p>You should check out <a href="https://processing.org" rel="nofollow">Processing dot org</a>.</p>

CC BY-SA 3.0
2016-08-03T17:26:22.183Z
2016-08-03T17:26:22.183Z
Jason Conrad
<p>As an alternative to Processing, if you prefer to use C#, you can write plugins for <a href="https://vvvv.org/" rel="nofollow">VVVV</a> using their <a href="https://vvvv.org/documentation/plugins" rel="nofollow">C# APIs</a>. Also if you prefer to just write pixel and vertex shaders which you can apply to your audio/video, <a href="https://vvvv.org/documentation/tutorial-of-effects-and-shaders" rel="nofollow">they support that as well</a>.</p>

CC BY-SA 3.0
2016-08-10T19:15:45.807Z
2016-08-10T19:15:45.807Z
glenneroo
<p>To add upon John's answer, one way to do this using FFmpeg would be to add your filters (in C), but another, more modular, way is to code a <a href="https://www.dyne.org/software/frei0r/" rel="nofollow">Frei0r</a> plugin, which can then be used with various host applications, whether FFmpeg, MLT..etc</p>

<blockquote>
  <p>Frei0r is a minimalistic plugin API for video effects. The main
  emphasis is on simplicity for an API that will round up the most
  common video effects into simple filters, sources and mixers that can
  be controlled by parameters. It’s our hope that this way these simple
  effects can be shared between many applications, avoiding their
  reimplementation by different projects.</p>
</blockquote>

CC BY-SA 3.0
2016-08-03T15:08:02.263Z
2016-08-03T15:08:02.263Z
Gyan
<p>You might want to look into using <a href="https://ffmpeg.org/" rel="nofollow">FFmpeg</a> - it seems like the kind of tool you're looking for. Adobe's After Effects allows some <a href="http://cgi.tutsplus.com/tutorials/introduction-to-the-basics-of-after-effects-scripting--ae-22518" rel="nofollow">custom scripting</a>, but not nearly to the extent you're looking for.</p>

CC BY-SA 3.0
2016-08-06T12:54:17.697Z
Unless you're using the SDK to write plugins for AE you don't get pixel-by-pixel access to the image.
stib
CC BY-SA 3.0
2016-08-03T14:35:11.737Z
2016-08-03T14:35:11.737Z
John Barton
<p>You can code custom plugins for <em>Shotcut</em> video editor using MLT, frei0r, or WebVfx plugin. MLT and frei0r plugins require C or C++ coding; whereas WebVfx lets you use HTML5, CSS3, JavaScript, SVG, and WebGL. : <a href="https://shotcut.org/notes/make-plugins/" rel="nofollow noreferrer">https://shotcut.org/notes/make-plugins/</a></p>

<p>I like unity3d for audio/video things</p>

<p>Obviously, have a go on Milkdrop if you want fluid dynamics code, else see if you can splice a video streaming code into the open source MD2 codebase, everyone would be well chuffed. </p>

<p>Research "VJ" programs, like dj programs for video. </p>

CC BY-SA 4.0
2019-11-18T15:30:20.857Z
2019-11-18T16:19:41.207Z
2019-11-18T16:19:41.207Z
bandybabboon
<p>I second the vote for Processing, but if C++ is more your thing than Java you could also look at <a href="http://openframeworks.cc/" rel="nofollow">openFrameworks</a>. It allows for more processor-intensive computation and as such is good at real-time synthesis and manipulation of video.</p>

<p>Another tool worth checking out is <a href="https://www.nodebox.net/" rel="nofollow">NodeBox</a>. It allows for coding, but wraps it all in a node-based GUI, the aim being to simplify the boring stuff.</p>

<p>Final Cut Pro ≤ 7 used to have a built in language called <a href="http://www.joemaller.com/fcp/fxscript_reference.shtml" rel="nofollow">fxcript</a> (based on c, but weird and buggy) that gave you pixel level access to the video stream and integrated into the editor seamlessly. I wrote a few plugins for that <a href="http://web.archive.org/web/20091023111748/http://www.pureandapplied.com.au/plugins.html" rel="nofollow">back in the day</a>, but I jumped ship after the initial FCPX fiasco so I can't comment on the new language for visual effects <a href="https://developer.apple.com/library/mac/documentation/AppleApplications/Conceptual/FXPlug_overview/FXPlugSDKOverview/FXPlugSDKOverview.html" rel="nofollow">FXPlug</a>. I believe it's quite powerful, and, like processing, leverages openGL to do the heavy lifting.</p>

CC BY-SA 3.0
2016-08-06T13:09:42.077Z
2016-08-15T14:13:49.800Z
2016-08-15T14:13:49.800Z
stib
<p>I'm the type of guy who likes to produce beauty from code - I mostly experiment with shaders.</p>

<p>Up to this day, I've only been doing so for the sake of games, but an idea popped up in my mind that I could decorate an actual film footage using handwritten postprocessing.</p>

<p>Is there any software that, apart from facilitating basic video editing tasks, lets me code (and later apply) my own effect completely from scratch, that is, lets me manipulate each input frame pixel by pixel? Either in some shader language or maybe CUDA or even C++?</p>

<video-editor><programming>
Video editor that lets me code my own effects from scratch?
Patryk Czachurski
# Video editor that lets me code my own effects from scratch?

(Asked by: username_0 on 2016-08-03)

I'm the type of guy who likes to produce beauty from code - I mostly experiment with shaders.

Up to this day, I've only been doing so for the sake of games, but an idea popped up in my mind that I could decorate an actual film footage using handwritten postprocessing.

Is there any software that, apart from facilitating basic video editing tasks, lets me code (and later apply) my own effect completely from scratch, that is, lets me manipulate each input frame pixel by pixel? Either in some shader language or maybe CUDA or even C++?

**Tags:** video-editor, programming


---

## 6 answers

### Answer by username_5 on 2016-08-06. Score: 3 | Accepted answer

I second the vote for Processing, but if C++ is more your thing than Java you could also look at [openFrameworks](http://openframeworks.cc/). It allows for more processor-intensive computation and as such is good at real-time synthesis and manipulation of video.

Another tool worth checking out is [NodeBox](https://www.nodebox.net/). It allows for coding, but wraps it all in a node-based GUI, the aim being to simplify the boring stuff.

Final Cut Pro ≤ 7 used to have a built in language called [fxcript](http://www.joemaller.com/fcp/fxscript_reference.shtml) (based on c, but weird and buggy) that gave you pixel level access to the video stream and integrated into the editor seamlessly. I wrote a few plugins for that [back in the day](http://web.archive.org/web/20091023111748/http://www.pureandapplied.com.au/plugins.html), but I jumped ship after the initial FCPX fiasco so I can't comment on the new language for visual effects [FXPlug](https://developer.apple.com/library/mac/documentation/AppleApplications/Conceptual/FXPlug_overview/FXPlugSDKOverview/FXPlugSDKOverview.html). I believe it's quite powerful, and, like processing, leverages openGL to do the heavy lifting.


---

### Answer by username_1 on 2016-08-03. Score: 3

You should check out [Processing dot org](https://processing.org).


---

### Answer by username_2 on 2016-08-10. Score: 2

As an alternative to Processing, if you prefer to use C#, you can write plugins for [VVVV](https://vvvv.org/) using their [C# APIs](https://vvvv.org/documentation/plugins). Also if you prefer to just write pixel and vertex shaders which you can apply to your audio/video, [they support that as well](https://vvvv.org/documentation/tutorial-of-effects-and-shaders).


---

### Answer by username_3 on 2016-08-03. Score: 2

To add upon John's answer, one way to do this using FFmpeg would be to add your filters (in C), but another, more modular, way is to code a [Frei0r](https://www.dyne.org/software/frei0r/) plugin, which can then be used with various host applications, whether FFmpeg, MLT..etc

> Frei0r is a minimalistic plugin API for video effects. The main
> emphasis is on simplicity for an API that will round up the most
> common video effects into simple filters, sources and mixers that can
> be controlled by parameters. It’s our hope that this way these simple
> effects can be shared between many applications, avoiding their
> reimplementation by different projects.


---

### Answer by username_4 on 2016-08-03. Score: 1

You might want to look into using [FFmpeg](https://ffmpeg.org/) - it seems like the kind of tool you're looking for. Adobe's After Effects allows some [custom scripting](http://cgi.tutsplus.com/tutorials/introduction-to-the-basics-of-after-effects-scripting--ae-22518), but not nearly to the extent you're looking for.

> **Comments:**
> **username_5**: Unless you're using the SDK to write plugins for AE you don't get pixel-by-pixel access to the image.  

---

### Answer by username_6 on 2019-11-18. Score: 1

You can code custom plugins for *Shotcut* video editor using MLT, frei0r, or WebVfx plugin. MLT and frei0r plugins require C or C++ coding; whereas WebVfx lets you use HTML5, CSS3, JavaScript, SVG, and WebGL. : <https://shotcut.org/notes/make-plugins/>

I like unity3d for audio/video things

Obviously, have a go on Milkdrop if you want fluid dynamics code, else see if you can splice a video streaming code into the open source MD2 codebase, everyone would be well chuffed.

Research "VJ" programs, like dj programs for video.


---

