Hiho Hive people! Today I'd like to introduce you to a little piece of software called Flowframes , which takes video interpolation to a new level thanks to the use of artificial intelligence. Don't worry, this probably sounds more complicated than it actually is. Even if you're not a motion graphics designer, a VFX artist or a video editor, you might still find this software useful. But let's start with the basics:

Framerate – Herz

Any moving image format, whether digital or analog, has a certain frame rate. This defines how many frames we see per second and is almost always referred to as FPS.

In the cinema we experience 24fps, on European TV 25fps and in North America 30fps[1]. Classic PC screens have 60fps (mostly specified in Herz), gaming monitors are now available with 360fps / 360Hz[2]. I don't want to dive too deep into the matter, but as a rule of thumb you can say: the more fps, the more "fluid" a movement on screen feels. That's why we have cinematic 24fps in the cinema, but professional gamers like to have ten times that much with 240Hz monitors.

It doesn't work the other way around. A classic video game that runs at 24fps feels sluggish and slow. On the other hand, many people find movies that are played at a higher frame rate disturbing. A very good example is "The Hobbit", which was shown at 48fps in some cinemas. The so-called "soap opera" effect[3] was disturbing for many viewers.

Hobbiton and Middle Earth didn’t feel like a different universe, it felt like a special effect, a film set with actors in costumes. It looked like behind the scenes footage. The movement of the actors looked… strange. Almost as if the performances had been partly sped up…. It didn’t look cinematic

Source: https://www.studiodaily.com/2012/04/the-hobbit-the-soap-opera-effect-and-the-48fps-and-faster-future-of-movies/

Frame Interpolation

If you now want to include slow motion in your 24fps cinema film, the respective sequence must be filmed with a higher frame rate. So, for example, for a 4x slowdown, it makes sense to film a scene at 96fps. If you slow down footage that does not have a higher frame rate, this results in unpleasant judder. Even in expensive cinema films this is, unfortunately, often to observe. For example, because the director decides to show a scene in slow motion during the editing process, which was not intended for this beforehand.


Clear stuttering at the end of the table

And this is where frame interpolation comes into play. In this process, artificial intermediate images are calculated with the help of software algorithms.

Source: https://en.wikipedia.org/wiki/Motion_interpolation

This determines which areas in the image have moved where. To do this, the algorithm may not only compare two images, but look at a large number of individual images to better determine the movement. Depending on the video source, this process can lead to very good, to terrible results. These are then visible as distracting artifacts or unnatural-looking movements.

Source: Artifacts from conventional motion interpolation

Video interpolation is supported by pretty much every video editing or compositing program nowadays, and is usually done with a few mouse clicks. Unfortunately, you often come up against the limits of classic algorithms. That's what @ninki.vienna and I had to realize during a video project. That's why we decided to take a new approach.

Flowframes – Video-Interpolation with AI (artificial intelligence)

During my thesis, when I was looking for examples of AI applications in video production, I came across FlowFrames. In fact, I first read their research paper before noticing that they also make their algorithms available in a free open source project. Here is a link to the research paper: RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation for Video Frame Interpolation. There you can also see that the results with their algorithm gives far better results than classical methods.


Left: Original, right with Frame Interpolation using Flowframes

Fortunately, although the underlying algorithms are highly complex, FlowFrames is very easy to use. This is already made clear by the minimalist interface:

Use the button "Browse (Video)" to select the desired video file, in the field below you can define where the newly created file should be saved. At "Set FPS / Interpolation Factor" you define how much you want to increase the frame rate. This is a great strength of Flowframes, many classic software products only allow a maximum of doubling. As "AI Model" you use the default selected model. You should only adjust this tab if you get incorrect results. In the "Output Mode" you can set the preferred video format. After that you only have to press the button "Interpolate!

And the computer starts working, and working hard. Like pretty much all classic AI applications, Flowframes needs a decent graphics card. As a minimum requirement I would recommend 4GB VRAM, but 6GB or more is optimal. If you are not sure what kind of graphics card you have, you can easily check this in Windows 10 via the Task Manager.

If you do not have an Nvidia graphics card, you will have to resort to the "RIFF - Vulkan" in the "Interpolation AI" tab. The larger the image resolution of the input video, the more enormous the required computing power. Up to FullHD (1920x1080), the interpolation is still feasible with classic gaming graphics cards without any problems, but from 4K (3840x2160) onwards, a really enormous amount of performance is required.

If you don't have the hardware but would like to try Flowframes for a clip, just write me and I can calculate it for you with my computer.

Use cases

The classic cases are, of course, people who edit video or simply publish videos on a regular basis. I don't know how many times it happened to me that I would have liked to use slow motion in a scene, but the corresponding source material prevented this. However, in my opinion, there is another obvious use for it: restoring old footage.

Maybe you, your parents or grandparents still have really old pictures. For example, from family celebrations or first trips. Super8 film, which used to be very common, was usually filmed at 18fps[4]. These stuttering shots can be given new life by frame interpolation. Here is an example:

Summary

As a video editor, I find it exciting how more and more AI software applications are diving into my field. However, often these are for such specific use cases that they are of no use to people who are not in that area of expertise. I think this is a bit different with Flowframes, and that's why I didn't want to keep it from you. Especially the restoration approach is worth a look. Because you can not only increase the frame rate, but also dramatically increase the image resolution (for example with the trial version of Video Enhance AI

And even if you don't see the benefit of restoration (yet), I have a task for you: Ask your family for old recordings and digitize them so that these precious memories don't get lost. You'll also put a smile on the face of the older generation when the decades-old videos suddenly reappear on the big TV in the living room.

If you have any questions about the topic, just ask!