Slow Motion A.I. Tutorial

Artificial Intelligence has come a long way. Now days it can be used for all kinds of things related to Videography and Photography. From automatically removing unwanted things from pictures in Photoshop to putting your face right into a movie clip via the Deep Fake A.I., it’s uses are only going to continue to grow

One such use intrigued me after I saw it in action. A group of programmers developed an A.I. called Super-SloMo using the PyTorch framework which allows you to take normal footage and believable turn it into slow motion. https://github.com/avinashpaliwal/Super-SloMo

This is challenging because the A.I. has to predict what should fill in the missing frames. If you convert a video at 30 FPS to 240 FPS, the A.I. has to auto generate 7 extra frames between every real frame, which is no small task.

My desire to try it out was limited until I was editing some footage from my recent mountain climbing trip. I had done a motion time-lapse video that involved taking a photo every 2 seconds for 10 minutes as it panned across a valley. The shot, below, is cool enough, but it got me wondering if I could use this A.I. to smooth out the motion.

View fullsize

Time-lapse that I wanted to smooth out

So I ran this video through the A.I. and was so amazed by the results that I had to share the process with the world so that other’s could use it. The first video below is a comparison between slowing the video down as shot and what the A.I. was able to do.

See the video below to learn how to do this yourself.

Here are the typed instructions:

1. Make a folder to keep all of the Slow Motion files together. Call it slomo

2. Make an Input and Output folder in your slomo folder.

3. Install a Python Distribution. I’m using Anaconda Distribution. Available for Windows, Mac and Linux - Download 3.7 64bit https://www.anaconda.com/distribution/

4. Download the Super-SloMo project from GitHub Repo - Click Clone or Download, Download ZIP https://github.com/avinashpaliwal/Super-SloMo

5. Extract Super-SlowMo Project into a folder. Rename superslomo

6. Train the model (if you want). Or use a pre-trained model here - https://drive.google.com/file/d/1IvobLDbRiBgZr3ryCRrWL8xDbMZ-KnpF/view

7. Place that model in the superslomo directory

8. Download FFmpeg Video Converter - Download the stable version, the correct architecture and choose the Static Linking https://ffmpeg.zeranoe.com/builds/

9. Unzip that in your slomo folder. Rename it to ffmpeg

10. Open Terminal (Mac) or Anaconda Prompt (Windows)

    1. On a Mac, this requires using bash, not the zsh default that comes with Catalina. You can check what version you are using by typing: echo $SHELL. It will either say /bin/bash or /bin/zsh. If it says /bin/zsh you must change your shell. To switch to bash type: chsh -s /bin/bash. Then you must restart terminal. To change back to zsh after you're done with this project type: chsh -s /bin/zsh and restart terminal.

11. Get the correct command for your setup - Choose Stable, Pick your OS, Choose Conda, Choose whatever version of Python you downloaded, Choose None for CUDA unless you have an External NVIDIA GPU https://pytorch.org/get-started/locally/

12. Copy the command in the box labeled “Run this Command:” and paste it into your Python Terminal. This will download and install all the files and and environments you need

13. cd into the slomo/superslomo directory

14. Adjust the following script to match your computer setup

    • python video_to_slomo.py --ffmpeg /Users/user/slomo/ffmpeg/bin/ --video /Users/user/slomo/input/mountain_timelapse.mp4 --sf 4 --checkpoint /Users/user/slomo/superslomo/SuperSloMo.ckpt --fps 119.880 --output /Users/user/slomo/output/mountain120.mp4 --batch_size 1

    • Definitions of terms in script

      • sf = multiple of the frame rate

      • fps = target frame rate

      • batch _size = how fast it will run, limited by your computer’s hardware. 1 is default

15. Take the finished footage into a video editor and slow it down by whatever multiple your sf was to see the magic.