The Road to Understand Aerial Cinematography
How does AI picks out the best views and shoot an aerial documentary? At Ailabs, we’ve built a system, equipped a drone and a 360 camera, to have an eye of a videographer.
We want to enable filmmaking to achieve camera movement and tracking shot by artificial intelligence itself. We design our system to be able to pick interesting object and features such as lighthouse or coastline from a 360-degree video and create a flat, standard documentary without manual control of camera movement and angle.
This project is inspired by Chi Po-lin‘s documentary film “Beyond Beauty: Taiwan from Above (看見台灣)“ which captures Taiwan completely in aerial cinematography and broke the Taiwan box office records for the largest opening weekend and the highest total gross of a locally produced documentary. Unfortunately, Chi died in a helicopter crash in a mountainous area in Hualien County’s when the group was shooting footage for the sequel.
Chi’s pointed out it is due to the poor image quality and the monotone of the camera movement that he did not consider to make film by drones. Besides, flying on helicopters for aerial cinematography put photographer in tremendous danger. For this reason, we started the “Chi Po-Lin project” using a drone, a 360 camera and our AI-powered post-editing software algorithm.
Automatic Cinematography for 360
When using a helicopter for videography, the pilot flies the route, and the videographer operates the video camera respectively. In the case of drone, the videographer is replaced by a 360 camera and an algorithm in post-editing to determines where to focus on in the 360 images.
As in the scenario above, we installed a 360 camera on the drone to let the AI control the perspective in the 360 image and render a portion of the 360 image to create virtual camera movements such as pan, tilt, and zoom.
A reason for the monotone of the aerial videos is that it is very difficult for a person to manipulate the controls of the drone and consider the composition details at the same time. Moreover, the conventional methods of automatic cinematography simplify the problem to aligning the camera to the center of the object of interest. So, we give the control of the camera to the AI which, unlike the existing method, takes into account the composition and the semantic flow of the scene.
360 cameras record every point of view. After the videos are collected, the AI recognizes the scenes and objects of interest encountered during the flight, selects the best angles of each moment, and automatically plans multiple sets of suitable trajectories to assist the user in editing the movie.
Automatic Color Enhancement
In order to enhance image quality, we start from color enhancement. We want our video looks as appealing and clearly as professional flim producer took. We leveraged a model which can learn color enhancement by the original input photo and high quality HDR image. The model is learned in an unsupervised way with GAN. In order to reduce model complexity and speed up training, the model is trained with low-resolution image and thus can only output low-resolution image. We extend the model to support high-resolution image by patch-based method. We divide high-resolution image to several overlapping patches and use alpha blending to stitch them. Although it is a image enhancement model, the model is stable enough that we can directly apply it on frames from video without temporal problem. The result looks more appealing than original video in color aspect. According to detail enhancement, result video have more detail than original. For example, the sky looks more clearly than original input video.
Comparison video
Final result with automatic color enhancement
Partners:
– AILabs (台灣人工智慧實驗室)
– Southern Taiwan Science Park Bureau (科技部南科管理局)
– Tainan City Government (台南市政府)
– Department of Aeronautics and Astronautics, NCKU (成大航太)
– GEOSAT Aerospace & Technology Inc (經緯航太)
Sponsored:
– Microsoft
– Nvidia
– Garmin
