So the community bonding period is over and so are the first 2 weeks of the GSoC Coding Period. During the community bonding period, I got in touch with my mentor and other members of the JuliaLang Community. I read up most of the research papers related to the Optical Flow work that I have to implement in the first few weeks. After a discussion with my mentor, we decided the final API for the Optical Flow algorithms.
About Optical Flow
Optical flow is the pattern of apparent motion of image objects between two consecutive frames caused by the movemement of object or camera. It is 2D vector field where each vector is a displacement vector showing the movement of points from first frame to second.
Optical Flow in a street scene with moving cycles and pedestrians.
Calculation of optical flow between two images involves solving what is called the Optical Flow equation which is as follows:
where fx, fy are the spatial image gradients, ft is the temporal image gradient and u and v are the velocities of the points in the image.
During the first 3 weeks of GSoC I have proposed to implement two Optical Flow algorithms: the Lucas-Kanade sparse optical flow method and the Farneback dense optical flow method.
Lucas-Kanade Optical Flow
To solve the optical flow constraint equation for u and v, the Lucas-Kanade method divides the original image into smaller sections and assumes a constant velocity in each section. Then, it performs a weighted least-square fit of the optical flow constraint equation to a constant model for [u v]T in each section of the image.
This method is described in detail in this paper by Bruce Lucas and Takeo Kanade. It gives accurate results but fails when the motion is large, so I implemented the Pyramidal approach to the Lucas-Kanade Optical Flow algorithm described in this paper by Jean-Yves Bouguet.
I implemented the basic algorithm in the first 2-3 days. After that I arranged the code according to the decided API and sent the Pull Request #1. After this, on the suggestion of my mentor I changed most of the data structures to StaticArrays to make the implementation more efficient.
Then I added tests for testing the function and worked on adding OffsetArrays compatibility to this function and also the gaussian_pyramid function in Images.jl and sent a Pull Request #722 for the same. Following this I started I started reading up on the Farneback algorithm.
Farneback Optical Flow
While the Lucas-Kanade method computes optical flow for a sparse feature set, the Farneback method computes the optical flow for all the points in the frame. This method is based on the polynomial representation of an image.
The Farneback dense optical flow algorithm is described in this paper by Gunnar Farneback. Polynomial Expansion is an important part of this algorithm and is explained in detail in the author's thesis. The idea of polynomial expansion is to approximate the neighbourhood of a point in a 2D function with a polynomial.
I read the algorithm and the polynomial expansion for the following 2 days as understanding the expansion part and the efficient implementation details took quite a while. I also had to go back to some of my basics to get all the stuff right. After this I started implenmenting the Polynomial Expansion part of the algorithm.