Deep Learning for Computer Vision

Fusing parallel CNN for video classification

Frame-wise, the prediction of a video may not yield good results due to the downsampling of images, which loses fine details. Using a high-resolution CNN will increase the inference time. Hence, Karpathy et al. (https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42455.pdf) propose fusing two streams that are run in parallel for video classification. There are two problems with doing frame-wise predictions, namely:

• Predictions may take a long time because of the larger CNN architecture 
• Independent predictions lose the information along the temporal dimension

The architecture can be simplified with fewer parameters with two smaller encoders running in parallel. The ...