Deep Learning for Computer Vision

Multi-modal fusion

Yang et al. (http://research.nvidia.com/sites/default/files/pubs/2016-10_Multilayer-and-Multimodal/MM16.pdf) proposed a multi-modal fusion, with 4 models, for video classification. The four models are 3D convolution features, 2D optical flow, 3D optical flow, and 2D convolution features.

The flow of data in this method is shown as follows:

Reproduced from Yang et al.

Now, let's learn about Convlet. A Convlet is the small convolutional output from a single kernel. The learning of spatial weights in the convolution layer by convlet is shown in the following image:

Reproduced from Yang et al.

A spatial weight indicates how ...