Deep Learning Essentials

Lip reading

Another interesting application of deep learning is lip reading sentences in the wild. Chung and their co-authors (https://arxiv.org/pdf/1611.05358v1.pdf) in their recent work proposed a method to recognize spoken words by a talking face, with or without audio. The core idea behind the model is a watch-listen-attend-spell network. This network models each output character, y_i as a conditional distribution of all previous characters, y<i, input visual sequence of lip images, x^v, and input audio sequence, x^a, as:

The following figure, Overview of a lip reading application using Watch, Listen, Attend, and Spell architecture, summarizes ...