Chapter | 12 Multimodal Input
Intrinsic load: It refers to the cognitive load created by the con-
ceptual complexity of the task and cannot be easily changed [34].
Extraneous load: It is the load created by the representation of the
task and can be easily manipulated by changing the format of the
presentation modalities and design [34].
Germane load: It is said to be caused by the conversion of
novel information into schema and then stored in long-term
memory [34].
The connection between cognitive load and multimodal process-
ing can be traced back to an early recurring observation reported
in the literature, dubbed the ‘modality effect’. It was found that if
two different modalities were used in task stimuli, subjects exhibited
both performance advantage and preference for multiple modality
conditions, compared with single modality tasks. Cognitive load
theory addresses this observation by attributing the advantages of
multimodal processing to a set of modality-specific working memory
resources that effectively expand working memory when multimodal-
ity is used [35]. The theory assumes a working memory model that
has semi-independent modules for processing visual/spatial and ver-
bal/auditory information and a central executive which interacts with
long-term memory stores [16, 17].
Furthermore, evidence shows users are known to change and
adapt their multimodal behaviour in complex, high-load situations.
For example, when tasks are more difficult, users prefer to interact
multimodally rather than unimodally across a variety of different
application domains. It is believed that this facilitates more effective
use of modality-based working memory resources and assists users
in self-managing cognitive load [35–40].
Modal theories or frameworks of working memory processes can
account for phenomena that occur on the user-side of multimodal
system processing, both in terms of input and response production.
Interface designers can leverage cognitive science contributions in a
PART | III Multimodal Human–Computer and Human-to-Human Interaction
number of ways. From an engineering perspective, an understanding
of the way multimodal information is produced can aid the design
and implementation of improved signal recognition and signal fusion
algorithms, developing more appropriate strategies for real-time
adaptation features in those technologies. In terms of content and
representational design, more effective output presentation strategies
can be developed, to optimise the users natural information acqui-
sition processes. The evaluation of existing interfaces can also be
judged according to how well they are able to cater and adapt to
naturally occurring multimodal input patterns.
12.4.1 Theoretical Frameworks
The models reviewed below describe the relationship between the
goal of accomplishing a cognitive task and the allocation of a subject’s
internal resources from a modal perspective. They provide a reference
point to begin the analysis of multimodal behaviours.
Modal Theory of Working Memory
Baddeley’s theory of working memory [16] suggests that working
memory is composed of independent modal processors that work
together in a coordinated and synchronous fashion. Baddeley and
colleagues draw evidence from patient pathologies, neural imaging
and empirical findings, incorporating these into a single model, which
is composed of at least four main components:
The central executive:Allocates subtasks to a modal processor and
coordinates interaction between the modal slave systems;
The phonological loop: The first of three slave systems, it
exclusively processes verbal, auditory and linguistic tasks;
The visuo-spatial sketchpad: Exclusively processes imagery and
spatial tasks and
The episodic buffer: Temporarily stores schema retrieved from
long-term memory and representations created by the executive
or either of the two other slave systems.
The popularity of Baddeley’s model lies in its ability to account for
much of the psychological evidence available as well as neurological

Get Multi-Modal Signal Processing now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.