Multi-Modal Signal Processing

Chapter | 12 Multimodal Input

239

●

Intrinsic load: It refers to the cognitive load created by the con-

ceptual complexity of the task and cannot be easily changed [34].

●

Extraneous load: It is the load created by the representation of the

task and can be easily manipulated by changing the format of the

presentation modalities and design [34].

●

Germane load: It is said to be caused by the conversion of

novel information into schema and then stored in long-term

memory [34].

The connection between cognitive load and multimodal process-

ing can be traced back to an early recurring observation reported

in the literature, dubbed the ‘modality effect’. It was found that if

two different modalities were used in task stimuli, subjects exhibited

both performance advantage and preference for multiple modality

conditions, compared with single modality tasks. Cognitive load

theory addresses this observation by attributing the advantages of

multimodal processing to a set of modality-specific working memory

resources that effectively expand working memory when multimodal-

ity is used [35]. The theory assumes a working memory model that

has semi-independent modules for processing visual/spatial and ver-

bal/auditory information and a central executive which interacts with

long-term memory stores [16, 17].

Furthermore, evidence shows users are known to change and

adapt their multimodal behaviour in complex, high-load situations.

For example, when tasks are more difficult, users prefer to interact

multimodally rather than unimodally across a variety of different

application domains. It is believed that this facilitates more effective

use of modality-based working memory resources and assists users

in self-managing cognitive load [35–40].

12.4 UNDERSTANDING MULTIMODAL INPUT

BEHAVIOUR

Modal theories or frameworks of working memory processes can

account for phenomena that occur on the user-side of multimodal

system processing, both in terms of input and response production.

Interface designers can leverage cognitive science contributions in a

240

PART | III Multimodal Human–Computer and Human-to-Human Interaction

number of ways. From an engineering perspective, an understanding

of the way multimodal information is produced can aid the design

and implementation of improved signal recognition and signal fusion

algorithms, developing more appropriate strategies for real-time

adaptation features in those technologies. In terms of content and

representational design, more effective output presentation strategies

can be developed, to optimise the user’s natural information acqui-

sition processes. The evaluation of existing interfaces can also be

judged according to how well they are able to cater and adapt to

naturally occurring multimodal input patterns.

12.4.1 Theoretical Frameworks

The models reviewed below describe the relationship between the

goal of accomplishing a cognitive task and the allocation of a subject’s

internal resources from a modal perspective. They provide a reference

point to begin the analysis of multimodal behaviours.

Modal Theory of Working Memory

Baddeley’s theory of working memory [16] suggests that working

memory is composed of independent modal processors that work

together in a coordinated and synchronous fashion. Baddeley and

colleagues draw evidence from patient pathologies, neural imaging

and empirical findings, incorporating these into a single model, which

is composed of at least four main components:

●

The central executive:Allocates subtasks to a modal processor and

coordinates interaction between the modal slave systems;

●

The phonological loop: The first of three slave systems, it

exclusively processes verbal, auditory and linguistic tasks;

●

The visuo-spatial sketchpad: Exclusively processes imagery and

spatial tasks and

●

The episodic buffer: Temporarily stores schema retrieved from

long-term memory and representations created by the executive

or either of the two other slave systems.

The popularity of Baddeley’s model lies in its ability to account for

much of the psychological evidence available as well as neurological