Mind Hacks | O'Reilly Media

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

It's never entirely true to say, "This bit of the brain is solely responsible for function X." Take the visual system [Hack #13] , for instance; it runs through many varied parts of the brain with no single area solely responsible for all of vision. Vision is made up of lots of different subfunctions, many of which will be compensated for if areas become unavailable. With some types of brain damage, it's possible to still be able to see, but not be able to figure out what's moving or maybe not be able to see what color things are.

What we can do is look at which parts of the brain are active while it is performing a particular task—anything from recognizing a face to playing the piano—and make some assertions. We can provide input and see what output we get—the black box approach to the study of mind. Or we can work from the outside in, figuring out which abilities people with certain types of damaged brains lack.

The latter, part of neuropsychology [Hack #6] , is an important tool for psychologists. Small, isolated strokes can deactivate very specific brain regions, and also (though more rarely) accidents can damage small parts of the brain. Seeing what these people can no longer do in these pathological cases, provides good clues into the functions of those regions of the brain. Animal experimentation, purposely removing pieces of the brain to see what happens, is another.

These are, however, pathology-based methods—less invasive techniques are available. Careful experimentation—measuring response types, reaction times, and response changes to certain stimuli over time—is one such alternative. That's cognitive psychology [Hack #1] , the science of making deductions about the structure of the brain through reverse engineering from the outside. It has a distinguished history. More recently we've been able to go one step further. Pairing techniques from cognitive psychology with imaging methods and stimulation techniques

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

It's never entirely true to say, "This bit of the brain is solely responsible for function X." Take the visual system [Hack #13] , for instance; it runs through many varied parts of the brain with no single area solely responsible for all of vision. Vision is made up of lots of different subfunctions, many of which will be compensated for if areas become unavailable. With some types of brain damage, it's possible to still be able to see, but not be able to figure out what's moving or maybe not be able to see what color things are.

What we can do is look at which parts of the brain are active while it is performing a particular task—anything from recognizing a face to playing the piano—and make some assertions. We can provide input and see what output we get—the black box approach to the study of mind. Or we can work from the outside in, figuring out which abilities people with certain types of damaged brains lack.

The latter, part of neuropsychology [Hack #6] , is an important tool for psychologists. Small, isolated strokes can deactivate very specific brain regions, and also (though more rarely) accidents can damage small parts of the brain. Seeing what these people can no longer do in these pathological cases, provides good clues into the functions of those regions of the brain. Animal experimentation, purposely removing pieces of the brain to see what happens, is another.

These are, however, pathology-based methods—less invasive techniques are available. Careful experimentation—measuring response types, reaction times, and response changes to certain stimuli over time—is one such alternative. That's cognitive psychology [Hack #1] , the science of making deductions about the structure of the brain through reverse engineering from the outside. It has a distinguished history. More recently we've been able to go one step further. Pairing techniques from cognitive psychology with imaging methods and stimulation techniques

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

How do you tell what's inside a black box without looking in it? This is the challenge the mind presents to cognitive psychology.

Cognitive psychology is the psychology of the basic mental processes—things like perception, attention, memory, language, decision-making. It asks the question, "What are the fundamental operations on which mind is based?"

The problem is, although you can measure what goes into someone's head (the input) and measure roughly what they do (the output), this doesn't tell you anything about what goes on in between. It's a black box, a classic reverse engineering problem.¹ How can we figure out how it works without looking at the code?

These days, of course, we can use neuroimaging (like EEG [Hack 2] , PET [Hack #3] , and fMRI [Hack #4] ) to look inside the head at the brain, or use information on anatomy and information from brain-damaged individuals [Hack #6] to inform how we think the brain runs the algorithms that make up the mind. But this kind of work hasn't always been possible, and it's never been easy or cheap. Experimental psychologists have spent more than a hundred years refining methods for getting insight into how the mind works without messing with the insides, and these days we call this cognitive psychology.

There's an example of a cognitive psychology-style solution in another book from the hacks series, Google Hacks (http://www.oreilly.com/catalog/googlehks). Google obviously doesn't give access to the algorithms that run its searches, so the authors of Google Hacks, Tara Calishain and Rael Dornfest, were forced to do a little experimentation to try and work it out. Obviously, if you put in two words, Google returns pages that feature both words. But does the order matter? Here's an experiment. Search Google for "reverse engineering" and then search for "engineering reverse." The results are different; in fact, they are sometimes different even when searching for words that aren't normally taken together as some form of phrase. So we might conclude that order does make a difference; in some way, the Google search algorithm takes into account the order. If you try to whittle a search down to the right terms, something that returned only a couple of hits, perhaps over time you could figure out more exactly how the order mattered.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

EEGs give you an overall picture of the timing of brain activity but without any fine detail.

An electroencephalogram (EEG) produces a map of the electrical activity on the surface of the brain. Fortunately, the surface is often what we're interested in, as the cortex—responsible for our complex, high-level functions—is a thin sheet of cells on the brain's outer layer. Broadly, different areas contribute to different abilities, so one particular area might be associated with grammar, another with motion detection. Neurons send signals to one another using electrical impulses, so we can get a good measure of the activity of the neurons (how busy they are doing the work of processing) by measuring the electromagnetic field nearby. Electrodes outside the skull on the surface of the skin are close enough to take readings of these electromagnetic fields.

Small metal disks are evenly placed on the head, held on by a conducting gel. The range can vary from two to a hundred or so electrodes, all taking readings simultaneously. The output can be a simple graph of signals recorded at each electrode or visualised as a map of the brain with activity called out.

The EEG technique is well understood and has been in use for many decades. Patterns of electrical activity corresponding to different states are now well-known: sleep, epilepsy, or how the visual cortex responds when the eyes are in use. It is from EEG that we get the concepts of alpha, beta, and gamma waves, related to three kinds of characteristic oscillations in the signal.
Great time resolution. A reading of electrical activity can be taken every few milliseconds, so the brain's response to stimuli can be precisely plotted.
Relatively cheap. Home kits are readily available. OpenEEG (http://openeeg.sourceforge.net), EEG for the rest of us, is a project to develop low-cost EEG devices, both hardware and software.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

PET is a radioactivity-based technique to build a detailed 3D model of the brain and its activity.

Positron emission tomography (PET) is more invasive than any of the other imaging techniques. It requires getting a radioactive chemical into the bloodstream (by injection) and watching for where in the brain the radioactivity ends up—the "positron emission" of the name. The level of radioactivity is not dangerous, but this technique should not be used on the same person on a regular basis.

When neurons fire to send a signal to other neurons, they metabolize more energy. A few seconds later, fresh blood carrying more oxygen and glucose is carried to the region. Using a radioactive isotope of water, the amount of blood flow to each brain location can be monitored, and the active areas of the brain that require a lot of energy and therefore blood flow can be deduced.

A PET scan will produce a 3D model of brain activity.

Scans have to take place in bulky, expensive machinery, which contain the entire body.
PET requires injecting the subject with a radioactive chemical.
Although the resolution of images has improved over the last 30 years, PET still doesn't produce as fine detail as other techniques (it can see activity about 1 cm across).
PET isn't good for looking at how brain activity changes over time. A snapshot can take minutes to be assembled.

—Myles Jones & Matt Webb

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

fMRI produces high-resolution animations of the brain in action.

Functional magnetic resonance imaging (fMRI) is the king of brain imaging. Magnetic resonance imaging is noninvasive and has no known side effects—except, for some, claustrophobia. Having an MRI scan requires you to lie inside a large electromagnet in order to be exposed to the high magnetic field necessary. It's a bit like being slid inside a large white coffin. It gets pretty noisy too.

The magnetic field pushes the hydrogen atoms in your brain into a state in which they all "line up" and spin at the same frequency. A radio frequency pulse is applied at this exact frequency, making the molecules "resonate" and then emit radio waves as they lose energy and return to "normal." The signal emitted depends on what type of tissue the molecule is in. By recording these signals, a 3D map of the anatomy of the brain is built up.

MRI isn't a new technology (it's been possible since the '70s), but it's been applied to psychology with BOLD functional MRI (abbreviated to fMRI) only as recently as 1992. To obtain functional images of the brain, BOLD (blood oxygen level dependent) fMRI utilizes the fact that deoxygenated blood is magnetic (because of the iron in hemoglobin) and therefore makes the MRI image darker. When neurons become active, fresh blood washes away the deoxygenated blood in the precise regions of the brain that have been more active than usual.

While structural MRI can take a long time, fMRI can take a snapshot of activity over the whole brain every couple of seconds, and the resolution is still higher than with PET [Hack #3] . It can view activity in volumes of the brain only 2 mm across and build a whole map of the brain from that. For a particular experiment, a series of fMRI snapshots will be animated over a single high-resolution MRI scan, and experimenters can see in exactly which brain areas activity is taking place.

Much of the cognitive neuroscience research done now uses fMRI. It's a method that is still developing and improving, but already producing great results.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Stimulate or suppress specific regions of the brain, then sit back and see what happens.

Transcranial magnetic stimulation (TMS) isn't an imaging technique like EEG [Hack 2] or fMRI [Hack #4] , but it can be used along with them. TMS uses a magnetic pulse or oscillating magnetic fields to temporarily induce or suppress electrical activity in the brain. It doesn't require large machines, just a small device around the head, and—so far as we know—it's harmless with no aftereffects.

Neurons communicate using electrical pulses, so being able to produce electrical activity artificially has its advantages. Selected regions can be excited or suppressed, causing hallucinations or partial blindness if some part of the visual cortex is being targeted. Both uses help discover what specific parts of the brain are for. If the subject experiences a muscle twitching, the TMS has probably stimulated some motor control neurons, and causing hallucinations at different points in the visual system can be used to discover the order of processing (it has been used to discover where vision is cut out during saccades [Hack #17] , for example).

Preventing a region from responding is also useful: if shutting down neurons in a particular area of the cortex stops the subject from recognizing motion, that's a good clue as to the function of that area. This kind of discovery was possible before only by finding people with localized brain damage; now TMS allows more structured experiments to take place.

Coupled with brain imaging techniques, it's possible to see the brain's response to a magnetic pulse ripple through connected areas, revealing its structure.

Affects neural activity directly, rather than just measuring it.

Apparently harmless, although it's still early days.

"Savant For a Day" by Lawrence Osbourne (http://www.nytimes.com/2003/06/22/magazine/22SAVANT.html or http://www.cognitiveliberty.org/neuro/TMS_NYT.html

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Neuropsychology is the study of what different parts of the brain do by studying people who no longer have those parts. As well as being the oldest technique of cognitive neuroscience, it refutes the oft-repeated myth that we only use 10% of our brains.

Of the many unscientific nuggets of wisdom about the brain that many people believe, the most common may be the "fact" that we use only 10% of our brains.

In a recent survey of people in Rio de Janeiro with at least a college education, approximately half stated that the 10% myth was true.¹ There is no reason to suppose the results of a similar survey conducted anywhere else in the world would be radically different. It's not surprising that a lot of people believe this myth, given how often it is claimed to be true. Its continued popularity has prompted one author to state that the myth has "a shelf life longer than lacquered Spam".²

Where does this rather popular belief come from?
It's hard to find out how the myth started. Some people say that something like it was said by Einstein, but there isn't any proof. The idea that we have lots of spare capacity is certainly true and fits with our aspirational culture, as well as with the Freudian notion that the mind is mostly unconscious. Indeed, the myth was being used to peddle self-help literature as early as 1929.³ The neatness and numerological potency of the 10% figure is a further factor in the endurance of the myth.
—A.B.

Neuropsychology is the study of patients who have suffered brain damage and the psychological consequences of that brain damage. As well as being a vital source of information about which bits of the brain are involved in doing which things, neuropsychology also provides a neat refutation of the 10% myth: if we use only 10% of our brains, which bits would you be happy to lose? From neuropsychology, we know that losing any bit of the brain causes you to stop being able to do something or being able to do it so well. It's all being used, not just 10% of it.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Take a brief tour around the spinal cord and brain. What's where, and what does what?

Think of the central nervous system like a mushroom with the spinal cord as the stalk and the brain as the cap. Most of the hacks in this book arise from features in the cortex, the highly interconnected cells that make a thin layer over the brain...but not all. So let's start outside the brain itself and work back in.

Senses and muscles all over the body are connected to nerves, bundles of neurons that carry signals back and forth. Neurons come in many types, but they're basically the same wherever they're found in the body; they carry electric current and can act as relays, passing on information from one neuron to the next. That's how information is carried from the sensory surface of the skin, as electric signals, and also how muscles are told to move, by information going the other way.

Nerves at this point run to the spinal cord two by two. One of each pair of nerves is for receptors (a sense of touch for instance) and one for effectors—these trigger actions in muscles and glands. At the spinal cord, there's no real intelligence yet but already some decision-making—such as the withdrawal reflex—occurs. Urgent signals, like a strong sense of heat, can trigger an effector response (such as moving a muscle) before that signal even reaches the brain.

The spinal cord acts as a conduit for nerve impulses up and down the body: sensory impulses travel up to the brain, and the motor areas of the brain send signals back down again. Inside the cord, the signals converge into 31 pairs of nerves (sensory and motor again), and eventually, at the top of the neck, these meet the brain.

At about the level of your mouth, right in the center of your head, the bundles of neurons in the spinal cord meet the brain proper. This tip of the spinal cord, called the brain stem, continues like a thick carrot up to the direct center of your brain, at about the same height as your eyes.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The forebrain, the classic image of the brain we know from pictures, is the part of the brain that defines human uniqueness. It consists of four lobes and a thin layer on the surface called the cortex.

When you look at pictures of the human brain, the main thing you see is the rounded, wrinkled bulk of the brain. This is the cerebrum, and it caps off the rest of the brain and central nervous system [Hack #7] .

To find your way around the cerebrum, you need to know only a few things. It's divided into two hemispheres, left and right. It's also divided into four lobes (large areas demarcated by particularly deep wrinkles). The wrinkles you can see on the outside are actually folds: the cerebrum is a very large folded-up surface, which is why it's so deep. Unfolded, this surface—the cerebral cortex—would be about 1.5 m² (a square roughly 50 inches on the side), and between 2 and 4 mm deep. It's not thick, but there's a lot of it and this is where all the work takes place. The outermost part, the top of the surface, is gray matter, the actual neurons themselves. Under a few layers of these is the white matter, the fibers connecting the neurons together. The cortex is special because it's mainly where our high-level, human functions take place. It's here that information is integrated and combined from the other regions of the brain and used to modulate more basic functions elsewhere in the brain. The folds exist to allow many more neurons and connections than other animals have in a similar size area.

The four cerebral lobes generally perform certain classes of function.

You can cover the frontal lobe if you put your palms on your forehead with your fingers pointing up. It's heavily involved in planning, socializing, language, and general control and supervision of the rest of the brain.

The parietal lobe is at the top and back of your head, and if you lock your fingers together and hook your hands over the top back, that's it covered there. It deals a lot with your senses, combining information and representing your body and movements. The object recognition module for visual processing

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

There's a veritable electrical storm going on inside your head: 100 billion brain cells firing electrical signals at one another are responsible for your every thought and action.

A neuron, a.k.a. nerve cell or brain cell, is a specialized cell that sends an electrical impulse out along fibers connecting it, in turn, to other neurons. These guys are the wires of your very own personal circuitry.

What follows is a simplistic description of the general features of nerve cells, whether they are found sending signals from your senses to your brain, from your brain to your muscles, or to and from other nerve cells. It's this last class, the kind that people most likely mean when they say "neurons," that we are most interested in here. (All nerve cells, however, share a common basic design.)

Don't for a second think that the general structure we're describing here is the end of the story. The elegance and complexity of neuron design is staggering, a complex interplay of structure and noise; of electricity, chemistry, and biology; of spatial and dynamic interactions that result in the kind of information processing that cannot be defined using simple rules.¹ For just a glimpse at the complexity of neuron structure, you may want to start with this free chapter on nerve cells from the textbook Molecular Cell Biology by Harvey Lodish, Arnold Berk, Lawrence S. Zipursky, Paul Matsudaira, David Baltimore, and James Darnell and published by W. H. Freeman (http://www.ncbi.nlm.nih.gov/books/bv.fcgi?call=bv.View..ShowSection&rid=mcb.chapter.6074), but any advanced cell biology or neuroscience textbook will do to give you an idea of what you're missing here.

The neuron is made up of a cell body with long offshoots—these can be very long (the whole length of the neck, for some neurons in the giraffe, for example) or very short (i.e., reaching only to the neighboring cell, scant millimeters away). Signals pass only one way along a neuron. The offshoots receiving incoming transmissions are called

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

When you think really hard, your heart rate noticeably increases.

The brain requires approximately 20% of the oxygen in the body, even during times of rest. Like the other organs in our body, our brain needs more glucose, oxygen, and other essential nutrients as it takes on more work. Many of the scanning technologies that aim to measure aspects of brain function take advantage of this. Functional magnetic resonance imaging (fMRI) [Hack #4] benefits from the fact that oxygenated blood produces slightly different electromagnetic signals when exposed to strong magnetic fields than deoxygenated blood and that oxygenated blood is more concentrated in active brain areas. Positron emission tomography (PET) [Hack #3] involves being injected with weakly radioactive glucose and reading the subsequent signals from the most active, glucose-hungry areas of the brain.

A technology called transcranial Doppler sonography takes a different approach and measures blood flow through veins and arteries. It takes advantage of the fact that the pitch of reflected ultrasound will be altered in proportion to the rate of flow and has been used to measure moment-to-moment changes in blood supply to the brain. It has been found to be particularly useful in making comparisons between different mental tasks. However, even without transcranial Doppler sonography, you can measure the effect of increased brain activity on blood flow by measuring the pulse.

For this exercise you will need to get someone to measure your carotid pulse, taken from either side of the front of the neck, just below the angle of the jaw. It is important that only very light pressure be used—a couple of fingertips pressed lightly to the neck, next to the windpipe, should enable your friend to feel your pulse with little trouble.

First you need to take a measure of a resting pulse. Sit down and relax for a few minutes. When you are calm, ask your friend to count your pulse for 60 seconds. During this time, close your eyes and try to empty your mind.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

More intense signals cause faster reaction times, but there are diminishing returns: as a stimulus grows in intensity, eventually the reaction speed can't get any better. The formula that relates intensity and reaction speed is Pieron's Law.

It's a common illusion that if you are in a hurry for the elevator you can make it come quicker by pressing the button harder. Or more often. Or all the buttons at once. It somehow feels as if it ought to work, although of course we know it doesn't. Either the elevator has heard you, or it hasn't. How loud you call doesn't make any difference to how long it'll take to arrive.

But then elevators aren't like people. People do respond quicker to more stimulation, even on the most fundamental level. We press the brake quicker for brighter stoplights, jump higher at louder bangs. And it's because we all do this that we all fall so easily into thinking that things, including elevators, should behave the same way.

Give someone this simple task: she must sit in front of a screen and press a button as quickly as she can as soon as she sees a light flash on. If people were like elevators, the time it takes to press the button wouldn't be affected by the brightness of the light or the number of lights.

But people aren't like elevators and we respond quicker to brighter lights; in fact, the relationship between the physical intensity of the light and the average speed of response follows a precise mathematical form. This form is captured by an equation called Pieron's Law. Pieron's Law says that the time to respond to a stimulus is related to the stimulus intensity by the formula:

Reaction Time 
 R₀ + kI^-β

Reaction Time is the time between the stimulus appearing and you responding. I is the physical intensity of the signal. R ₀ is the minimum time for any response, the asymptotic value representing all the components of the reaction time that don't vary, such as the time for light to reach your eye.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

All abilities are skills; practice something and your brain will devote more resources to it.

The sensory homunculus looks like a person, but swollen and out of all proportion. It has hands as big as its head; huge eyes, lips, ears, and nose; and skinny arms and legs. What kind of person is it? It's you, the person in your head. Have a look at the sensory homunculus first, then make your own.

You can play around with Jaakko Hakulinen's homunculus applet (http://www.cs.uta.fi/~jh/homunculus.html; Java) to see where different bits of the body are represented in the sensory and motor cortex. There's a screenshot of it in Figure 1-3.

Figure 1-3: The figure shown is scaled according to the relative sizes of the body parts in the motor and sensory cortex areas; motor is shown on the left, sensory on the right

This is the person inside your head. Each part of the body has been scaled according to how much of your sensory cortex is devoted to it. The area of cortex responsible for processing touch sensations is the somatosensory cortex. It lives in the parietal lobe, further toward the back of the head than the motor cortex, running alongside it from the top of the head down each side of the brain. Areas for processing neighboring body parts are generally next to each other in the cortex, although this isn't always possible because of the constraints of mapping the 3D surface of your skin to a 2D map. The area representing your feet is next to the area representing your genitals, for example (the genital representation is at the very top of the somatosensory cortex, inside the groove between the two hemispheres).

The applet lets you compare the motor and sensory maps. The motor map is how body parts are represented for movement, rather than sensation. Although there are some differences, they're pretty similar. Using the applet, when you click on a part of the little man, the corresponding part of the brain above lights up. The half of the man on the left is scaled according to the representation of the body in the primary motor cortex, and the half on the right is scaled to represent the somatosensory cortex. If you click on a brain section or body part, you can toggle shading and the display of the percentage of sensory or motor representation commanded by that body part. The picture of the man is scaled, too, according to how much cortex each part corresponds to. That's why the hands are so much larger than the torso.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The puzzle that is vision lies in the chasm between the raw sensation gathered by the eye—light landing on our retinas—and our rich perception of color, objects, motion, shape, entire 3D scenes. In this chapter, we'll fiddle about with some of the ways the brain makes this possible.

We'll start with an overview of the visual system [Hack #13] , the limits of your vision [Hack #14] , and the active nature of visual perception [Hack #15] .

There are constraints in vision we usually don't notice, like the blind spot [Hack #16] and the 90 minutes of blindness we experience every day as vision deactivates while our pupils jump around [Hack #17] . We'll have a look at both these and also at some of the shortcuts and tricks visual processing uses to make our lives easier: assuming the sun is overhead [Hack #20] and [Hack #21] , jumping out of the way of rapidly expanding dark shapes [Hack #32] (a handy shortcut for faster processing if you need to dodge quickly), and tricks like the use of noisy neurons [Hack #33] to extract signal out of visual noise.

Along the way, we'll take in how we perceive depth [Hack #22] and [Hack #24] , and motion [Hack #25] and [Hack #29] . (That's both the correct and false perception of motion, by the way.) We'll finish off with a little optical illusion called the Rotating Snakes Illusion [Hack #30] that has all of us fooled. After all, sometimes it's fun to be duped.

The visual system is a complex network of modules and pathways, all specializing in different tasks to contribute to our eventual impression of the world.

When we talk about "visual processing," the natural mode of thinking is of a fairly self-contained process. In this model, the eye would be like a video camera, capturing a sequence of photographs of whatever the head happens to be looking at at the time and sending these to the brain to be processed. After "processing" (whatever that might be), the brain would add the photographs to the rest of the intelligence it has gathered about the world around it and decide where to turn the head next. And so the routine would begin again. If the brain were a computer, this neat encapsulation would be how the visual subsystem would probably work.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The puzzle that is vision lies in the chasm between the raw sensation gathered by the eye—light landing on our retinas—and our rich perception of color, objects, motion, shape, entire 3D scenes. In this chapter, we'll fiddle about with some of the ways the brain makes this possible.

We'll start with an overview of the visual system [Hack #13] , the limits of your vision [Hack #14] , and the active nature of visual perception [Hack #15] .

There are constraints in vision we usually don't notice, like the blind spot [Hack #16] and the 90 minutes of blindness we experience every day as vision deactivates while our pupils jump around [Hack #17] . We'll have a look at both these and also at some of the shortcuts and tricks visual processing uses to make our lives easier: assuming the sun is overhead [Hack #20] and [Hack #21] , jumping out of the way of rapidly expanding dark shapes [Hack #32] (a handy shortcut for faster processing if you need to dodge quickly), and tricks like the use of noisy neurons [Hack #33] to extract signal out of visual noise.

Along the way, we'll take in how we perceive depth [Hack #22] and [Hack #24] , and motion [Hack #25] and [Hack #29] . (That's both the correct and false perception of motion, by the way.) We'll finish off with a little optical illusion called the Rotating Snakes Illusion [Hack #30] that has all of us fooled. After all, sometimes it's fun to be duped.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The visual system is a complex network of modules and pathways, all specializing in different tasks to contribute to our eventual impression of the world.

When we talk about "visual processing," the natural mode of thinking is of a fairly self-contained process. In this model, the eye would be like a video camera, capturing a sequence of photographs of whatever the head happens to be looking at at the time and sending these to the brain to be processed. After "processing" (whatever that might be), the brain would add the photographs to the rest of the intelligence it has gathered about the world around it and decide where to turn the head next. And so the routine would begin again. If the brain were a computer, this neat encapsulation would be how the visual subsystem would probably work.

With that (admittedly, straw man) example in mind, we'll take a tour of vision that shows just how nonsequential it all really is.

And one need go no further than the very idea of the eyes as passive receptors of photograph-like images to find the first fault in the straw man. Vision starts with the entire body: we walk around, and move our eyes and head, to capture depth information [Hack #22] like parallax and more. Some of these decisions about how to move are made early in visual processing, often before any object recognition or conscious understanding has come into play.

This pattern of vision as an interactive process, including many feedback loops before processing has reached conscious perception, is a common one. It's true there's a progression from raw to processed visual signal, but it's a mixed-up, messy kind of progression. Processing takes time, and there's a definite incentive for the brain to make use of information as soon as it's been extracted; there's no time to wait for processing to "complete" before using the extracted information. All it takes is a rapidly growing dark patch in our visual field to make us flinch involuntarily

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The high-resolution portion of your vision is only the size of your thumbnail at arm's length. The rest of your visual input is low res and mostly colorless, although you seldom realize it.

Your vision isn't of uniform resolution. What we generally think of as our visual ability, the sharpness with which we see the world, is really only the very center of vision, where resolution is at its highest. From this high-resolution center, the lower-resolution periphery, and using continual movements of our head and eyes [Hack #15] , we construct a seamless—and uniformly sharp—picture of the universe. But how much are we compensating? What is the resolution of vision?

The eye's resolution is determined by the density of light-sensitive cells on the retina, which is a layer of these cells on the back of the eye (and also includes several layers of cells to process and aggregate the visual signals to send on to the rest of the brain). If the cells were spread evenly, we would see as well out of the corners of our eyes as directly ahead, but they're not. Instead, the cells are most heavily packed right in the center of the retina, a small region called the fovea, so the highest-resolution part of the vision is in the middle of your visual field. The area corresponding to this is small; if you look up at the night sky, out of everything you see, your fovea just about covers the full moon. Away from this, in your peripheral vision, resolution is much coarser.

Color also falls off in peripheral vision. The light-sensitive cells, called photoreceptors, come in different types according to what kinds of light they convert into neural signals. Almost all the photoreceptors that can discriminate colors of light are in the fovea. Outside of this central area you can still make out color, but it's harder; the oter type of cell, more sensitive but able to recognize only brightness, is more abundant.

Figure 2-1 is a variant of the usual eye chart you will have encountered at the optometrist, constructed by Stuart Anstis. Hold it in front of you, and rest your gaze on the central dot. The letters in the chart are smallest in the middle and largest at the outer edge; they scale up at a rate to exactly compensate for your eyes' decrease in resolution from the central fovea to the periphery.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Think of perception as a behavior, as something active, rather than as something passive. Perception exists to guide action, and being able to act is key to the construction of the high-resolution illusion of the world we experience.

The other hacks in this chapter could give the impression that seeing is just a matter of your brain passively processing the information that comes in through the eyes. But perception is far more of an active process. The impression we have of the world is made up by sampling across times, as well as just by sampling across the senses. The sensation we receive at any moment prompts us to change our head position, our attention, maybe to act to affect something out in the world, and this gives us different sensations in the next moment to update our impression of the world.

It's easier for your brain to take multiple readings and then interpolate the answers than it is to spend a long time processing a single scene. Equally important, if you know what you want to do, maybe you don't need to completely interpret a scene; you may need to process it just enough to let you decide what to do next and in acting give yourself a different set of sensations that make the scene more obvious.

This school of thought is an "ecological" approach to perception and is associated with the psychologist J. J. Gibson.¹ He emphasized that perception is a cognitive process and, like other cognitive processes, depends on interacting with the world. The situations used by vision scientists in which people look at things without moving or reaching out to touch them are extremely unnatural, as large as the difference between a movie at the theater directed by someone else and the freewill experience of regular real life.

If you want people to see something clearly, give them the chance to move it around and see how it interacts with other objects. Don't be fooled into thinking that perception is passive.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Find out how big your visual blind spot is and how your brain fills the hole so you don't notice it.

Coating the back of each eye are photoreceptors that catch light and convert it to nerve impulses to send to the brain. This surface, the retina, isn't evenly spread with receptors—they're densest at the center and sparse in peripheral vision [Hack #14] . There's also a patch that is completely devoid of receptors; light that falls here isn't converted into nerve signals at all, leaving a blind spot in your field of view—or actually two blind spots, one for each eye.

First, here's how to notice your blind spot (later we'll draw a map to see how big it is). Close your left eye and look straight at the cross in Figure 2-6. Now hold the book flat about 10 inches from your face and slowly move it towards you. At about 6 inches, the black circle on the right of the cross will disappear, and where it was will just appear grey, the same color as the page around it.

Figure 2-6: A typical blind spot pattern

You may need to move the book back and forth a little. Try to notice when the black circle reappears as you increase the distance, then move the book closer again to hide the circle totally. It's important you keep your right eye fixed on the cross, as the blind spot is at a fixed position from the center of vision and you need to keep it still to find it.

Now that you've found your blind spot, use Jeffrey Oristaglio and Paul Grobstein's Java applet at the web site Serendip (http://serendip.brynmawr.edu/bb/blindspot; Java) to plot its size.

The applet shows a cross and circle, so, as before, close your left eye, fix your gaze on the cross, and move your head so that the circle disappears in your blind spot. Then click the Start button (at the bottom of the applet) and move your cursor around within the blind spot. While it's in there, you won't be able to see it, but when you can (only just), click, and a dot will appear. Do this a few times, moving the cursor in different directions starting from the circle each time.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Our eyes constantly dart around in extremely quick movements called saccades. During each movement, vision cuts out.

Despite the fact that the eye has a blind spot, an uneven distribution of color perception, and can make out maximal detail in only a tiny area at the center of vision, we still manage to see the world as an uninterrupted panorama. The eye jumps about from point to point, snapshotting high-resolution views, and the brain assembles them into a stunningly stable and remarkably detailed picture.

These rapid jumps with the eyes are called saccades, and we make up to five every second. The problem is that while the eyes move in saccade all visual input is blurred. It's difficult enough for the brain to process stable visual images without having to deal with motion blur from the eye moving too. So, during saccades, it just doesn't bother. Essentially, while your eyes move, you can't see.

Put your face about 6 inches from a mirror and look from eye to eye. You'll notice that while you're obviously switching your gaze from eye to eye, you can't see your own eyes actually moving—only the end result when they come to rest on the new point of focus. Now get someone else to watch you doing so in the mirror. They can clearly see your eyes shifting, while to you it's quite invisible.

With longer saccades, you can consciously perceive the effect, but only just.

Hold your arms out straight so your two index fingers are at opposite edges of your vision. Flick your eyes between them while keeping your head still. You can just about notice the momentary blackness as all visual input from the eyes is cut off. Saccades of this length take around 200 ms (a fifth of a second), which lies just on the threshold of conscious perception.

What if something happens during a saccade? Well, unless it's really bright, you'll simply not see it. That's what's so odd about saccades. We're doing it constantly, but it doesn't look as if the universe is being blanked out a hundred thousand times a day for around a tenth of a second every time.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Our sense of time lends a seamless coherence to our conscious experience of the world. We are able to effortlessly distinguish between the past, present, and future. Yet, subtle illusions show that our mental clock can make mistakes.

You only have to enjoy the synchrony achieved by your local orchestra to realize that humans must be remarkably skilled at judging short intervals of time. However, our mental clock does make mistakes. These anomalies tend to occur when the brain is attempting to compensate for gaps or ambiguities in available sensory information.

Such gaps can be caused by self-generated movement. For example, our knowledge about how long an object has been in its current position is compromised by the suppression of visual information [Hack #17] that occurs when we move our eyes toward that object—we can have no idea what that object was actually doing for the time our eyes were in motion. This uncertainty of position, and the subsequent guess the brain makes, can be felt in action by saccading the eyes toward a moving object.

Sometimes you'll glance at a clock and the second hand appears to hang, remaining stationary for longer than it ought to. For what seems like a very long moment, you think the clock may have stopped. Normally you keep looking to check and see that shortly afterward the second hand starts to move again as normal—unless, that is, it truly has stopped.

This phenomenon has been dubbed the stopped clock illusion. You can demonstrate it to yourself by getting a silently moving clock and placing it off to one side. It doesn't need to be an analog clock with a traditional second hand; it can be a digital clock or watch, just so long as it shows seconds. Position the clock so that you aren't looking at it at first but can bring the second hand or digits into view just by moving your eyes. Now, flick your eyes over to the clock (i.e., make a saccade

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

It takes longer to shift your attention to a new object if the old object is still there.

Shifting attention often means shifting your eyes. But we're never fully in control of what our eyes want to look at. If they're latched on to something, they're rather stubborn about moving elsewhere. It's faster for you to look at something new if you don't have to tear your eyes away—if what you were originally looking at disappears and then there's a short gap, it's as if your eyes become unlocked, and your reaction time improves. This is called the gap effect.

The gap effect can be spotted if you're asked to stare at some shape on a screen, then switch your gaze to a new shape that will appear somewhere else on the screen. Usually, switching to the new shape takes about a fifth of a second. But if the old shape vanishes shortly before the new shape flashes up, moving your gaze takes less time, about 20% less.

It has to be said: the effect—on the order of just hundredths of a second—is tiny in the grand scheme of things. You're not going to notice it easily around the home. It's a feature of our low-level cognitive control: voluntarily switching attention takes a little longer under certain circumstances. In other words, voluntary behavior isn't as voluntary as we'd like to think.

We take in the world piecemeal, focusing on a tiny part of it with the high-resolution center of our vision for a fraction of a second, then our eyes move on to focus on another part. Each of these mostly automatic moves is called a saccade [Hack #15] .

We make saccades continuously—up to about five every second—but that's not to say they're fluid or all the same. While you're taking in a scene, your eyes are locked in. They're resistant to moving away, just for a short time. So what happens when another object comes along and you want to move your eyes toward it? You have to overcome that inhibition, and that takes a short amount of time.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

How do you figure out the three-dimensional shape of objects, just by looking? At first glance, it's using shadows.

Looking at shadows is one of many tricks we use to figure out the shape of objects. As a trick, it's easy to fool—shading alone is enough for the brain to assume what it's seeing is a real shadow. This illusion is so powerful and so deeply ingrained, in fact, that we can actually feel depth in a picture despite knowing it's just a flat image.

Have a look at the shaded circles in Figure 2-8, following a similar illustration in Kleffner and Ramachandran's "On the Perception of Shape from Shading."¹

Figure 2-8: Shaded figures give the illusion of three-dimensionality

I put together this particular diagram myself, and there's nothing to it: just a collection of circles on a medium gray background. All the circles are gradient-filled black and white, some with white at the top and some with white at the bottom. Despite the simplicity of the image, there's already a sense of depth.

The shading seems to make the circles with white at the top bend out of the page, as though they're bumps. The circles with white at the bottom look more like depressions or even holes.

To see just how strong the sense of depth is, compare the shaded circles to the much simpler diagram in Figure 2-9, also following Kleffner and Ramachandran's paper.

Figure 2-9: Binary black-and-white "shading" doesn't provide a sense of depth

The only difference is that, instead of being shaded, the circles are divided into solid black and white halves. Yet the depth completely disappears.

Shadows are identified early in visual processing in order to get a quick first impression of the shape of a scene. We can tell it's early because the mechanism it uses to resolve light source ambiguities is rather hackish.

Ambiguities occur all the time. For instance, take one of the white-at-top circles from Figure 2-8. Looking at it, you could be seeing one of two shapes depending on whether you imagine the shape was lit from the top or the bottom of the page. If light's coming from above, you can deduce it's a bump because it's black underneath where the shadows are. On the other hand, if the light's coming from the bottom of the page, only a dent produces the same shading pattern. Bump or dent: two different shapes can make the same shadow pattern lit from opposite angles.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Moving shadows make us see moving objects rather than assume moving light sources.

Shadows get processed early when trying to make sense of objects, and they're one of the first things our visual system uses when trying to work out shape. [Hack #20] further showed that our visual system makes the hardwired assumption that light comes from above. Another way shadows are used is to infer movement, and with this, our visual system makes the further assumption that a moving shadow is the result of a moving object, rather than being due to a moving light source. In theory, of course, the movement of a shadow could be due to either cause, but we've evolved to ignore one of those possibilities—rapidly moving objects are much more likely than rapidly moving lights, not to mention more dangerous.

Observe how your brain uses shadows to construct the 3D model of a scene. Watch the ball-in-a-box movie at:

Table of Contents