Subtitles section Play video
- [automated message] This UCSD TV program is presented by University of California
Television. Like what you learn? Visit our website, or follow us on Facebook and
Twitter to keep up with the latest programs.
♪ [music] ♪
- [male] We are the paradoxical ape, bi-pedal, naked, large brain, long the
master of fire, tools, and language, but still trying to understand ourselves.
Aware that death is inevitable, yet filled with optimism. We grow up slowly. We hand
down knowledge. We empathize and deceive. We shape the future from our shared
understanding of the past. CARTA brings together experts from diverse disciplines
to exchange insights on who we are and how we got here. An exploration made possible
by the generosity of humans like you.
♪ [music] ♪
- [Evelina] Thanks very much for having me here. So today, I will tell you about the
language system and some of its properties that may bear on the question of how this
system came about. I will begin by defining the scope of my inquiry, because
people mean many different things by language. I will then present you some
evidence, suggesting that our language system is highly specialized for language.
Finally I will speculate a bit on the evolutionary origins of language.
So linguistic input comes in through our ears or our eyes. Once it's been analyzed
perceptually, we interpret it by linking the incoming representations toward our
stored language knowledge. This knowledge, of course, is also used for generating
utterances during language production. Once we've generated an utterance in our
heads, we can send it down to our articulation system. The part that I focus
on is this kind of high-level component of language processing.
This cartoon shows you the approximate locations of the brain regions that
support speech perception, shown in yellow, visual letter and word recognition
in green. This region is known as the visual word-form area. Articulation in
pink and high-level language processing in red.
What differentiates the high-level language processing regions from the
perceptual and articulation regions is that they're sensitive to the
meaningfulness of the signal. So for example, the speech regions will respond
just as strongly to foreign speech as they respond to meaningful speech in our own
language. The visual word-form area responds just as much to a string of
consonants as it does to real words. The articulation regions can be driven to a
full extent by people producing meaningless sequences of syllables.
But in contrast, the high-level language-processing system or network, or
sometimes I refer to it just as the language network for short, cares deeply
about whether the linguistics signal is meaningful or not.
In fact, the easiest way to find this system in the brain is to contrast
responses to meaningful language stimuli, like words, phrases, or sentences. Some
control conditions like linguistically degraded stimuli.
The contrast I use most frequently is between sentences and sequences of
non-words. A key methodological innovation that laid the foundation for much of my
work was the development of tools that enable us to define the regions of the
language network, functionally at the individual subject level, using contrasts
like these. Here, I'm showing you sample language regions in three individual
brains. This so-called functional localization approach has two advantages.
One, it circumvents the need to average brains together, which is what's done in
the common approach, and it's a very difficult thing to do because brains are
quite different across people. Instead, in this approach, we can just average the
signals that we extract from these key regions of interest.
The second advantage is that it allows us to establish a cumulative research
enterprise, which I think we all agree is important in science, because comparing
results across studies and labs is quite straightforward if we're confident that
we're referring to the same brain regions across different studies. This is just
hard or impossible to do in the traditional approach, which relies on very
coarse anatomical landmarks, like the inferior frontal gyrus or the superior
temporal sulcus, which span many centimeters of cortex and are just not at
the right level of analysis. So what drives me and my work is the
desire to understand the nature of our language knowledge and the computations
that mediate language comprehension and production. However, these questions are
hard, especially given the lack of animal models for language. So for now, I settle
on more tractable questions. For example, one, what is the relationship between the
language system and the rest of human cognition? Language didn't evolve, and it
doesn't exist in isolation from other evolutionarily older systems, which
include the memory and attention mechanisms, the visual and the motor
systems, the system that supports social cognition, and so on. That means that we
just can't study language as an isolated system. A lot of my research effort is
aimed at trying to figure out how language fits with the rest of our mind and brain.
The second question delves inside the language system, asking, "What does its
internal architecture look like?" It encompasses questions like, "What are the
component parts of the language system? And what is the division of labor among
them, in space and time?" Of course, both of those questions
ultimately should constrain the space of possibilities for how language actually
works. So they place constraints on both the data structures that underlie language
and the computations that are likely performed by the regions of the system.
Today, I focus on the first of these questions. Okay. So now onto some
evidence. So the relationship between language and the rest of the mind and
brain has been long debated, and the literature actually is quite abundant with
claims that language makes use of the same machinery that we use for performing other
cognitive tasks, including arithmetic processing, various kinds of executive
function tasks, perceiving music, perceiving actions, abstract conceptual
processing, and so on. I will argue that these claims are not
supported by the evidence. Two kinds of evidence speak to the relationship between
language and other cognitive systems. There is brain-imaging studies,
brain-imaging evidence, and investigations of patients with brain damage. In fMRI
studies, we do something very simple. We find our regions that seem to respond a
lot to language, and then we ask how do they respond to other various
non-linguistic tasks. If they don't show much of a response, then we can conclude
that these regions are not engaged during those tasks. In the patient studies, we
can evaluate non-linguistic abilities in individuals who don't have a functioning
language system. If they perform well, we can conclude that the language system is
not necessary for performing those various non-linguistic tasks. So starting with the
fMRI evidence, I will show you responses in the language regions to arithmetic,
executive function tasks, and music perception today. So here are two sample
regions, the regions of the inferior frontal cortex around Broca's area and the
region in the posterior middle temporal gyrus, but the rest of the regions of this
network look similar in their profiles. The region on the top is kind of in and
around this region known as Broca's area, except I don't use that term because I
don't think it's a very useful term. In black and gray, I show you the responses
to the two localizer conditions, sentences and non-words. These are estimated in data
that's not used for defining these regions. So we divide the data in half,
use half of the data to define the regions and the other half to quantify their
responses. I will now show you how these regions respond when people are asked to
do some simple arithmetic additions, perform a series of executive function
tasks, like for example, hold a set of locations in spacial memory, spacial
locations in working memory, or perform this classic flanker task, or listen to
various musical stimuli. For arithmetic and various executive tasks, we included a
harder and an easier condition, because we wanted to make sure that we can identify
regions that are classically associated with performing these tasks, which is
typically done by contrasting a harder and an easier condition of a task.
So I'll show you now, in different colors, responses to these various tasks, starting
with the region on the lower part of the screen. So we find that this region
doesn't respond during arithmetic processing, doesn't respond during working
memory, doesn't respond during cognitive control tasks, and doesn't respond during
music perception. Quite strikingly to me at the time, a very similar profile is
observed around this region, which is smack in the middle of so-called Broca's
area, which appears to be incredibly selective in its response for language.
Know that it's not just the case that these, for example, demanding tasks fail
to show a hard versus an easy difference. They respond pretty much at or below
fixation baseline when people are engaged in these tasks. So that basically tells
you that these language regions work as much when you're doing a bunch of math in
your head or hold information in working memory, as what you're doing when you're
looking at a blank screen. So they really do not care. So of course, to interpret
the lack of the response in these language regions, you want to make sure that these
tasks activate the brain somewhere else. Otherwise, you may have really bad tasks
that you don't want to use. Indeed, they do.
So here, I'll show you activations for the executive function tasks, but music also
robustly activates the brain outside of the language system. So here are two
sample regions, one in the right frontal cortex, one in the left parietal cortex.
You see the profiles of response are quite different from the language regions. For
each task, we see robust responses, but also a stronger response to the harder
than the easier condition across these various domains. These regions turn out to
be part of this bilateral frontal parietal network, which is known in the literature
by many names, including the cognitive control network or the multiple demand
system, the latter term advanced by John Duncan, who wanted to highlight the notion
that these regions are driven by many different kinds of cognitive demands. So
these regions appeared to be sensitive to effort across tasks, and their activity
has been linked to a variety of goal-directed behaviors. Interestingly if
you look at the responses of these regions to our language localizer conditions, we
find exactly the opposite of what we find in the language regions. They respond less
to sentences than sequences of non-words, presumably because processing sentences
requires less effort, but clearly this highlights, again, the language and the
cognitive control system are clearly functionally distinct.
Moreover, damage to the regions of the multiple demand network has been shown to
lead to decreases in fluid intelligence. So Alex Woolgar reported a strong
relationship between the amount of tissue loss in frontal and parietal cortices and
a measure of IQ. This is not true for tissue loss in the temporal lobes. It's
quite striking. You can actually calculate for this many cubic centimeters of loss in
the MD system, you lose so many IQ points. It's a strong, clear relationship. So this
system is clearly an important part of the cognitive arsenal of humans because the
ability to think flexibly and abstractly and to solve new problems are exactly . .
. These are the kinds of abilities that IQ tests aim to measure, are considered kind
of one of the hallmarks of human cognition. Okay. So as I mentioned, the
complementary approach for addressing questions about language specificity and
relationship to other mental functions is to examine cognitive abilities in
individuals who lack a properly functioning language system. Most telling
are cases of global aphasia. So this is a severe disorder which affects pretty much
the entire front temporal language system, typically due to a large stroke in the
middle cerebral artery and lead to profound deficits in comprehension and
production. Rosemary Varley at UCL has been studying this population for a few
years now. With her colleagues, she has shown that
actually these patients seem to have preserved abilities across many, many
domains. So she showed that they have in-tact arithmetic abilities. They can
reason causally. They have good nonverbal social skills. They can navigate in the
world. They can perceive music and so on and so forth. Of course, these findings
are then consistent with the kind of picture that emerges in our work in fMRI.
Let's consider another important non-linguistic capacity, which a lot of
people often bring up when I tell them about this work. How about the ability to
extract meaning from non-linguistic stimuli? Right? So given that our language
regions are so sensitive to meaning, we can ask how much of that response is due
to the activation of some kind of abstract, conceptual representation that
language may elicit, rather than something more language-specific, a semantic
representation type. So to ask these questions, we can look at how language
regions respond to nonverbal, meaningful representations. In one study, we had
people look at events like this or the sentence-level descriptions of them, and
either we had them do kind of a high-level semantic judgment test, like decide
whether the event is plausible, or do a very demanding perceptual control task.
Basically what you find here is, again, the black and gray are responses to the
localizer conditions. So in red, as you would expect, you find strong responses to
this and to the condition where people see sentences and make semantic judgments on
them. So what happens when people make semantic judgments on pictures? We find
that some regions don't care at all about those conditions, and other regions show
reliable responses, but they're much weaker than those elicited by the
meaningful sentence condition. So could it be that some of our language regions are
actually abstract semantic regions? Perhaps. But for now, keep in mind that
the response to the sentence-meaning condition is twice stronger, and it is
also possible that participants may be activating linguistic representations to
some extent when they encounter meaningful visual stimuli. So to answer this question
more definitively, we're turning to the patient evidence again. If parts of the
language system are critical for processing meaning in non-linguistic
representations, then aphasic individuals should have some difficulties with
nonverbal semantics. First, I want to share a quote with you from Tom Lubbock, a
former art critic at The Independent, who developed a tumor in the left temporal
lobe which eventually killed him. As the tumor progressed, and he was losing his
linguistic abilities, he was documenting his impressions of what it feels like to
lose the capacity to express yourself using verbal means.
So he wrote, "My language to describe things in the world is very small,
limited. My thoughts, when I look at the world, are vast, limitless, and normal,
same as they ever were. My experience of the world is not made less by lack of
language, but is essentially unchanged." I think this quote quite powerfully
highlights the separability of language and thought. So in work that I'm currently
collaborating on with Rosemary Varley and Nancy Kanwisher, we are evaluating the
global aphasics performance on a wide range of tasks, requiring you to process
meaning in nonverbal stimuli. So for example, can they distinguish between real
objects and novel objects that are matched for low-level visual properties? Can they
make plausibility judgments for visual events? What about events where
plausibility is conveyed simply by the prototypicality of the roles? So you can't
do this task by simply inferring that a watering can doesn't appear next to an egg
very frequently. Right? It seems like the data so far is suggesting that they indeed
seem fine on all of these tasks, and they laugh just like we do when they see these
pictures because they're sometimes a little funny. So they seem to process
these just fine. So this suggests, to me, that these kinds of tasks can be performed
without a functioning language system. So even if our language system stores some
abstract conceptual knowledge in some parts of it, it tells me at least that
that code must live somewhere else as well. So even if we lose our linguistic
way to encode this information, we can have access to it elsewhere.
So to conclude this part, fMRI in patients sudies converge suggesting that the front
temporal language system is not engaged in and is not needed for non-linguistic
cognition. Instead, it appears that these regions are highly specialized for
interpreting and generating linguistics signals. So just a couple minutes on what
this means. So given this highly selective response to language stimuli that we
observe, can we make some guesses already about what these regions actually do? I
think so. I think a plausible hypothesis is that this network houses our linguistic
knowledge, including our knowledge of the sounds of the language, the words, the
constraints on how sounds and words can combine with one another. Then essentially
the process of language interpretation is finding matches between the pieces of the
input that are getting into our language system and our previously stored
representations. Language production is just selecting the relevant subset of the
representations to then convey to our communication partner. This way . . . The
form that this knowledge takes is a huge question in linguistic psychology and
neuroscience. So one result I don't have time to discuss is that contra some
claims, it doesn't seem to be the case that syntactic processing is localized to
a particular part of this language system. It seems it's widely distributed across.
Anywhere throughout the system, you find sensitivity to both word-level meanings
and compositional aspects of language, which is much in line with all current
linguistic theorizing, which doesn't draw a short boundary between the lexicon and
grammar. So this way of thinking about the language system as a store of our language
knowledge makes it pretty clear that the system is probably not innate. In fact, it
must arise via experience with language as we accumulate this language store. It's
also presumably dynamic, changing all the time as we get more and more linguistic
input through our lifetimes. I assume that our language knowledge is plausibly
acquired with domain general statistical learning mechanisms, just like much other
knowledge. So what changed in our brains that allowed for the emergence of this
system? So one thing that changed is that our association cortices expanded. So
these are regions that are sensory and motor regions and include frontal,
temporal, and parietal regions. Okay. So these people have noted for a long time. I
think I'm kind of in the camp of people who think that our brains are not
categorically different in any way. They're just scaled-up versions of other
primate brains. I think there's quite good evidence for that. So how does the system
emerge? So I think one thing that was different
between us and chimps is that there is a protracted course to the brain development
in humans. So between birth and adulthood, our brains increase threefold, compared to
just twofold in chimps. It's a big difference. Basically this just makes us
exceptionally susceptible to environmental influences, and we can soak stuff up from
the environment very, very easily. So as our brains grow, we make more glial cells.
We make more synapses. Our axions continue to grow and become myelinated, and it's
basically tissue that's ready to soak up the regularity that we see in the world.
Of course, it comes at a cost. That's why we have totally useless babies that can't
do anything, but apparently somehow it was worth. . . The tradeoffs were worth it.
Okay. So the conclusions. We have this system. It's highly selective in its
responses. It presumably emerges over the course of our development and would enable
probably some combination of the expansion of these association cortices, where we
can store vast amounts of symbolic information and this protracted brain
development, which makes us great learners early on. Thank you.
[applause]
♪ [music] ♪
- [Rachel] So I want to start this story actually in the 1800s. So in 1800, a young
physician named Itard decided to take on the task of teaching a young boy French.
Turns out that this young boy, who they think was between 10 and 12 years old, he
had no language. He'd been discovered running around in the woods, naked and
unable to communicate. Itard thought that this was a very important task because he
thought that civilization was based on the ability to empathize and also on language.
So he tried valiantly, for two years, to teach this young boy named Victor . . . He
named him Victor because over this two-year time span, the only language or
sounds that this boy could make was the French 'er', which sounds like Victor.
Hence, he had his name. But after two years, Victor was unable to speak any
French and comprehended very little French and primarily communicated with objects.
Then Itard wrote this up after two years. He wrote up his findings, and he said that
he thought that a major reason why Victor didn't learn French was because he was
simply too old. But of course, this was the 1800s, and he
didn't speculate as to what it was about being 10 years old with no language that
would prevent you from learning language if you had a daily tutor trying to teach
you French. There's an enormous amount of irony in this particular story, because
Itard was the house physician for the first school for the deaf in the entire
world. Here, we have a picture of it. The first school for the deaf was begun in
1760 in Paris. At the time that Itard was teaching Victor, he was at the school with
all of these children who used sign language and all of these teachers who
used sign language. Nonetheless, it never occurred to him to try to teach Victor
sign language, but we can't blame him because in the 1800s, sign language was
not considered to be a language. In fact, that particular discovery and realization
wouldn't happen for another century and a half. So we can imagine why he didn't
teach Victor sign language. The question is: could Victor have actually learned
sign language if Itard had tried to teach it to him? So I'm going to try to answer
that question today through a series of studies. But before I talk about our
studies, I want to talk about something that all of the speakers who have preceded
me have talked about, which is that one thing that's . . . The defining
characteristic of language is that it's highly structured, and it's highly
structured at all of these multiple levels, so that speech sounds make up
words. Words make up words and phonemes. These words are strung together in
sentences with syntax, and the specific syntax helps us understand and produce
very specific kinds of meanings. Now one aspect of this language structure
is that humans have evolved to the point where children learn this structure
naturally. Nobody has to teach them. Nobody has to have a tutor to sit with
them for two years to teach them the structure of French or the structure of
English or the structure of ASL. Simply by being around people who use the language,
young children naturally acquire all of this multilevel and complicated structure.
That is to say all children do this if, in fact, they can access the language around
them, if they have normal hearing. They're born with normal hearing. They hear people
talk. Before you know it, they're talking themselves. But if children are born
profoundly deaf, they cannot hear the speech around them. We know that
lip-reading is insufficient to learn language because most of the speech sounds
are invisible. What happens to these children? If there's no sign language in
the environment, they can't learn a visual form of language either. So it happens
that there are large numbers of children who actually are like Victor, in the sense
that they grow up without language, without learning a language, but they are
like Victor-- they are unlike Victor, in that they weren't running around in the
woods, nude and having a very harsh life. So how does the lack of language in
childhood affect the ability to learn language? Or does it affect the ability to
learn language? This has been the focus of studies that we have been doing for many
years in our laboratory. Because we're using deaf children and sign language as a
means to model language acquisition and its effect on the brain, I think it's
appropriate that I talk a little bit about the kinds of stimuli we do and about
American Sign Language. So first of all, you should know that American Sign
Language, unlike many of the sign languages that have been discussed up to
this point, is a very sophisticated language. It's evolved clearly over 200
years. We might even say that American Sign Language evolved with the development
of the United States of America and spread as civilization went across, as white men
went across the continent. So American Sign Language has a phonological system, a
morphological system, syntax, and so forth. So for those of you who don't know
sign . . . I know there are many people here who do know sign. I want to show you
what some of this structure looks like. So I'm going to play you two video tapes. For
those of you who don't know sign, I would like you to guess which one is
syntactically structured. For those of you who do know sign, maybe you could keep the
answer to yourselves.
That's one. Here's two.
So how many think two? How many think one? Okay. For those of you who think two, I
captioned this. So this is really 'kon, dird, lun, blid, mackers, gancakes.'
Number one is a fully formed sentence with a subordinate clause.
The reason I'm showing you these two sentences is, for those of you who don't
know the language, you can't perceive or parse this particular structure. This is
what knowing a language is about. For those of you who know ASL and who know
this language, you know that the second example had all of these signs which were
non-signs, possible signs, but really just non-signs. This is part of what knowing
language is about. How do people learn this particular structure? The question
that we're interested in is: how does being a young child help people learn this
particular structure? So we did a series of experiments. When we started this work,
it wasn't even clear that age would make a difference in sign language acquisition.
Sign language is gestural. Sign language is mimetic. Maybe anybody can learn sign
language at any time in their lives. So in one experiment, what we did is we
recruited a number of people. This is in Canada, who were born deaf and who used
ASL. We asked them. We created an experiment where we had a set of sentences
that varied in complexity, and we showed them these ASL sentences, and we asked
them simply to point to a picture that reflected the meaning of the sentence that
they saw. We were quite struck by our findings. What you see here is that deaf
people who learned ASL from birth, from their parents, performed very, very well
on this task, in contrast to deaf people who were adults, who've been signing for
over 20 years performed at chance. So they had great difficulty understanding some of
these basic sentences in ASL. So this suggests that there are age of acquisition
effects for sign, as there are for spoken language. Everybody sort of . . . The word
on the street is it's much harder to learn a language if you're an adult than you're
a child. But what if it's something deeper than this? What if there's something about
learning a language in childhood that sets up the ability to learn language, that
creates the ability to learn language? So we did another experiment, also in Canada,
but we decided in order to test this particular hypothesis, we should switch
languages. So we're no longer testing ASL here. What we're testing here is English.
We devised an experiment in English, where we had a set of sentences, and some of
them were ungrammatical, and some of them were grammatical. This is a common kind of
task that psycholinguists use. Notice here that the people who were born
profoundly deaf and for whom ASL was a first language are near-native in English.
So this is a second language. So learning a language early, even though it's in
sign, helps people learn a second language. Notice also that they performed
. . . Their performance was indistinguishable from normally-hearing
people who had learned other languages at birth, German, Urdu, Spanish, and French.
So there seems to be that there's something about learning a language early
in life, regardless of whether it's sign language or spoken language, that actually
helps people learn more language. It's not simply learning language when one is
little. But as also part of this experiment, we tested a group of
individuals who had been signing for 20 years and had gone through the educational
system in Canada, who were born deaf. On this task, they performed at chance. On a
grammaticality judgment task, it's either yes or no. So it's at chance. So we see
that individuals who are deprived of language, who aren't able to learn
language at a young age, perform poorly on their primary language, sign language, and
they perform very poorly on a second language, which is ASL, and we see the
reverse. So there's something really special going on here about learning
language at an early age. What might this be? And might it be in the brain?
So in another set of studies, what we did is take this population that we had been
looking at, and we decided to neuro-image their language processing to see whether
this might give us some clues as to differences between first and
second-language learners and people who had language and people who did not have
language. So in this study, also done in Canada, with colleagues at the Montreal
Neurological Institute, we did fMRI. Maybe many of you have had MRIs. We showed the
subjects sentences, like the sentences that you saw, and we asked them to make
grammatical judgments on these sentences. We tested 22 people. They were all born
profoundly deaf. They all used American Sign Language as a primary language. They
had all gone through the educational system, but they ranged in the age at
which they were first able to acquire language. This is all the way from birth
up to age 14. So if the age at which you learn your first language doesn't make a
difference, then we should expect the neuro-processing patterns of all of these
individuals to be similar. If age of acquisition makes a difference, we should
see different patterns in the brain. In fact, this is what we have. This is what
we found. When we did the analysis, we found that there were seven regions in the
brain, primarily in the left hemisphere. As you know now, the left hemisphere has
areas that are responsible for language. One effect that we found was that in the
language regions of the left hemisphere, the earlier the person learned their first
language, the more activation we saw in the language hemisphere. However, the
older the person was when they learned their first language, the less activation
we got in the language areas of the brain. So if there's less activation, is there
something else going on here? We actually found a second effect, which we were not
expecting at all, which is in the back part of the brain, the posterior part of
the brain, in visual processing. This particular effect was that the longer the
person matured, the older the . . . Without language, the older they were when
they learned language, we found greater activation, more neural resources being
devoted to visual processing. So we see that here in this group of deaf signers,
we have two complementary reciprocal effects of when a child learns his or her
. . .when an individual learns their first language and what the brain seems to be
doing in terms of processing that language. So that for people who learned
language early in life, almost all of their neural resources are devoted to
processing the meaning and structure of that language. For people who learned
their first language later in life, more neural resources are devoted to just
trying, perhaps, to figure out what the signal was. Was this a word? Was it glum?
Or was it gleam? So we have this reciprocal relationship between perceptual
processing and language processing. This particular pattern is not unique to deaf
signers. There's work by Tim Brown and Shleger that showed that younger children
often have more posterior activation than older children. There are also some
clinical populations, such as autistic individuals, particularly those who have
low . . . whose language skills are not well-developed, will often show more
processing in the occipital lobe. So this is not a pattern that is unique to
deafness. So then the next question we had is
whether, in fact . . . How does language develop when an individual first starts to
learn it when they're much older, for example, when they're a teen? We have been
very fortunate to have followed five or six children in our laboratory who had no
language until they were 13 to 14 years of age, for a variety of reasons. Two of
these children are from the United States. These other children are from other
countries. Actually this particular circumstance, while we might think of it
as being very rare, is actually very common, particularly in underdeveloped
countries. So the way in which we have observed or analyzed language acquisition
is to use normal procedures that people use to study children's language
acquisition. We get a lot of spontaneous data from them, and we analyze it. So one
question we had is: if you're 13 years old, and you don't have a language system,
will you develop language like a baby? Or will you do something else? Because you
have a developed cognitive system. Will you jump in the middle of the task? How
will this progress? To answer this question, we need to look a little bit at
how normally-hearing children or deaf or hearing children develop language when
they are exposed to it as a young age. The major hallmark of children's language
acquisition is that they very quickly, as they're acquiring the grammar of their
language, their sentences get longer and longer.
The reasons their sentences get longer and longer is because they're learning all of
these . . .the morphology, the syntax. As they say ideas, as they're expressing
their ideas, they're better able to use grammar to express them. So these data
show the average length of children's expressions. Two of these children are
normally hearing and acquiring English, and two of these children are acquiring
ASL. So we see that, in fact, the teens that we have been following show no
increase in their language. They're able to learn language and put words together,
but, in fact, we don't see an increase in their grammar. In the last study, we
wanted to neuro-image these children. We wanted to see what are their brains doing
with the language that they have. So we used magnetoencephalography, which is a
different technique, which is complementary to the fMRI. What we did for
this is we studied their vocabulary, and we made stimuli that we knew that they
knew, words that were in their vocabulary, and that looked something like this. In
the first instance, the picture matched the sign. In the second instance, it
didn't. When that happens, the brain goes, "Uh-oh," and you get this N-400 response.
That's what we were localizing in the brain for these children. Because we're
using vocabulary that they have, we know that they knew these words. We asked them
to press buttons while they were doing this task, and we knew that they were
accurate. We didn't only test these children. We also tested control groups.
So some of these control groups are deaf. Some are hearing. Some are first language
learners, and some are second language learners.
The first panel shows the response of a group of normally hearing adults doing
this task while looking at pictures and listening to words. This is data that
Katie Travis used also to look at children's development, neural
development. The second panel, these are deaf adults who learned ASL from birth.
You can see that their processing is very much like the hearing adults who are
speaking English, primarily left hemisphere in the language areas, with
some support or help from the right hemisphere. Actually these patterns are
indistinguishable. Both the hearing adults and the deaf adults learned language from
birth, even though it was in a different form. What's this last panel? These are
college students who are normally hearing. They have been learning sign for about
three years, which is about the same amount of time that our cases were
learning language. So we see that responding on this task in speech in ASL,
whether it's a first language or a second language, so long as the subject had
language from birth, looks fairly similar. What about the cases? We were able to
neuro-image two cases, and you can see that their neural processing patterns are
quite different, and they look neither like second language learners, nor do they
look like deaf adults. These children have been signing for three years, and they had
no language before they started to learn how to sign. You can see primarily that
there's a huge response in the right hemisphere, in the occipital and parietal
areas. One of the subjects also shows some response in the language areas.
We can see that even though they're acquiring language, they're doing it in a
very different way, and their brains are responding very differently. So we see
that, in fact, there are huge effects of language environment on both the
development of language, but also how the brain processes language. So we see that
it seems to be that the human language capacity, both understanding language and
expressing language, but also the brain's ability to process language is very
dependent upon the baby's brain being exposed to or immersed in language from
birth. It's through this analyzing language and working on the data that it's
being fed that, in fact, I think the neural networks of language are being
created. So language is a skill that is not innate but emerges from the
interaction of the child with the environment, through linguistic
communication. That was probably the answer that Itard was looking for and
might be the reason why Victor did not acquire language. Thank you.
[applause]
♪ [music] ♪
- [Edward] Okay. So I wanted to first thank the organizers really for this kind
invitation to join this cast of stellar thinkers about the biology and behavior of
language. I actually want to switch the title a little bit to something more
specific. I want to talk to you about organization, in particular a kind of
organization that I refer to as a taxonomy. The organization that I'm
actually really referring to is the organization of sound, in particular
speech sounds and how those are actually processed in a very important part of the
brain called the superior temporal gyrus, also known as Wernicke's area. The main
focus of my lab is actually to understand the basic transformation that occurs when
you have an acoustic stimulus and how it becomes transformed into phonetic units.
In other words, basically, how do we go from the physical stimulus that enters our
ears into one that's essentially a mental construct, one that's a linguistic one? To
basically ask the simple question, what is the structure of that kind of information
as it's processed in the brain? Now this actually turns out to be a very
complex problem because it's one that actually arises from many levels of
computation that occur in the ascending auditory system. As sounds actually come
through the ears, they go up through at least seven different synaptic connections
across many different parts of the brain, even bilaterally, to where they're
actually processed in the non-primary auditory cortex in the superior temporal
gyrus. What we know about this area from animal studies and non-human primates, for
example, is this is an area that no longer is tuned to basic low-level sound
features, like pure tones, pure frequencies, but in one that is actually
tuned to very broad, complex sounds. There have been very nice work in fMRI that's
actually demonstrated that this area is far more selective to complex sounds, like
speech, over non-speech sounds. So the basic question is not really about where
is this processing going on. The question is how. Okay. What is the structure of
information in this transformation that's going on? In particular, what kind of
linkages can we make between that physical stimulus and the internal one, which is a
phonetic one? For me, I think it's really important to acknowledge some really
important fundamental contributions that occur, that give us some insight and put
them in a very important perspective. For me, I think one of the most important
pieces of work that led to this work that I'm about to describe from our lab was
actually 25 years ago, using an approach that's actually far more complicated and
difficult to achieve than what we do in our own work.
This is using single unit and single neuron recordings that were recorded from
patients that were undergoing neurosurgical procedures for their
clinical routine care. This very extremely rare but precious opportunity to actually
record from certain brain areas while someone is actually recording-- listening
to speech. These are from my close colleague and mentor, Dr. Ojamin, who's in
the audience today. But why I think it's so important to acknowledge this work is a
lot of the clues about what I'm about to describe were actually seen 25 years ago.
This figure that's extracted from that paper, where they actually showed and
could record from single brain cells, called neurons, in the superior temporal
gyrus, that they were active and corresponding to very specific sounds. But
if you actually look at where those sounds are, they're not exactly corresponding to
the same exact, let's say, phonemes or the same exact sounds, but, in fact, they are
corresponding to a class of sounds. This was an observation that was made in this
paper. They thought perhaps this is some mention of phonetic category
representation there. But it wasn't that clear, actually, and there were a lot of
other really important observations that were made in that paper. Now from a
linguistic perspective, in thinking about how, behaviorally, we organize this
information in the brain, there's actually a wonderful way to approach it. It's not
perfect, but a very wonderful way to think about how languages across the world
actually share a similar and shared inventory of speech sounds, not all
completely the same. Each language has a different number, but they highly overlap.
The reason why they overlap is because they are produced by the same vocal tract.
This is essentially like a periodic table of sound elements for human language and
speech. So this table actually has two really important dimensions. The
horizontal dimension is actually one that we call the place of articulation. It's
referencing where in the vocal track these sounds are made. For example, bilabial
sounds, the 'P' and the 'B' require you to actually have a transient occlusion at the
lips, 'ba. ' You cannot make those sounds without that particular articulatory
movement. Whereas some of the other sounds, like a 'D' or a 'T', a 'da' a 'ta'
a 'da' we call alveolar because the front of the tongue tip is actually placed
against the teeth. So these are actually referencing where occlusions are occurring
in the vocal tract when we speak, and those actually correlate necessarily to
very specific acoustic signatures. The other dimension is what we call the manner
of articulation. So the manner of articulation is actually telling you a
little bit more about not so much where, but how in the vocal tract the
constrictions are made in order to produce those sounds. We have certain ones, like
plosives, where you have complete closure of the vocal tract and then a transient
release, other sounds where you have near-complete, like a fricative, like
'sha,' 'za,' those sounds that we call fricative.
If you actually look at vowels, they actually have a similar structural
organization. There actually is something that actually references where in the
vocal tract, either the front, middle, or back, or the degree of open and closure.
So for both consonants and vowels, there actually is a structure that we know
about, linguistically and phonologically, about how these things are organized. I
think the thing that interests me is that, like I referenced before, that this is
something like a periodic table. There is something fundamental about these units to
our ability to perceive speech. These phonological representations are not
necessarily the ones that we think of as these letters that we call phonemes, but
actually groups of phonemes that share something in common, what we call
features. These are the members of small categories which combine to form the
speech sounds of human language. This became very attractive to me as a model of
something to look for in the brain because of . . . Essentially why it could be so
important is that languages actually do not vary without limit, but they actually
reflect some single or limited general pattern, which is actually rooted in both
the physical and cognitive capacities of the human brain, and I would add the vocal
tract. This is not a new kind of thinking, but it's one that has not been clearly
elucidated in terms of its biological mechanisms. So in order for us to get this
information, it requires a very special opportunity, the one where we can't
actually record directly from the brain. In many ways, this is actually a lot more
coarse than the kind of recordings that were done almost 25 years ago. These are
ones from electrode sensors that are placed on the brain in order to localize
seizures in patients that have epilepsy. In the seven to 10 days that they are
usually waiting to be localized, we have a very, again, precious opportunity to
actually have some of the participant, the patient volunteers, listen to natural,
continuous speech and look at those neural responses on these electrode recordings to
see how information is distributed in the superior temporal gyrus when they're
listening to these sounds. This gives you a sense of actually what that neural
activity pattern looks like. [audio sample]
- We're going to slow down that sentence a lot here.
- [audio sample] Ready tiger go to green five now.
- So you can see that the information is being processed in a very precise, both
spacial and temporal, manner in the brain. This is exactly the reason why this kind
of information has been elusive, because we do not currently have a method that
actually has both spacial and temporal resolution and, at the same time, covers
all of these areas simultaneously. So it's, again, in the context of these rare
opportunities with human patient volunteers that we can conduct this kind
of research. So the natural question is . . . Of course, now that I've shown you
that we can actually see a pattern in the brain, both that's temporally and
spatially specific, what actually happens when we try to deconstruct some of those
sound patterns from the brain? This just gives you an example, again, in the
superior temporal gyrus, where those sounds are activating the brain. An
example of the spectrogram for a given sentence, in this case, it's, "In what
eyes there were." The last part of that figure basically shows you that pattern
across different electrodes. It's not all happening in the same particular way. You
have very specific evoked responses that actually occur at different parts of the
superior temporal gyrus. I want to show you what happens when you
look at just one of those electrodes. If you look at the neural response of that
one particular electrode that's labeled e1, and you organize the neural response
by different phonemes, okay, you can actually see, again, on the vertical
access, starting with 'da,' 'ba,' 'ga,' 'ta,' 'ka.' You can see that this
electrode . . . Those hundreds or thousands of neurons that are under this
electrode are very selectively responsive to this set of sounds that we call
plosives. It's not one phoneme, but a category, and they share this feature that
we actually know, linguistically, to be called plosive. I can show you a series of
other electrodes. Electrode two has a very different kind of sensitivity. It's
showing you that it really likes those sounds 'sha,' 'za,' 'sa,' 'fa.' This
is an electrode that is, again, not tuned to one phoneme, but actually tuned to the
category of sibilant fricatives in linguistic jargon. We have another
electrode, e3, that is selective to low-back vowels, these "ah" based ones.
Another one that is a little bit more selective to high-fronted vowels, 'E.'
Even another electrode, e5, that is corresponding to nasal sounds.
So this is a very low-level description, but it's actually the first time we've
ever seen in this kind of principled way, obtained through very precise spatial and
temporal recordings, the ability to resolve phonetic feature selectivity at
single electrodes in the human brain. Now this is not enough. We need to really
address this issue of structure. That's one of the themes here. Are all of these
things just equally distributed as features? In the original thinking about
these things, you could have a binary list of features. It turns out that features,
in and of themselves, actually have structure and have relationships with one
another. So what we did, in order to look at that structure in the brain, we looked
at hundreds of electrodes that were recorded over a dozen patients. Each one
of those columns actually corresponds to one electrode and one particular superior
temporal gyrus in someone's brain. Like I just showed you before, the vertical axis
is actually how they're organized by different phonemes. What we did here was
we used a statistical method called hierarchical clustering. What hierarchical
clustering is used for is finding the patterns in this data. What the
hierarchical clustering showed us and sorted this data was that, in fact, there
is, indeed, structure in the brain's responses to human speech sounds, and it
looks like this. So we've organized the hierarchical
clustering as a function of a single electrode's, again, a single column's
selectivity to different phonemes, but we've also organized this clustering as a
population response across all of the electrodes and looking at that selectivity
for different phonemes. So we have two different axes that we're actually looking
at the brains large distributed response to speech sounds. We're using this method
which is what we call unsupervised, meaning we're not telling it any
linguistic information, or we're not organizing the data. We're just saying,
"Tell us how the brain is organizing this information." What we see from this is
that when we actually look at where this information is being organized, one of the
biggest divisions between different parts of different kinds of selectivity in the
brain are what we would call the difference between consonants and vowels
or really, actually, between obstruents and continuants, in linguistic jargon. But
within those different categories, you actually have sub-classification. So
within the consonants, you actually have a subdivision between plosives and
fricatives. Between the sonorants you actually have referencing for different
positions of the tongue, low back, low front, high front, different classes of
vowels and, in fact, nasal. So basically this is telling you that feature
selectivity in the brain is actually hierarchically structured. The second
thing is that instead of using phonemes in order to organize the responses, we
actually use features. So as an example, that term dorsal actually refers to the
tongue position when it's fairly back, like for 'G' 'K' sounds.
You can see that when we organize things by features, you have a much cleaner
delineation. The electrode responses seem to be much more tuned to phonetic features
shown below, as they are, compared to when you plot them as phonemes. Okay. So this
essentially disproves any idea that there is individual phoneme representation in
the brain, at least not one that's locally encoded, but tells you that the brain is
organized by its sensitivity to phonetic features. Now relating it to a phonetic
feature is the first step, and it's one that's really important because it's
referencing the one we know about from linguistics and the one that we know
behaviorally. But how do we connect this to the physical stimulus that's actually
coming through our ears? That's where we have to make a linkage to actually
something about the sound properties. Are these things truly abstract features that
are being picked up by the brain? Or actually, are they referencing specific
sound properties? Basically the answer is the latter. It's that what we're actually
seeing is sensitivity to particular spectral temporal features. In the top
row, I am showing you basically . . . When we look at the average tuning curves, the
frequency versus time, tuning curves for each one of those different
classifications for plosive fricatives, they're very similar to the acoustic
structure when you average those particular phonemes in the brain.
So what this means is the tuning that we're seeing that's corresponding to
phonetic features is, in fact, one that is tuned to high order acoustic spectral
temporal ones. The brain is selecting specific kind of acoustic information and
converting it into what we perceive as phonetic. In the interest of time, I'm
going to sort of skip more in-depth information about vowels and plosives and
how those are specifically encoded. But in summary, what we've found is that there's
actually a multidimensional feature space, actually, for speech sounds in the human
superior temporal gyrus. This feature space is organized in a way that actually
shows hierarchical structure. The hierarchical structure is fairly strongly
driven by the brain, in particular, this auditory cortex sensitivity to acoustic
differences, which are most signified actually in the manner of articulation
distinctions, linguistically. What's interesting about this is it actually does
correlate quite well with some known perceptual behavior. So I would like to
conclude there and acknowledge some of the really important people from my lab,
postdoctoral fellow Nima Mesgarani who did most of this work with one of our graduate
students, Connie Cheung. Thank you.
[applause]
♪ [music] ♪