Two Jews and an Englishman are crossing the ocean on a ship. The Jews, who can’t swim, start arguing with each other about what they should do if it sinks. As they argue, they gesticulate with such vigor that the Englishman backs away to avoid injury. Suddenly, the boat begins to sink. All the passengers except for the Jews, who are too wrapped up in their argument to notice, jump overboard. After a long, exhausting swim, the Englishman finally reaches the shore. He is amazed to find the two Jews there, happily waving him in. Astonished, he asks them how they got there. “We have no idea,” says one of them. “We just kept on talking in the water.”

A version of this joke appears in a 1941 dissertation on “the gestural behavior of eastern Jews and southern Italians in New York City, living under similar as well as different environmental conditions.” The study was written by David Efron, who grew up in an orthodox Jewish home in Argentina and arrived in New York for graduate study in the 1930s. By his own account, when he spoke Spanish, he gestured with “the effervescence and fluidity of those of a good many Argentinians.” When he spoke Yiddish, his gestures were more “tense, jerky, and confined.” He sometimes combined the two styles, as when “discussing a Jewish matter in Spanish, and vice versa.” After living in the United States for a few years, he found his gestures becoming “in general less expansive, even when speaking in his native tongue.” His gestural identity was further complicated by the “symbolic Italian movements” he had picked up from Argentine-Italians and reinforced on a trip through Italy. But no matter what language he spoke, he proved to be “an adroit table-pounder.”

Efron was one of the last students of the famous anthropologist Franz Boas. Boas spent his career arguing that it was culture and environment, not biological race, that accounted for differences in how groups of people behaved. Efron’s study was designed as a challenge to the impressionistic explanations of gesture that the race theorists of the 1930s were passing off as science. One claimed that Jews of mixed race who no longer had other Jewish physical traits could still be identified by their gestures. Another categorized gesture by race: Nordic gestures were restrained; Mediterranean gestures were playful; the gestures of the Phalic race (as in the German region of Westphalia) reminded one of a fleeing chicken; Italian gestures were explained with reference to hot blood, light bones, and poor impulse control.

Efron observed the conversations of 1,250 Lithuanian and Polish Jews and 1,100 Italians from Naples and Sicily in and around New York City. In each group, about half were recent immigrants and half were “assimilated.” They were observed in a range of settings: parks, markets, social clubs, schools, universities, Catskills resorts, Adirondack hotels, and the Saratoga racetrack. He recorded five thousand feet of film and, with an artist, produced two thousand sketches of spontaneous gestures.

The results paint a picture of a stereotype, but a lovingly detailed and specific one. According to Efron, Jews used a limited range of motion, mostly from the elbow. Their movements were more angular, jabbing, intricate, and vertical than those of the Italians, who used larger, smoother, more curved lateral gestures which pivoted from the shoulder. Jews tended to use one hand, Italians both. Italians touched their own bodies, Jews touched the bodies of their conversational partners. Efron describes with delight an episode he witnessed where one man grabbed the arm of his interlocutor and started gesturing with it. That man, becoming annoyed, finally grabbed the first man’s wrist in retaliation and “started admonishing him back with his own…hand.” Jews also did more gesturing with objects such as pencils or, in one case, a meatball on the end of a fork. Italians used less finger and wrist movement but more repetition. They also had a vocabulary of symbolic gestures with standard meanings—from “I know more than you think I do” to “I’ll sew your lips together” to “I’ll poke your eyes out”—that could be understood without any speech at all. 

The main result, the one the study was designed to find, was that as the Jews and Italians assimilated, they began to gesture alike. When Efron tested a group of students at a high school in Little Italy on the meanings of the symbolic gestures used by the unassimilated Italians, less than half of their judgments were correct. He came to the anticipated conclusion: as Jews and Italians became American, so did their gestures. 

The conclusion was unsurprising; it was Efron’s method that made his study important. In order to make his study empirical, Efron had to develop a way to break gestures down into countable units so that he could explain differences with respect to those units. There were “emblems” that could be understood without speech, those of the Italian “I’ll poke your eyes out” variety. There were also gestures that had no meaning independent from speech: “physiographics” and “kinetographics” that trace out the objects or actions under discussion, “ideographics” that trace out the metaphorical pathways of the speaker’s thoughts, and “batons” that beat out the rhythm of speech. 

The gestures of the subjects that Efron observed didn’t differ only by the qualities of how they moved or how many hands they used or who they touched. They also seemed functionally different. Italians used emblems; Jews didn’t. Italians sometimes used physiographics, depicting the size and shape of the things they talked about; Jews used ideographics, depicting features of the discourse itself. When Jews pointed a thumb toward the ground and then scooped it upward quickly, they were highlighting the crux of the discourse, physically and metaphorically digging it out for consideration. When they traced an angular zigzag with a finger, they were outlining the back and forth of an argument, linking one salient bit to the next. 

After he published his study on the gestures of New York immigrants, Efron left academia for a career advocating for workers’ rights at the UN’s International Labor Organization. But his dissertation went on to become a foundation for the field of “gesture studies”—a label applied to the activities of various psychologists, anthropologists, and linguists who look at the things people do with their hands while they speak. Efron not only laid the groundwork for a more systematic method of studying gestures, he introduced the idea that gesture was not a companion to speaking, but a product of it. 

In his Institutes of Oratory, the first-century rhetorician Quintilian says the hands “almost equal in expression the powers of language itself,” and he praises them for all the things they can do:

With our hands we ask, promise, call persons to us and send them away, threaten, supplicate, intimate dislike or fear; with our hands we signify joy, grief, doubt, acknowledgment, penitence, and indicate measure, quantity, number, and time. Have not our hands the power of inciting, of restraining, of beseeching, of testifying approbation, admiration, and shame? Do they not, in pointing out places and persons, discharge the duty of adverbs and pronouns? So that, amidst the great diversity of tongues pervading all nations and people, the language of the hands appears to be a language common to all men.

Quintilian implies, like many after him, that gesture is some kind of universal natural language. It was a language, however, that needed to be cultivated and practiced. Since gesture represented thoughts, or, as Cicero said, “the motions of the soul,” orators had to learn how to marshal their gestures to put their thoughts in the best light. Quintilian laid out specific dos and don’ts. He tells us, for example, that when “the middle finger is drawn in toward the thumb, the other three fingers being open,” this is an appropriate gesture to use during the introduction to a speech, provided it is “moderately exerted and with a gentle movement of the hand in either direction.” It can add confirmation when stating the facts if the movement is “somewhat more decided,” but “in invective and refutation, it must be spirited and impressive.” It should never, however, be aimed sideways so that the middle finger points toward the left shoulder. 

For centuries, discussion of gesture was couched in terms of what was proper or effective. There were guides for orators, preachers, and actors, and rulebooks for courtly behavior that laid down standards for gesture. In the seventeenth century there were even dictionaries of gesture: Giovanni Bonifacio’s The Art of Signs (1616) and John Bulwer’s Chirologia and Chironomia (1644) list hundreds of gestures, citing passages from the classics on their meanings. According to Bulwer, we know that “to smite suddenly on the left hand with the right” signifies anger because Seneca used it in a description of an angry man.

Most guides to gesture advised against mere mimicry or acting out the content of the speech they accompanied. Quintilian believed the gestures of an orator “should be suited rather to his sense than to his words.” The purpose of gesture was not to repeat information, but to add it. Indeed this is how even those untrained in oratorical gesture seem to use it. We use gestures to show how the events we narrate happened, and to point to the particular things or people we talk about. The gestures of the Italians Efron studied added information about the physical qualities of the things they talked about as well as their attitudes toward them. The gestures of the Jews he studied illustrated the connections they were making between ideas and their relative importance. Gesture can communicate a layer of meaning missing from the speech.

But it would be wrong to say that the reason we gesture is to communicate. Almost anything can communicate—the clothes you wear, the flowers you send, the way you flutter your fan or fold your handkerchief. Gestures communicate too, but they are much more intimately tied to the act of speaking. They are not a language in themselves, but they are a complement to language, a partner with language, a byproduct of language. Subsequent research in the field that Efron founded has failed to find a culture that does not gesture during speech. Not everyone does it as colorfully as the Italians and Jews, but everyone does it, even Englishmen. While aspects of the way we do it are learned or culturally conditioned, and while some of our gestures are intentionally formed with the goal of communication in mind, imitation can’t explain why congenitally blind people gesture, especially when they know they’re speaking to other blind people, and communicative intent can’t explain why people gesture when they’re on the phone. Gesture is simply a part of language use. When we form our thoughts into speech, some of it leaks through our hands.

The sense that gesture is a language of its own is even more pronounced in those cases where it seems to replace speaking entirely. In the nineteenth century, visitors came back from Italy with news of an exotic “gesture language” that was spoken without words at all. After the discovery of the archaeological sites of Herculaneum and Pompeii in the eighteenth century, Naples had become a can’t-miss stop on the grand tour. Letters home and travelogues in magazines told tales of complete conversations silently conducted between balconies, gossip and treachery performed by hand alone, and love affairs arranged without a word spoken. In one apocryphal anecdote, a young swain woos his beauty over the course of months, without discovery by her father, through gestures and looks exchanged from street to balcony. When he finally arrives at the decided meeting place to run away with her, he hears in the darkness an abrasive squawk asking, “are you there?” Realizing it’s the voice of his love, which he has never heard before, he runs the other way. 

For the benefit of these foreigners “who had been born in distant regions and who, on account of their cool and sluggish temperament are rather unsuited to gesturing,” Andrea de Jorio, an archaeologist at the Royal Borbonic Museum in Naples, produced one of the only works before Efron’s to look at gestures as they were used rather than as they ought to be used, an 1832 study that catalogued hundreds of gestures used in the streets of Naples.

De Jorio provides an alphabetized index of gesture meanings for everything from “abbondanza” (abundance) to “uomo panciuto” (paunchy man). He not only describes what the gestures look like—the tips of the index finger and thumb joined together facing one another, and then separated by the index finder of the other hand means “I am not friends with you anymore”—he gives little scenarios of the gestures used in context, showing some of the varying shades of meaning they can acquire. In one example, he tells the story of “a certain count, noting that someone he did not know had joined the conversation, and who made a somewhat bad impression, asked his friends, in gesture, who this person was.” The first man responded by placing the outside of his thumb at his ear, with the palm facing downward, “thus declaring him to be an ass.” The second friend made the same gesture, but with both hands at his ears, “meaning the fellow was more than an ass.” The third friend placed the tips of his extended thumbs on his temples with the other fingers wide open and oscillating, confirming that the poor fellow was “not just a fool, he was positively asinine.”

Despite stereotypes, the Italians have never had a monopoly on the wordless gesture. Even the most sluggish-armed among us can get all kinds of messages across without saying a word: “come here,” “he’s crazy,” “check her out,” “yes,” “no,” “I don’t know,” “peace,” “it’s a secret,” “I’m thinking,” “wait a minute,” “stop right there,” “something stinks,” “I’m not listening,” “screw you,” “check, please.” These gestures aren’t exotic to us because they’re the ones we use. They seem somehow to belong to the language “common to all men” that Quintilian was talking about.

But of course, they aren’t common to all men, as anyone who’s ever looked at a travel guidebook can tell you. Remember to avoid the “okay” sign in Brazil, where it means “asshole.” Watch out in Bulgaria, where a head nod means “no” and a head shake means “yes.” Don’t give the thumbs up in Iran unless you mean to say “up yours.” Many of the gestures we use in place of speech aren’t transparent at all. Their forms are arbitrary and need to be translated just like words.

It is in this silent use of gesture, where the gestures become like words—quotable, conventionally defined, intentionally produced, and meant to communicate—that gesture really does start to look like a language. But looked at more closely, these gestures distinguish themselves from words in interesting ways.

For one, they can be remarkably durable over time. The gestures we inherited from the Greeks and Romans are much more immediately identifiable today than are the words we inherited from them. The digitus impudicus that Romans insulted each other with is the same digit we use for that purpose today (and while the phrase “digitus impudicus” takes some education and background to decipher, a display of the “impudent finger” does not). 

Also, these quotable gestures—emblems, as Efron called them—function quite differently from words. They almost never play the role of nouns or verbs. There are gestures that seem like adjectives—the finger twirling at the temple for “crazy,” the fingertip kiss for “delicious”—however, they act not as descriptors, but as attitude-laden comments. Emblems don’t work like words so much as complete speech acts. They don’t say, they do. They request (come here!), admonish (shhh!) insult (up yours!), promise (cross my heart!), and compliment (delicious!).

Only in the case of full sign languages of the deaf do gestures take on all the properties of words. Sign languages have nouns and verbs and rules for how they fit into sentences. Signs can say, “rosemary really brings out the flavor in this roast” as well as “delicious!” Signs, like words, are composed out of a finite inventory of units that are defined with discrete boundaries. In American Sign Language, the position of the thumb in a fist—whether it lies next to the fingers, in front of the fingers, or inside the fingers—can make the difference between one meaning, another meaning, and nonsense, in the way that in speech, tiny alternations in vibration and airflow can make the difference between “pine” and “mine.” Gestures are wholes. Their internal parts aren’t important. When I punch my hand with a fist to tell you that I’m going to beat you up, it hardly matters what my thumb is doing. 

The greater the burden of communication gestures have to carry, the more languagelike they become. But if we already have a full language to communicate, then why do we gesture? Clearly it’s useful for cases where we can’t or don’t want to speak. With gestures, baseball players exchange secrets on the open field, stock traders make deals in the noisy hubbub of the pit, scuba divers communicate through the barrier of water, and drivers make their frustration known to other drivers through the barrier of car windows.

These special cases don’t represent the bulk of gesturing we do. Most of our gestures happen while we can speak or are speaking. But the act of using language is ephemeral; words disappear as they are spoken. Of course, we’ve had the ability to preserve the words of the past ever since the invention of writing. But the solid, linear permanence of written language encourages the illusion that language is just an object, a container for thought. In fact, language is also a behavior, a laboratory for thought creation and negotiation. Gestures are thoughts, ideas, speech acts made tangible in the air. They can even, for a moment, outlive the speaker. Death-row inmates have been executed with their middle fingers extended in a final gesture of defiance.

David McNeill, a psychologist who has spent his career studying gesture, first took notice of it watching two of his colleagues converse. They looked to him like “sculptors working in different media. One was always pounding and pushing some heavy blocklike stuff. I imagined that his medium was clay or marble. The other was drawing out and weaving some incredibly delicate, spidery stuff. His medium looked like strings or spiderwebs.” Research of the past few decades has shown that putting our thoughts in our hands can help us learn and remember better, can help us speak more fluently and find the right words.

When we speak, we shape our thoughts for language, and when we gesture, we shape them in the space in front of us. We may be different kinds of sculptors using different kinds of media, but our molding, weaving, and chiseling does us good.