Excerpted from Experience on Demand: What Virtual Reality Is, How It Works, and What It Can Do by Jeremy Bailenson. Out this month from W.W. Norton & Company, Inc.
In his 1984 cyberpunk thriller Neuromancer—a canonical literary journey in virtual reality—William Gibson introduced the terms “cyberspace” and the “matrix.” The words are familiar now, but the way he discusses them still feels new: He calls them “a consensual hallucination.” What Gibson suggests is that it won’t be the graphics or photorealistic avatars that will make virtual worlds feel real—it will be the community of people interacting within them, bringing the world alive through their mutual acknowledgment of its reality.
People often ask me what the “killer app” of VR will be. What is going to drive mass adoption of this expensive and admittedly awkward technological apparatus? I tell them it’s not going to be trips to space, courtside seats at sporting events, VR films, cool video games, or underwater whale-watching. At least, it’s not going to be those things if you have to do them alone, or if, when you do share them with other people, you are excessively constrained in your ability to interact.
VR developers so often stress the reality-bending properties of virtual reality, focusing on how the technology will allow individuals to do impossible things in virtual settings. But virtual reality is going to become a must-have technology when you can simply talk and interact with other people in a virtual space in a way that feels utterly, unspectacularly normal.
It remains one of the most difficult problems in VR—how can you put two or more users in a virtual space and allow them to interact in a human way, with each other and with their virtual environment? How do you capture and convey the subtleties of human social interaction? To understand how to make our virtual avatars feel real, we have to know what we humans are doing—consciously and unconsciously—that makes our daily encounters in real life feel real. That, philosophers and psychologists will agree, is a complicated question. But given the world-altering benefits a truly effective virtual social network could have—allowing people separated by great distances to socialize, collaborate, and conduct business in new ways, all while limiting the toll burning gasoline and jet fuel takes on the environment—it’s crucially important that those of us working on this problem get it right, and quickly.
While it’s true that conferencing over phones or video applications has become more common in businesses, the stubborn fact remains that most managers still want bodies at meetings. Sales reps are still expected to travel from city to city to meet with customers. Consultants are still required to visit corporate headquarters and meet their clients in person. Academics still need to travel to conferences to present papers. When it gets down to brass tacks, the most important encounters still need to be done face-to-face, often over a meal or drinks.
“Physical contact is important,” a jet-setting businessman, Shaun Bagai, told me when I asked him why he frequently flies to Asia for one-day business trips. “You can’t put your hand on someone’s shoulder, look them in the eyes, and close a deal over a telephone or even a video conference.” Physical contact is a major part of how he communicates. He may be able to convey his impressive knowledge, keen ability to present scientific data, and forward-thinking vision over a monitor, but charisma and trustworthiness are still best shared in person.
If virtual travel and telepresence are going to replace physical travel, there will have to be a system that allows a virtual version of Shaun Bagai to be just as warm and charismatic as his physical self. To do that, the people who are creating avatars and virtual worlds will need to understand and to some degree replicate in virtual bodies the complex choreography of body language, eye movement, facial expressions, hand gestures, and physical touch that occur in real-life social interactions.
The notion of the handshake, and its virtual future, is one I think about often. I’ve been working on some aspects of this social VR question for years, and in June 2010 I visited Philips Corp., based in Holland. Philips is one of the largest electronics companies and its technology has been a fixture around the globe for more than a century. Philips funded research in my lab to produce virtual encounters that were as compelling as—or even more compelling than—face-to-face ones, including encounters that convey intimacy through physical touch.
In addition to me, Philips had reached out to another one of the handful of scholars who had rigorously investigated the psychological implications of sending touch over the Internet: my colleague and friend Wijnand (pronounced Vi-Nan) IJsselsteijn, who is a professor at the Eindhoven University of Technology. Wijnand is an expert at building haptic devices that can measure touch on one end using sensors that detect body movement, and can create the sensation on the other end using electric current, vibration, motors, and air puffs. He is also an expert at testing to see if they are effective at eliciting psychological intimacy.
Touch can be a powerful dimension of social life. Studies show that waitstaff who touch customers on the shoulder can influence them to order more drinks and also get bigger tips than ones who don’t. (But only if they’re not being flirtatious and definitely not if it inspires jealousy from another customer at the table.) This “Midas Touch” effect is well documented in psychological literature that goes back to the 1970s. Wijnand and I wondered if virtual touch worked the same way. This effect is a particular interest of Wijnand’s, who has worked for almost a decade to answer this question. He has spent so much time working in this area, in fact, that he was inspired to name his youngest son Midas.
In the first study to look at the Midas Touch in virtual space, Wijnand had two people chat online using instant messaging. One was the subject, while the other was a “confederate,” a person who is part of the experiment team but pretends to be another participant in the study. During the interaction, experimental subjects wore a sleeve on their arm that used six points of vibration contact to simulate being tapped on the arm. Some subjects received the virtual touch from the confederate, who could activate the vibrating sleeve on a networked computer, and others did not. After the experiment was over, the confederate got up from the computer station in front of the subject and dropped 18 coins on the ground. Wijnand measured whether subjects helped pick up the coins, with the prediction that those who had been touched virtually would be more helpful than those who had not. He demonstrated that a majority of those who had been touched helped, while those who did not get touched helped less than 50 percent of the time. In other words, the Midas Touch worked virtually.
Wijnand’s work is important replication work. In other words, it shows that virtual touch can bring about the same benefits that physical touch can bring by replicating a finding that is fairly accepted by scholars worldwide. The work conducted in my lab took a different approach by examining some benefits of touch that can only occur in the virtual world. We looked at mimicry. It’s long been known that nonverbal mimicry causes influence—simply by
subtly matching another person’s gestures, you can get him to like you. Tanya Chartrand, while a professor at New York University, was perhaps the first to provide rigorous data on this “chameleon effect.” She had people go into job interviews and mimic the nonverbal gestures, such as leg-crossing, of the interviewers. People who mimicked the interviewer were more likely to get favorable evaluations than those who did not, even though the interviewer had no conscious idea she was being mimicked.
Have you ever shaken your own hand? Mimicry via touch might also provide influence. To test this, we bought a rubber hand from a Halloween costume store and attached it to the stick of a “force feedback joystick” that uses motors to play back movements that are recorded by another, similar joystick. We actually built a virtual handshake by having one person hold the rubber hand and move the joystick, and having another person feel the movement on a similar handshake machine. Two people met face-to-face in the lab but never touched physically. We then had them virtually shake hands using our newly crafted machine. But one person never actually felt the other person’s shake. Instead, he received his own handshake that we had previously recorded, and was told it was the other person’s. In other words, he shook his own hand.
People who received their own shake liked their partners better—they treated those “digital chameleons” more softly in a negotiation task and also rated them as more likable—than those who received the other person’s actual movements. People had no idea they had been mimicked, but the subtle effect of familiar touch was a winner.
Imagine giving a talk to 1,000 people, trying to convince them of some ridiculous notion—say, for example, why they should travel virtually instead of physically. I do this from time to time. Perhaps, if I could deliver a personal Midas Touch to all the people in the audience, they would be more receptive to my message. Unfortunately, it would take me hours to shake every single person’s hand physically. But if each person had the application that Philips Electronics is devising, which uses the accelerometer and vibration motors in today’s smartphone to simulate touch, then I could shake all of their hands at once, leveraging Wijnand’s experimental findings. But, even better than just tapping them on the shoulder, I could send different versions of my handshakes to all of them simultaneously, scaling up my ability to mimic. It would be a politician’s dream.
Of course, there are subtle insidious aspects of this kind of transformed social interaction. To put it bluntly: We tend to prefer those who look, sound, and behave like we do. And, in a prospective world of communication mediated by virtual avatars, we should be prepared for many more manipulations like the one we engineered. It’s the logical extension of what we already do in social encounters, where we change our clothing, our speech, our body language, and other aspects of our self-presentation depending on the situation we are in. We dress and act much differently in a job interview than we do when we are hanging out with our friends in a nightclub.
We will also need to keep an eye on how people behave in virtual environments. The past few decades have seen the gradual dismantling of the utopian hopes of early internet users that online social spaces would develop as a kind of information-rich digital agora, where ideas would be freely exchanged and intelligent conversation would flow. However, the anonymity that was supposed to protect people’s speech and identity has also made possible a powerful subculture of internet trolls who take delight in making other people miserable. If VR innovators are right, and public debate shifts to virtual spaces, we can only guess as to what these encounters will be like. Avatar bodies cannot hurt you, but their perceived physicality does make a greater impact than a comment on an article or a Tweet, and indeed there have already been allegations of sexual harassment and avatar trolling in some early virtual environments.
But I hold out an optimistic hope that good social VR might actually improve things. It is much easier to dissociate a person from her basic humanity and the respect she deserves when you are perceiving her through short text messages. If we start seeing people online again as humans and are able to achieve, in some new way, the elements of genuine social connection that help bind us to other people, it might improve the dialogue online and open up a more productive and civil public space.
Excerpted from Experience on Demand: What Virtual Reality Is, How It Works, and What It Can Do by Jeremy Bailenson. Copyright © 2018 by Jeremy Bailenson. Used with permission of the publisher, W.W. Norton & Company, Inc. All rights reserved.
