Pinpointing Agency and Realism in the Brain During Electrical Muscle Stimulation

If there’s one familiar sound whenever a volunteer tries out an interactive device that uses electrical muscle stimulation (EMS), it is probably laughter. Even for experienced users of the technology, the sensation of a machine controlling your body, rather than the other way around, feels unnatural and strange. Something about the experience disrupts people’s sense of agency — the feeling of being in control of one’s actions — which could interfere with the potential of EMS to improve learning, make virtual reality more realistic, and many other applications.

As Pedro Lopes explored EMS devices through a human-computer interface lens, first in his doctoral work at the Hasso Plattner Institute in Germany and ongoing today at UChicago CS, he grew interested in whether agency can be measured, controlled, or even restored during EMS use. In two recent papers at the international SIGCHI conference in Glasgow and a new publication in the neuroscience journal Cerebral Cortex, Lopes and collaborators have been on the trail of agency, using everything from pitching machines to fMRI brain scanners.

“We started by asking the question, does EMS always have to feel that unnatural, or is there anything we can do to make it feel more in tune with your own volition?,” Lopes said. “I think we just started to answer it, but there are infinite ways to look at this thing because it's such a philosophical question.”

The agency pursuit started with a simple demo performed at the SIGGRAPH 2018 conference by Jun Nishida, who's now a postdoctoral researcher at UChicago CS: can EMS help people catch a marker dropped by another person at short range? It did, and as expected, most volunteers attributed the action to the machine, not their own reflexes. But a small minority of participants disagreed, saying that the EMS must not have been on because they caught the marker unassisted. What happened?

Those outliers inspired an experiment where Lopes, Nishida, and Shunichi Kasahara of Sony Computer Science Laboratories adjust the timing of when EMS triggered a user to perform a particular task, such as hitting a button in response to a stimulus, or more advanced actions such as photographing a fast-moving baseball. They found that there’s a window of time wherein it becomes very hard for users to distinguish between their own actions and those generated by EMS (volunteers can try the baseball photography themselves at this year’s SIGGRAPH).

“If we do it very early on it will feel very artificial, because you do something completely superhuman: you hit a ball much faster or you grab the pen super early,” Lopes said. “But if we would do the stimulation just a little bit before you would normally do it yourself, you might not even spot that this was the muscle stimulation and not you. You feel like you did it.”

That experiment utilized self-reporting — subjects were asked verbally after the task whether they felt like they did it themselves or were helped by the EMS device. In his lab’s second CHI paper, Lopes went deeper, hearing directly from participants’ brains whether they felt agency by measuring its electrical activity during a virtual reality task. 

In a VR environment, users were asked to touch a virtual box. Precisely when their fingers contacted the virtual box they received either visual feedback, or visuals accompanied by vibrations on their fingertips, or all of the above accompanied also by muscle stimulation. But in a certain subset of trials, that feedback was “wrong,” occurring before the user appeared to touch the box. When this mismatch occurred, researchers saw a notable change in the brain’s electrical activity — a neural signal that the VR experience was less realistic.

“Your brain has a pretty good detector for whether you feel like you're there,” Lopes said. “It's kind of dynamically measuring how you feel in those virtual worlds, even though you don't have to report it.”

That project dovetailed with a concurrent collaboration between Lopes and neuroscientists at University College London & FU Berlin, who wanted to use EMS and fMRI to search for brain areas that distinguish between self-generated and externally-generated motion. The question reflects the selective nature of touch, where some sensations — such as from your fingers while searching for a coin in your pocket — can be extremely sensitive, while others — the motion of your arms to and from the pocket — are virtually imperceptible. 

Lopes helped design a system where participants in an fMRI scanner were asked to flex their middle finger with or without the help of EMS, receiving a tactile stimulus that simulates touching a hard surface on half the trials. This setup allowed researchers to tease apart the motor and sensory systems of the brain, and search for the brain area that potentially tunes these responses. The data pointed to a region called the posterior insular cortex (PIC), which responded strongly to touch during self-generated motion, but not during EMS stimulation.

“It suggests there's actually a mechanism associated to touch that is checking for agency and regulating these inputs, which could explain why we can so quickly change between one thing to another, if it's top-down,” Lopes said. “That little process seems to know what's going on.”

Zeroing in on this brain area and its electrical activity could guide engineers designing the next wave of EMS devices. Monitoring a user’s brain activity might help developers know, without asking, whether a VR experience is perceived as real, so they can fine-tune the software accordingly. More broadly, preserving a sense of agency in the user might make EMS more effective at teaching someone how to perform a physical action, such as improving a tennis stroke or playing an instrument.

“The next thing we're trying to see is, using these precise timings that help you be a little bit faster but still provide you with a lot of agency, does this influence the way you would learn with such a system?,” Lopes said. “It wouldn't dramatically improve you, because we know that if it's too much change from your expectation you're not going to believe it. These smaller changes might make you actually feel like you're improving and give you a lot of chances to get better.”