- Short report
- Open Access
Speaking and cognitive distractions during EEG-based brain control of a virtual neuroprosthesis-arm
Journal of NeuroEngineering and Rehabilitation volume 10, Article number: 116 (2013)
Brain-computer interface (BCI) systems have been developed to provide paralyzed individuals the ability to command the movements of an assistive device using only their brain activity. BCI systems are typically tested in a controlled laboratory environment were the user is focused solely on the brain-control task. However, for practical use in everyday life people must be able to use their brain-controlled device while mentally engaged with the cognitive responsibilities of daily activities and while compensating for any inherent dynamics of the device itself. BCIs that use electroencephalography (EEG) for movement control are often assumed to require significant mental effort, thus preventing users from thinking about anything else while using their BCI. This study tested the impact of cognitive load as well as speaking on the ability to use an EEG-based BCI.
Six participants controlled the two-dimensional (2D) movements of a simulated neuroprosthesis-arm under three different levels of cognitive distraction. The two higher cognitive load conditions also required simultaneously speaking during BCI use. On average, movement performance declined during higher levels of cognitive distraction, but only by a limited amount. Movement completion time increased by 7.2%, the percentage of targets successfully acquired declined by 11%, and path efficiency declined by 8.6%. Only the decline in percentage of targets acquired and path efficiency were statistically significant (p < 0.05).
People who have relatively good movement control of an EEG-based BCI may be able to speak and perform other cognitively engaging activities with only a minor drop in BCI-control performance.
Users of BCI-controlled devices, such as an upper-limb neuroprosthesis [1–4], must be able to use their device while talking and performing other cognitive tasks. Talking could potentially degrade EEG-controlled BCIs due to power spectral changes associated with verbal and cognitive engagement and the large electrical signals from muscles under the scalp. However, EEG-based BCI systems are usually evaluated with the subjects sitting quietly and focusing exclusively on the BCI task. This study tested the effects of speaking and cognitive load on the ability to command an upper-limb neuroprosthesis using EEG. All study activities were approved by the Institutional Review Board of the Louis Stokes Cleveland Veteran’s Affairs Medical Center.
Six able-bodied individuals with little or no prior BCI experience used either 16 or 32 channels of EEG to control a virtual neuroprosthesis arm with realistic dynamics in a 2D reach-and-hold task along a tabletop. Changes in mu and beta power associated with hand and feet movements were used to generate proportional 2D velocity commands in real time. Decoding functions (least-squares regression) and spatial filters (2D common spatial pattern) were generated from 5.3 minutes of actual, open-loop movements (methods detailed in ). Figure 1 illustrates how EEG modulations associated with different degrees of physically moving/resting the right hand or both feet were translated into a proportional 2D velocity command. This study included only participants with above-chance level EEG modulation associated with both hand and foot movements. Velocity commands were used to control the continuous motion of the fingertip of the virtual neuroprosthesis viewed on a computer screen as if looking down on the arm from above. Realistic dynamics of a paralyzed arm activated via electrical stimulation were added to the virtual neuroprosthesis using previously reported methods [6, 7].
Participants performed a ‘center-out-and-hold’ task where they had 12 seconds to move a cursor located at the fingertip of the virtual arm from the center of the workspace to one of four radial targets. Participants had to keep the fingertip/cursor touching the target for one second for the trial to count as a success.
To assess the effects of performing cognitive tasks and speaking on movement control, participants were tested under three levels of cognitive load (CL), two of which involved speaking. Random letters were played out loud every two seconds as participants performed the center-out-and-hold task. Participants were instructed to either: 1) only focus on controlling the virtual neuroprosthesis arm (‘No CL’), 2) repeat each letter immediately after hearing it (‘Moderate CL’), or 3) remember and repeat the previous letter immediately after hearing the current letter in a ‘one-back’ fashion (‘High CL’). The second and third conditions both required speech encoding and production but engaged working memory to different degrees . Subjects five and six were asked to speak each cued letter five times instead of once to generate more speech-related activity while maintaining the same cognitive load. Subjects five and six also only used 16 instead of 32 electrodes distributed over the same area due to connector issues.
Letters were played during the ‘No CL’ condition to ensure the auditory environment was the same across all tests. No speaking or cognitive tasks were performed during the initial open-loop data collection that was used to build the decoder. After one block of practice, each participant generated at least 40 center-out-and-hold movements under each of the three conditions. The order of test conditions was varied across subjects and each subject repeated their sequence of conditions twice to minimize learning and fatigue confounds.
Performance was assessed with three metrics: a) the percentage of trials that were successful, b) the average time required to successfully reach a target, and c) the average path efficiency (i.e. shortest path to the target divided by the length of the actual path taken).
Mean movement trajectories for each participant under each cognitive load condition are shown in Figure 2. On average, similar directional control was seen across all three cognitive load conditions.
Performance metrics for the BCI task are shown in Figure 3. On average, all metrics showed a decline in control performance with increased cognitive load, although this decline was not significant in the more variable movement-time metric (paired t-test).
This study sought to quantify the impact of additional cognitive load on an EEG-BCI where participants used different combinations of hand and foot movements to command the 2D velocity of the fingertip of a virtual neuroprosthesis. Our results demonstrated that the added cognitive load imposed by the letter repetition tasks produced only a moderate drop in control performance on average. However, given the added challenges of using different types of BCIs, it is important to further characterize how cognitive tasks and speech production impact BCI performance.
We have shown previously that strong jaw muscle activity associated with teeth clenching can be detected on EEG electrodes across much of the scalp . Therefore we anticipated that jaw muscle activity associated with talking might broadly increase the power in the recorded signals during the moderate-CL and high-CL tasks. Such an increase in power would have resulted in a command bias toward the left-distal direction because that is the direction associated with increased mu and beta power due to resting the hands and feet. However, the trajectories in Figure 2 did not show any systematic skewing toward the left-distal part of the workspace. Therefore, these results suggest the muscle activity required for simple speech production was not enough to cause a significant bias in directional control. However, additional studies are needed to determine if more animated speech, facial expressions, and additional cognitive burdens (e.g. mental calculations, emotions) may still disrupt BCI use.
The ability to use an EEG-BCI can vary widely between individuals . This study only included participants that naturally had movement-related modulation because it is likely that relatively good modulation will be needed for an individual to adopt a BCI into their daily life. People with more difficulty using a sensorimotor-rhythm-based BCI may have more problems performing cognitive tasks during BCI use. It is expected that these users could improve their control with additional training . However, further studies are needed that include individuals with paralysis and people who have improved their EEG modulation with training to fully assess the practicality of BCI technology for a broad range of users.
Peckham PH, Knutson JS: Functional electrical stimulation for neuromuscular applications. Ann Rev Biomed Eng. 2005, 7: 327-360. 10.1146/annurev.bioeng.6.040803.140103.
Chae J, Kilgore K, Triolo R, Creasey G: Functional neuromuscular stimulation in spinal cord injury. Phys Med Rehabil Cli. 2000, 11: 209-226. x
Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM: Brain-computer interfaces for communication and control. Clin Neurophysiol. 2002, 113: 767-791. 10.1016/S1388-2457(02)00057-3.
Collinger JL, Foldes ST, Bruns TM, Wodlinger B, Gaunt R, Weber DJ: Neuroprosthetic Technology for Individuals with Spinal Cord Injury. J Spinal Cord Med. 2013, 36 (4): 258-272. 10.1179/2045772313Y.0000000128.
Foldes ST, Taylor DM: Offline comparison of spatial filters for two-dimensional movement control with noninvasive field potentials. J Neural Eng. 2011, 8: 046022-10.1088/1741-2560/8/4/046022.
Chadwick EK, Blana D, Van den Bogert AJT, Kirsch RF: A real-time, 3-D musculoskeletal model for dynamic simulation of arm movements. Ieee T Bio-Med Eng. 2009, 56: 941-948.
Chadwick EK, Blana D, Simeral JD, Lambrecht J, Kim SP, Cornwell AS, Taylor DM, Hochberg LR, Donoghue JP, Kirsch RF: Continuous neuronal ensemble control of simulated arm reaching by a human with tetraplegia. J Neural Eng. 2011, 8: 034003-10.1088/1741-2560/8/3/034003.
Frankemolle AMM, Wu J, Noecker AM, Voelcker-Rehage C, Ho JC, Vitek JL, McIntyre CC, Alberts JL: Reversing cognitive-motor impairments in Parkinson’s disease patients using a computational modelling approach to deep brain stimulation programming. Brain. 2010, 133 (Pt 3): 746-761.
Foldes ST, Taylor DM: Discreet discrete commands for assistive and neuroprosthetic devices. Ieee T Neur Sys Reh. 2010, 18: 236-244.
Blankertz B, Sannelli C, Halder S, Hammer EM, Kübler A, Müller K-R, Curio G, Dickhaus T: Neurophysiological predictor of SMR-based BCI performance. Neuroimage. 2010, 51: 1303-1309. 10.1016/j.neuroimage.2010.03.022.
The project described was supported by Dept. of Veterans Affairs Merit Review #B4195R and Pre-Doctoral Rehabilitation Engineering Fellowship, National Institutes of Health NINDS F31-NS065710, NINDS R01-NS058871, NIBIB T32-EB004314, NINDS N01-NS-5-2365, and NICHD N01-HD-5-3403. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. We are very grateful to Holle Carey and Melissa Schmitt for their assistance in collecting this data.
The authors declare that they have no competing interests.
SF participated in the design of the study, collecting and analyzing data, and drafting the manuscript. DT participated in the conception and design of the study, collecting and analyzing data, and drafting the manuscript. Both authors read and approved the final manuscript.
About this article
Cite this article
Foldes, S.T., Taylor, D.M. Speaking and cognitive distractions during EEG-based brain control of a virtual neuroprosthesis-arm. J NeuroEngineering Rehabil 10, 116 (2013). https://doi.org/10.1186/1743-0003-10-116
- Brain-computer interface (BCI)
- Electroencephalography (EEG)
- Cognitive load
- Functional electrical stimulation (FES)
- Brain-machine interface (BMI)