Boosting brain–computer interfaces with functional electrical stimulation: potential applications in people with locked-in syndrome

Abstract

Individuals with a locked-in state live with severe whole-body paralysis that limits their ability to communicate with family and loved ones. Recent advances in brain–computer interface (BCI) technology have presented a potential alternative for these people to communicate by detecting neural activity associated with attempted hand or speech movements and translating the decoded intended movements to a control signal for a computer. A technique that could potentially enrich the communication capacity of BCIs is functional electrical stimulation (FES) of paralyzed limbs and face to restore body and facial movements of paralyzed individuals, allowing to add body language and facial expression to communication BCI utterances. Here, we review the current state of the art of existing BCI and FES work in people with paralysis of body and face and propose that a combined BCI-FES approach, which has already proved successful in several applications in stroke and spinal cord injury, can provide a novel promising mode of communication for locked-in individuals.

Introduction

Locked-in individuals have a neurological impairment that leads to severe whole-body paralysis with preserved consciousness and cognitive functioning [11]. Locked-in syndrome (LIS) can be caused by various conditions, including brainstem stroke, trauma, or a progressive neurodegenerative disease (e.g., amyotrophic lateral sclerosis, ALS). Depending on the severity of the paralysis, researchers identify three types of LIS [25]. Classical LIS is characterized by paralysis of all four limbs (quadriplegia), bilateral facial paralysis, and loss of voice and speech (aphonia), while retaining the ability to produce eye movements [269]. Incomplete LIS is characterized by remnants of voluntary movements other than eye movements. Total, or complete, LIS is characterized by whole body paralysis including the eye muscles, causing total immobility and inability to communicate. While individuals with brainstem stroke or trauma either retain a static LIS state or even recover from it [220], individuals with a neurodegenerative disease may gradually progress through different LIS stages over time, in some cases ultimately transitioning towards a total whole-body paralysis with no means to communicate with their family, friends and caregivers [206].

A crucial factor in determining the quality of life of locked-in individuals is their ability to communicate with family and loved ones [52, 251]. Bruno et al. [52] showed the inability to speak in locked-in individuals with LIS due to brainstem stroke is associated with poor subjective wellbeing. Individuals with classical LIS can typically use assistive communication technology based on residual muscle movement, for example, vertical eye movements and eyeblinks [52]. Individuals with total or near-total LIS are not able to use residual movements reliably, and inability to use assistive technology for communication significantly lowers the quality of life [251].

An alternative to assistive communication devices based on residual muscle movement is brain–computer interface (BCI) technology. Both non-implantable and implantable BCI systems record brain signal changes associated with cognitive processing or attempts to produce functional movements (e.g., attempt to move a limb, to make a facial expression or to speak) and use them to control a computer, robotic arm, or another external device. Identifying and directly connecting neural signatures of intended movements to external output systems opens up a possibility for the locked-in individuals to communicate while bypassing their impaired motor pathways [31, 50, 161, 164, 306, 308]. Several studies have demonstrated the potential of BCI to provide communication in persons with LIS [34, 62, 124, 151, 193, 217, 292].

Using decoded movement attempts of locked-in individuals to communicate via external output systems is not the only promising BCI avenue. Studies on individuals with stroke and spinal cord injury resulting in severe body paralysis have shown that BCI combined with functional electrical stimulation (FES) can create an electronic neural bypass of damaged neural pathways by using neural signals to drive stimulation of peripheral nerves and thereby trigger paralyzed muscle activation [5, 44, 201, 226].

Given the fast progress being made in the fields of communication BCIs and BCI-FES, here, we argue that it is worthwhile to investigate whether these concepts can be combined in order to enrich the communication abilities of persons with LIS. In this previously unexplored avenue of research, the application of BCI-triggered FES could be extended to facial muscles to restore simple movements, such as eye blinks or finger taps, or more complex movements, such as facial expressions and potentially even vocalizations and speech articulation.

In the present review, we assess feasibility of the FES technology to boost existing communication BCI approaches for locked-in individuals. We review existing work on assistive communication technology for locked-in individuals, including non-implantable and implantable BCIs, discuss studies on FES applied to body and face muscles and overview existing developments on combining BCI and FES. Finally, we discuss the potential, challenges, and possible future directions of implementing BCI-FES systems for the restoration of communication function in locked-in individuals.

BCI communication for locked-in individuals

Augmentative and alternative communication devices

Studies on augmentative and alternative communication devices typically distinguish three types of such technology: no-tech, low-lech and high-tech devices [94]. No-tech devices rely on body movements and residual speech without aid of additional devices such as computers or eye trackers. Individuals with classical and incomplete LIS, who retain the ability to produce eye movements, can use no-tech “yes–no” response strategy with eyeblinks to answer closed questions or spell with the help of a communication partner reciting letters out loud. Other residual movements such as chin, jaw, forehead, head movements, remaining vocalizations and their combinations can also be employed [24, 90, 101]. Low-tech devices rely on non-computer based equipment such as pen and paper or communication boards with letters, words or pictures. Many systems employ low-tech solutions that combine “yes–no” strategy and communication boards with eye movements and eyeblinks [98, 139]. High-tech devices use computer-assisted technology including audio synthesis of selected and typed text. Depending on the strength and reliability of residual movements, these systems can either facilitate the use of alphabet spelling and pictogram communication boards with automatic selection scrolling (switch scanning) or provide more elaborate text-to-speech solutions based on mouse or joystick control with residual movements. Other high-tech systems can rely on gaze control of a computer cursor based on eye-tracking recordings [20, 169].

The described systems are used extensively with locked-in individuals who retain a certain level of reliable voluntary muscle control [176, 270]. Unfortunately, these systems gradually become unreliable in LIS caused by ALS and can be unreliable after total LIS onset caused by other conditions [121, 203]. Inability to use assistive communication technology based on residual movement may be related to difficulty maintaining stable head position [271], progressive oculomotor impairment and eye gaze fatigue [16, 206, 262], pupil dilation due to the use of medication [64] and other reasons. In such cases, BCIs can be a promising alternative means of communication for the affected individuals [291].

Non-implantable BCI for communication in LIS

Non-implantable neural recording modalities include scalp electroencephalography (EEG), magnetoencephalography (MEG), functional near-infrared spectroscopy (fNIRS) and functional magnetic resonance imagining (fMRI). Practically, scalp EEG and, to some extent, fNIRS are more widely used in BCI research given their portability and affordability compared to MEG and fMRI. Scalp EEG setups can vary in design and number of electrodes depending on the clinical, research or commercial use. For BCI applications, EEG caps of 16 to 128 electrodes covering the entire brain are typically used. Scalp EEG records electrical neural signals at a high temporal resolution but suffers from low spatial resolution and specificity, where the recorded signals typically represent a mixture of neural activity from widespread brain areas resulting from passing of the signals through the protective layers of the brain, skull, and scalp. The working principle of fNIRS is similar to fMRI – the technique measures local changes in blood flow around the brain surface associated with changes in cognitive activity. Because blood flow changes are slow, fNIRS suffers from low temporal resolution. Signal quality of both EEG and fNIRS is affected by motion-related and other artifacts. Moreover, continuous wearing of the EEG or fNIRS cap is not practical for 24/7 home use [312], and in most current setups requires subject and session specific recalibration [174].

A number of strategies have been employed for decoding EEG brain activity for BCI communication, including decoding of mental effort, motor or speech imagery and movement attempts. Use of mental effort is based on a premise that reliable decoding of any meaningful cognitive effort, not specifically related to motor activity or speech, can be used for communication. In that regard, an important subset of early EEG-based BCI research focused on evoked potentials—time-domain EEG signatures of task-related activity. This includes steady-state visually evoked potentials (SSVEP)—a visual cortex response to frequency and position of flickering visual stimuli [3, 246],P300—cortical responses to infrequent stimuli, and motor evoked potentials [87]. P300 has been widely used for spelling and choice-based communication applications and has been adapted to various modalities: visual, auditory and vibrotactile [136, 151, 209]. Frequency-domain EEG signatures widely used in BCI, on the other hand, include slow cortical potentials—slow (below 1 Hz) widespread positive and negative signal amplitude shifts related to task activity [32] and sensorimotor rhythms in mu and beta frequency bands modulated by executed, imagined or attempted movement [211, 310]. Another venue of research is based on error potential responses associated with processing of errors [83]. Error potentials can complement other paradigms, such as P300-based BCIs, wherein P300 responses indicate decoding targets and error potentials arise as response to decoded misses.

Decoding motor and speech imagery with EEG was based on the idea that intention to communicate in BCI target users results in neural processes similar to those underlying imagery in able-bodied individuals. Previous work has shown successful decoding of imagined movements and speech information from EEG [76, 194, 208, 225, 300], see [172] for reviews). In the recent years, however, the BCI field has been focusing more on decoding of actual movements and speech recognizing that attempted movements of BCI users may have more in common with actual movements of able-bodied subjects than motor imagery [126], which in its turn may involve specific and distinct neural mechanisms. Several studies have demonstrated decoding produced speech and movement information from EEG [46, 166, 213, 261, 298]. Multiple reviews cover EEG applications for BCI [2, 112, 160, 210, 244, 293, 310].

Work on fNIRS has so far been more limited but already shown successful decoding of motor, mental workload [117], speech-related [57, 118, 170] and other task-related [205] information from neural signals.

Many of the proposed paradigms have been tested in individuals with LIS, including SSVEP [71, 163], P300 [106, 230], slow cortical potentials [152], sensorimotor rhythms [153] and error potentials [272]. Limited testing with locked-in individuals was performed with fNIRS [1, 97]. Overall, the performance of non-implantable BCIs is promising for a number of tasks but varies greatly across individuals. Some research, however, indicates that prolonged use of paradigms based on SSVEPs and P300 may be tiresome and time-consuming [227]. Compared to naturalistic verbal and non-verbal communication, these paradigms may not be entirely intuitive and comfortable for users. Importantly, gaze-based BCI control paradigms, such as P300, may not work long-term for LIS caused by ALS [51, 121, 203, 291], as oculomotor activity is progressively affected by the disease. In total LIS, with non-implantable BCIs, slow cortical potentials, P300 and sensorimotor rhythms paradigms may not be able to produce consistent and reliable BCI control [150]. Instead, BCI paradigms based on attempted movement and speech may be more natural and intuitive, and therefore preferred strategies for BCI-based communication [48]. Reliance on motor-based paradigms in BCI may additionally be relevant in the context of the “motor theory of thinking” [130] and the “extinction of thought” hypothesis [35, 150] that link the importance of motor control to goal-directed thinking and voluntary cognitive functioning, particularly in people with ALS (see Clinical considerations about LIS).

Implantable BCI for communication in LIS

Implantable BCIs for LIS individuals capitalize on superior signal quality provided by direct recordings from the brain tissue and the high temporal and spatial resolution that implanted neural recording techniques provide compared to EEG, MEG and fNIRS. Specifically, intracranial EEG that includes electrocorticography (ECoG), or surface grid electrodes, and stereo-EEG (sEEG), or depth electrodes, allows recording from 1 to 2.3 mm diameter patches of cortex around each electrode and can sample from up to 256 electrodes. ECoG provides a spatial resolution of 3–10 mm with gaps between electrodes and typically requires an invasive surgical procedure, yet signals remain stable over time [159], and there is no tissue scarring. Therefore, ECoG has been tested and approved for long-term use in human participants of BCI trials. Related stentrode technology records brain signals via electrode arrays arranged on a flexible mesh tube [216], Fig. 1). An array of up to 16 stentrodes, 750 μm diameter each, can be inserted through a jugular vein and guided towards a vessel adjacent to sensorimotor brain area with a catheter. Multielectrode arrays (MEAs) and neurotrophic electrodes allow individual neural cell recordings. While providing superior spatial resolution, MEAs offer smaller cortical coverage with up to 256 needles per array covering up to 2.8 × 2.8 mm of cortex. MEAs are made of stiff materials and penetrate the cortex, which can lead to tissue scarring and gradual signal loss [68]. Typically, high frequency (> 60 Hz) and low frequency (13–30 Hz) components of ECoG and stentrode signals are used for decoding information from the brain. In MEA, spike information and spike band power are used.

Fig. 1

Example of an implantable BCI system for a locked-in individual with ALS based on attempted hand movements. Copyright: [217]

Various neural decoding strategies have been explored in pre-clinical studies using invasive recordings in non-human primates and able-bodied individuals. Non-human primate studies with ECoG and MEAs have targeted decoding of reaching and grasping hand movements as well as continuous hand movement trajectories for potential translation to human BCIs [58, 259, 282, 294]. Human recordings with temporarily implanted ECoG or sEEG electrodes are possible in patients who undergo clinical epilepsy monitoring. In such cases, electrode implantation can help localize epileptogenic sources and guide subsequent resection of the affected neural tissue. Research in this setting has contributed to detailed study of sensorimotor brain areas, such as the “hand knob” and “face area”, revealing somatotopic organization of individual fingers [257] and speech articulators [43], respectively.

In addition, this pre-clinical human research has demonstrated successful decoding of discrete motor information, such as basic hand movements [228, 273], hand movement towards a specific location [53, 301], hand gestures [37, 47, 165, 218], basic mouth movements [255], facial expressions [256] and continuous motor information, including 2D and 3D finger trajectories, speed, acceleration and force of movement [44, 53, 70, 267, 303]. In parallel, decoding of speech has shown success in decoding individual words [28, 138, 186, 192, 198, 274], phonemes [36, 42, 116, 199, 242], syllables [171, 274] and even continuous decoding of full phrases and sentences [14, 115, 116, 183, 198, 278, 304].

Success of pre-clinical studies with non-human primates and able-bodied humans led the way towards a demonstration of a proof-of-concept BCI system that could enable communication for locked-in individuals [292]. The study demonstrated an ECoG-based BCI for computer control and spelling based on accurate decoding of a finger tapping movement attempted by a user with ALS. Later studies have shown similar success using stentrodes and decoding of attempted leg movement [195, 217]. MEA-BCI systems have been employed for decoding of hand movement “point and click” patterns for computer cursor control by participants with severe paralysis [17, 219]. Another recent study trained a user with total LIS to spell letters using a neurofeedback strategy: the user’s neural activity levels were mapped to auditory tones and played back to the user, who was asked to modulate his brain activity to match the target tone [62]. The most recent BCI work with both ECoG and MEAs in people with motor impairment demonstrated successful attempted word and speech decoding for display on a computer screen for communication [192, 198, 304]. Importantly, only a handful of studies investigated the potential of implanted BCIs in individuals with late-stage ALS in total or near-total LIS, including complete inability to vocalize [62, 292]. For translation of the BCI technology from research to real-world user applications, fully-implantable home-use BCI systems will be needed, wherein BCI users can control the computer with their brain activity continuously throughout day and night without supervision by researchers [217, 292].

Overall, implantable BCI approaches offer several attractive features and have shown high performance when used in naturalistic communication paradigms based on attempted movements and speech. At the same time, non-implantable BCI systems can also reliably decode cognitive and motor attempts that can be used for communication while offering inexpensive portable BCI solutions without complex surgical intervention. In both implantable and non-implantable BCI setups, decoded communication attempts can be displayed on a computer, used for brain-driven control of a cursor, a custom or commercial speller or communication app. External devices and tools can also be used to augment the user experience. These can involve a speech synthesis device, a virtual avatar, an external mobility device, such as a robotic arm or exoskeleton [26, 79, 123, 305], see [39] for a review).

Functional electrical stimulation (FES)

Among such tools used in BCI applications for rehabilitation in stroke and spinal cord injury is also functional electrical stimulation (FES). Similar to the use of external mobility devices, FES is aimed to restore or improve BCI user’s motor control. It does so by reanimating weakened or paralyzed muscles of user’s body and limb. With the majority of FES applications being directed to people with stroke or spinal cord injury, it has not yet been employed directly for communication purposes.

What is FES?

FES refers to application of electrical stimulation to neural tissue with the aim to restore its lost or damaged function [222, 252, 254]. Often, FES is applied to peripheral nerves to generate contraction of the connected muscle and induce functional movement. The electrical stimulation can be administered using non-invasive setups (transcutaneously, with electrodes and stimulator on the surface of the skin) or invasive setups (percutaneously, with electrodes piercing the skin and an external stimulator, or via fully-implantable systems with electrodes and a stimulator implanted under the skin).

Transcutaneous FES is carried out with electrodes placed on the surface of the skin. It is a non-invasive, relatively inexpensive, low maintenance and widely popular approach in clinical and research settings. However, because there are different structures between the surface electrode and the stimulation target, including pain fibers, other non-target nerves and muscles, it may be difficult to use transcutaneous FES for activation of isolated muscles, especially deep muscles. As a result, stimulation typically requires higher charges compared to invasive techniques. Since pain fibers can be in the way of stimulating the target muscle, unlike invasive approaches, surface stimulation can cause skin sensations that can range from slightly tingling to strong discomfort and painful [129, 222].

FES with percutaneous electrodes involves applying stimulation directly to the nerve in the muscle, which can be achieved using needle electrodes that pierce the skin [185, 188, 222, 239, 241]. Percutaneous electrodes are less likely to cause pain as they bypass the skin and can activate isolated and deep muscles. In FES setups with percutaneous electrodes, the stimulation device is placed externally. Fully-implantable FES systems, on the other hand, were designed for long-term use and involve under-the-skin implantation of the electrodes as well as the stimulator device that can be powered and controlled with an external unit. Cochlear implants that restore hearing and implants for restoring grasp in tetraplegia are examples of implantable FES systems.

Typically, FES is applied to the point where the nerve enters the target muscle – the motor point, as stimulating the motor point requires the lowest stimulation threshold to induce muscle contraction [252]. Implantable FES systems may allow for more fine-grained stimulation targeting and can include electrode placement on the muscle surface, within a muscle, adjacent to a nerve, or around a nerve [222]. In most cases, these different configurations aim at exciting the nerves that innervate the muscles. Some literature refers to direct stimulation of muscles in the case of severe muscle denervation (see “Challenges and future directions for more detail”).

Apart from the differences in the outlined FES setups, other factors can influence FES outcomes and performance including conductivity of the underlying tissue, muscle training and the degree and treatment of muscle fatigue [222, 252]. Moreover, FES is characterized by a number of stimulation parameters (Fig. 2), such as the stimulus waveform, the pulse duration, or width, the current amplitude, and the stimulation frequency, which can all affect stimulation results. Biphasic waveforms are deemed safer for the underlying nervous tissue as they can balance out the electrochemical processes caused by stimulation thereby minimizing tissue damage [197]. Some research has examined the effects of distinct waveform shapes and other parameters on FES performance and user comfort [19, 27, 45, 72, 129, 180], but overall such effects are non-trivial to compare across studies due to the large degree of variability in FES applications.

Fig. 2

A visual illustration of the stimulation parameters used in FES research including pulse frequency, pulse amplitude, pulse duration and different types of waveform: symmetric biphasic, asymmetric biphasic and monophasic waveform

As such, FES applications vary with respect to the area of stimulation (face, hand, leg, ear, bladder, etc.), specific underlying neural impairment (stroke, spinal cord injury, deafness, etc.), target application (restoration of respiration, grasp movements, walking, hearing, etc.), design of the control system (user-controlled open-loop systems, systems with continuous stimulation, closed-loop or feedforward control, etc.). Multiple reviews cover FES principles and applications [81, 185, 188, 222, 232, 236, 237, 253, 263, 311], including for stroke [8, 127, 149, 239, 289], spinal cord injury [49, 100, 122, 177, 234, 241], Parkinson’s disease [277] and facial nerve paralysis [54], [86].

FES in limb, body, and face paralysis

Given the focus of the present paper on communication for individuals with severe whole-body paralysis, below we review studies on clinical and research FES applications that can induce higher-level voluntary motor activity, such as functional movements of body, limbs, and face. Many people with LIS use residual movements of the hand or face to answer closed questions or to control their communication technology device. We propose that several types of FES-generated movements can potentially be used either for coded communication, similar to augmentative and alternative communication devices based on residual movements, for example, eye blinks, finger taps or other upper or lower limb movements, or for direct verbal or non-verbal communication, for example, facial expressions, sign language or speech.

Limb and body paralysis

The utilization of FES is well-established within research and clinical settings for the improvements or restoration of motor functioning following several medical conditions, for example, stroke, spinal cord injury, cerebral palsy, multiple sclerosis, Parkinson’s disease, trauma, and others. Many of these conditions result in weakness or paralysis of different body parts, which can affect functional upper and lower limb movements and balance.

Most work on inducing or strengthening upper limb movements with FES has been done with transcutaneous electrodes (except for studies that use the implanted Freehand system). Previous work has shown that FES delivery to peripheral nerves can facilitate and induce various functional movements of hand and arm [222, 253]. Concurrent transcutaneous stimulation in large shoulder and arm muscles, such as anterior and posterior deltoids, triceps and biceps, can induce reaching movements in able-bodied participants [248, 302] and individuals with upper limb paralysis [286].

Different types of hand grasp, including the palmar power grasp used to hold large, heavy objects by flexing fingers against the palm, and the pinch precision grasp used to hold objects between the thumb and other fingers for fine-grained manipulation [89], can be improved and induced by transcutaneous FES in able-bodied participants [296] and individuals with upper limb paralysis [286]. This can be done by concurrent stimulation in forearm muscles, such as wrist and finger flexion muscles (flexor carpi radialis, flexor carpi ulnaris, flexor digitorum profundus, flexor digitorum superficialis and others), and wrist and finger extension muscles (extensor carpi radialis, extensor carpi ulnaris, extensor digitorum and others). Implanted FES has been shown to induce grasp movements in individuals with paralysis [221].

In addition to basic grasp movements, transcutaneous FES has been shown to induce finger extension in the paralyzed arm [59, 65, 148],hand opening, fist clenching and wrist extension with the intact and paretic arm [65, 113],functional gestures such as closing a drawer, button pressing, switching on a light switch in both able-bodied participants and individuals with upper limb paralysis [157] and individual finger movements and hand postures for playing a musical instrument in an able-bodied participant [280].

Transcutaneous and implantable FES can also facilitate and induce movements of the lower limb including movements of the thigh, ankle and foot by stimulating in hamstring, quadricep, peroneus longus and other leg muscles [18, 142, 158, 285].

Stroke and spinal cord injury have a long history of using FES-based therapy in a clinical setting with commercially developed FES systems with transcutaneous stimulation, such as the NESS Handmaster [10, 249], the H200 Wireless Hand Rehabilitation System [9], the COMPEX Motion Stimulator [236, 286], MyndMove [114], Parastep-1 [103] and implanted setups, such as the Freehand System (B. [268]. FES-based therapy been shown to significantly improve upper and lower limb motor function in both stroke [80, 114, 185, 233, 275, 283, 286] and spinal cord injury [132, 135, 137, 146, 184, 223, 235, 253, 284, 285]. The knowledge and experience about FES, its clinical practice and commercial applications accrued in the field of motor restoration after stroke and spinal cord injury can be particularly informative for the development of stimulation-based applications in novel fields, such as FES for communication in individuals with LIS.

Facial nerve paralysis

Another type of FES application for inducing functional movements that could be used for communication is work on facial nerve paralysis. Facial paralysis can occur as a result of damage to the facial nerve or facial muscles caused by various conditions. Unilateral facial nerve paralysis is rather common and is typically caused by Bell’s palsy, infection, trauma, stroke, developmental condition or tumor [215]. Bilateral facial nerve paralysis is rarer and can be caused by the same conditions as well as the Moebius syndrome. Facial nerve paralysis affects 20–30 per 100,000 individuals each year across several countries [7, 60, 134, 224].

The facial muscles (Fig. 3) are innervated by the facial nerve (VII cranial nerve) and are responsible for essential functional movements of the face. Transcutaneous FES administration in the eye muscle (orbicularis oculi) has been shown to produce eyeblinks at low levels of pain and discomfort in able-bodied participants [95, 128, 178, 243] and in individuals with facial nerve paralysis [96, 179, 181, 245].

Fig. 3

Illustration of the facial muscles. The facial muscles that have been stimulated in previous facial FES studies are highlighted in red: the frontalis (forehead), orbicularis oculi (eye), zygomaticus major (cheek), masseter (cheek), and orbicularis oris (mouth). Adapted from [309], copyright by the authors

Recent work has demonstrated that transcutaneous FES can induce eyebrow raises in able-bodied participants [128] and in individuals with facial nerve paralysis [180, 182] by triggering forehead muscle (frontalis) contraction (Fig. 4). Transcutaneous FES in the mouth muscle (orbicularis oris) can induce lip puckering in able-bodied participants [128] as well as in individuals with facial nerve paralysis [182, 245, 297]. Cheek muscle (zygomaticus major) activation with transcutaneous FES can induce a smile in able-bodied participants [128] and in individuals with facial nerve paralysis [15, 182, 245, 297]. Inducing a smile with transcutaneous FES has however been more challenging compared to eyeblinks, eyebrow raises and lip pucker due to cross-stimulation of the eye and increased levels of discomfort [128]. In all other cases, studies reported low levels of user pain and discomfort in both able-bodied participants and individuals with facial paralysis.

Fig. 4

Example of applying FES electrodes for external activation of facial muscles. Copyright: [128]

Percutaneous stimulation in facial muscles of able-bodied individuals has been shown to trigger appearance of fine-grained facial movements, such as inner and outer eyebrow raises, cheek raising, eyebrow lowering, smiling, lip stretching, chin raising, nose wrinkling and others [299]. Percutaneous FES in the temporarily anesthetized forehead muscle of an able-bodied participant induced bilateral eyebrow raises [154]. In individuals with facial nerve paralysis, intraoperative transcutaneous and percutaneous stimulation in the chewing muscle (temporalis) has been shown to produce a smile [111]. Overall, however, despite these promising results, research on facial stimulation with percutaneous and implanted electrodes appears to be rather scarce.

Finally, a recent study showed that some speech-related activity can be induced in an able-bodied participant by application of transcutaneous FES in the cheek muscle (zygomaticus major, [258]. The study demonstrated that FES could induce pronunciation of a consonant /v/ with acoustic sound features similar to those of the consonant /v/ freely produced by the participant without applying stimulation. This result, however, could only be achieved with self-controlled stimulation by the participant but not with externally controlled stimulation by the experimenter.

FES-based therapy is an actively developing field that aims to alleviate physical consequences of facial nerve paralysis, such as the inability to blink, produce facial expressions, difficulty with speech production, and inability to selectively contract facial muscles (synkinesis), which can lead to altered facial expressions [73, 245, 297]. In unilateral facial nerve paralysis, much research effort with transcutaneous FES is dedicated to facial pacing: restoring movements on the paralyzed side of the face based on preserved movements on the intact side of the face. Multiple reports have demonstrated successful application of transcutaneous FES for promoting facial nerve paralysis recovery and symptom management [145, 175, 238, 288], see [54, 86, 156] for reviews). Ongoing research on facial nerve paralysis, its psychological and physical manifestations and clinical FES applications for improving and restoring facial movements can be particularly informative for developing FES systems based on natural verbal and non-verbal communication.

Complementation of BCI with FES

BCI-FES for the regulation of stroke and spinal cord injury

Recent research has shown that a combination of a BCI system and a FES system has the potential to create an electronic neural bypass to externally activate muscles affected by paralysis in individuals with stroke and spinal cord injury [133, 143, 185, 265, 266, 281] (Fig. 5). This can help improve motor control by strengthening weakened muscles or restore motor control by inducing lost functional movements [307]. In a combined BCI-FES neuroprosthesis system, neural patterns associated with attempted movements are detected from brain activity, and this information is then used to trigger FES administration in the corresponding muscles. Non-implantable and implantable BCI systems complemented with transcutaneous or implantable FES have been employed to promote rehabilitation in people with stroke and spinal cord injury.

Fig. 5

Example of a combined BCI-FES system for restoring upper limb movement in an individual with spinal cord injury. A 96-channel Utah MEA. B Zoomed-in view of array orientation (yellow) on the left motor cortex. C Head computerized tomography image displaying the implant location. D Representation of the array location (yellow) on the precentral gyrus. Copyright: [40]

Non-implantable BCI-FES

Most of the research using non-implantable BCI combined with transcutaneous FES systems on stroke patients has focused on the restoration of arm and hand movements. Initial case report data that showed improvements in upper limb motor function for BCI-controlled FES compared to FES-only therapy [200] was further supported by larger clinical trials [30, 63]. Although fewer studies have used such BCI-FES systems for post-stroke rehabilitation of lower limb motor function, their findings confirm that BCI-FES systems lead to improvements of motor function [77, 189, 190, 279], and are more effective compared to FES-only therapy [66].

A combination of EEG-based BCI and transcutaneous FES was first used to restore hand grasp in an individual with spinal cord injury in the pioneering work by Pfurtscheller et al. [226]. A subsequent study combined an EEG-based BCI with an implanted FES system [202]. More recent studies on BCI-FES for people with spinal cord injury have replicated and extended these findings [99, 119, 167, 201], also confirming that combined BCI-FES setups provide significantly greater neurological recovery and functional improvement compared to FES-only therapy [214]. A recent usability study on combined BCI-FES technology for spinal cord injury rehabilitation showed that users positively evaluated the perceived technology effectiveness, particularly appreciating their apparent active role in the experience (seeing their hand move) and the potential such technology offered for improving interaction with their loved ones [314]. This is in agreement with earlier work on user priorities for BCI where user surveys indicated preference of individuals with spinal cord injury for coupling BCI with FES over a BCI that controls a computer, wheelchair, or robotic arm [69].

Implantable BCI-FES

While non-implantable BCI technology provides advantages in terms of availability and safety, implantable BCI technology provides greater neural signal quality in terms of spatial resolution and signal-to-noise ratio [21]. This can potentially allow for decoding of more fine-grained information from the brain about the attempted movement and conceptually could help improve or restore complex continuous control of dynamic functional movements, using FES, as opposed to simple discrete motor actions, such as grasp and release movements.

Recent research has attempted to utilize implantable BCI-FES systems based on ECoG or MEA recordings for the restoration of functional movements in paralyzed individuals. In 2016, a combined application of an implantable MEA-based BCI and transcutaneous FES in a human participant with tetraplegia led to successful restoration of volitional finger, hand, and wrist movements of a paralyzed limb [44]. The clinical evaluation results showed that, while using the system, the user’s motor impairments significantly improved, which granted the tetraplegic patient with the functional ability to grasp, manipulate, and release objects. These findings were further supported and extended in follow-up research with MEA-based BCI systems combined with FES [5, 13, 39, 67, 93, 260]. Ajiboye et al. [5] combined an implantable BCI and a FES device with percutaneous electrodes and an external stimulator to restore functional reaching and grasping movements in an individual with severe tetraplegia, allowing him to repeatedly drink a cup of coffee and feed himself using his own arm and hand, solely on his own volition. Colachis et al. [67] employed an implantable BCI and transcutaneous FES system that allowed a tetraplegic patient to complete dynamic functional movements using FES-stimulated arm and hand muscles simultaneously with non-paralyzed shoulder and elbow muscles on the same side. Their findings showed that a tetraplegic patient could utilize the system to perform functional tasks ~ 900 days post implementation, reflecting the strong translational potential of implantable BCI-FES systems for daily life settings. Furthermore, Bockbrader et al. [40] used an implantable BCI and transcutaneous FES system to restore skillful and coordinated functional grasping movements (lateral, palmar, and tip-to-tip grips) that provided clinically significant gains in assessment of upper limb functioning in an individual with spinal cord injury (Fig. 5).

A recent study has presented a combination of an EcoG-based BCI and a transcutaneous FES system for successful restoration of volitional hand grasp in an individual with a spinal cord injury [55]. Finally, a recent fully-implantable EcoG-FES system has been used for restoration of walking in an individual with spinal cord injury [173]. The researchers used an implanted spinal stimulation device controlled by an EcoG-based BCI that detected intended movement in bilateral motor cortex neural activity.

Future BCI-FES research avenues for communication in LIS

Locked-in individuals experience a loss of motor function, which can cause a total body paralysis. As a result, the affected individuals may be left with little to no means of communication. Augmentative and alternative communication devices and BCI communication systems can offer these individuals alternative ways to communicate. Here, we propose that a technology based on combined BCI-FES systems for the restoration of functional motor activity could represent another research direction that may be worthwhile exploring to provide communication means in the case of LIS.

Restoring coded communication

Any intentional natural movement that has been accurately decoded from brain activity and subsequently induced with FES could potentially be used for coded communication and spelling in LIS. Transcutaneous, percutaneous, and implanted FES have demonstrated promising results in producing various movements of the hand, arm, leg, and face. FES-induced movements can allow for binary “yes–no” and more elaborate multi-class communication modes. This could be coupled with eye trackers, virtual reality and robotics, and potentially new combined technology can be developed.

Restoring natural communication

Questionnaires about user needs in assistive communication technology have shown that individuals with LIS have a strong preference for natural personal communication if possible via attempted speech [48, 251]. In fact, having access only to coded “yes–no” type of communication is associated with lower quality of life compared to richer communication modes [251]. Based on existing work in brain signal decoding and FES-induced body movements, it may also be possible to develop combined BCI-FES technology for more naturalistic communication than coded messages and spelling. Such technology could provide a richer mode of communication that not only delivers the content of a communication message but can help to express its affective component as well by restoring elements of body language and facial expressions, thereby going beyond what most current communication BCIs aim to offer. For example, a BCI-FES device could allow for pointing and gestures, and fine-grained hand control could make sign language and handwriting possible. Previous BCI work has shown that individual hand gestures, signs of the American Sign Language and handwritten characters can be decoded from neural activity [47, 162, 165, 218, 303], and both transcutaneous and implantable FES could facilitate finger muscle control.

Another potential strategy to communicate can be based on restoration of facial expressions. It has been previously shown that information about facial movements and facial expressions can be decoded from human brain activity for BCI purposes [38, 191, 255, 256]. Transcutaneous and percutaneous FES can induce various facial movements, such as eyebrow raises, smiles, and lip puckers, which separately or in combination can lead to perception of a number of facial expressions including anger, happiness, fear, sadness and others [82]. This can facilitate rich and natural communication of experienced emotions of the affected individuals with their family and caregivers and have a positive effect on their social interactions overall. Previous studies have indicated that individuals who are unable to produce facial expressions, such as Moebius patients with facial nerve paralysis and persons with LIS, may exhibit deficits in recognizing emotional facial expressions in others [23, 56, 74, 229]. Restoring the ability of locked-in persons to produce facial expressions may potentially improve their facial mimicry – an automatic facial reaction that attempts to reproduce perceived emotional facial expression and that is thought to facilitate emotion recognition [212, 231, 276]. FES-induced smiling in FNP patients and individuals with major depressive disorder has been linked to improved quality of life [75, 168] and might provide similar benefits to persons with LIS.

Although likely unrealistic in the foreseeable future, yet still potentially conceivable, is the idea to decode spontaneous speech from brain activity and use it to trigger orofacial and laryngeal muscles for inducing speech in locked-in individuals. Implantable BCI research has shown impressive results in decoding speech from the brain including individual phonemes, words and even continuous speech segments [29, 41, 61, 187, 240] for reviews). Transcutaneous FES research indicates that laryngeal and speech articulators can be activated with external stimulation for producing speech-related motor activity [155, 258]. Although these FES results are preliminary, they are also promising, as BCI-driven FES-induced speech production has the potential to truly transform the way locked-in individuals could communicate in the future.

Overall, restoring either basic coded communication or more natural communication with combined BCI-FES systems could potentially provide previously inaccessible benefits to users with LIS by offering (1) direct interpersonal communication without having to rely on a computer screen, (2) a natural way to interact with the world, (3) a possibility to engage body language and facial expressions to communicate emotion and social signals, (4) an increased sense of agency and control, (5) somatosensory feedback by design—an important component that has been shown to have positive effects on accuracy and speed of produced movements in BCI users [91].

Challenges and future directions

There are several challenges and considerations that need to be addressed when testing the feasibility of BCI-FES systems to restore functional movements of body and face in locked-in individuals for the purpose of communication.

Clinical considerations about LIS

The first challenge relates to the variability of conditions that can potentially lead to LIS. LIS can be caused by various conditions, such as brainstem stroke, trauma, or a progressive neurodegenerative disease (e.g., ALS), and each of them likely differently affects the neuromuscular integrity of limbs and face and therefore the potential success of complementing a communication BCI with FES.

In a situation where paralysis is caused by a disruption of the neural connections between the upper neural pathways and the peripheral and cranial nerves, as is typically the case in brainstem stroke or trauma, it may be feasible to use FES successfully. Notably, individuals with LIS caused by brainstem lesion usually preserve spontaneous involuntary facial expressions while being unable to produce voluntary facial expressions [125, 287]. This dissociation of reflexive and voluntary control may indicate that the facial muscles and peripheral nerves can in principle be excited to produce target expressions, but the upper neural pathways involved in voluntary facial movements are impaired. Thus, a BCI-FES combination may help bypass the impaired pathways and restore voluntary facial expressions and potentially other movements that could serve communication in locked-in individuals with a brainstem stroke.

Another important consideration in paralysis that needs to be taken into account is the degree of muscle denervation. Muscle denervation refers to the reduction of nerve inputs into the muscle that causes a decrease in neural input necessary for muscle activation and promotes muscle atrophy. Previous work, however, has shown that transcutaneous FES can be used to induce movement of denervated muscles even several years after the onset of paralysis [15, 141, 156]. Activation of denervated muscles with FES may be possible partly due to the process called reinnervation, in which intact neural pathways take over the damaged nerves in controlling muscle activation.

In a neurodegenerative disease, such as ALS, however, the ongoing reinnervation may not be sufficient to preserve functional motor units and as the disease progresses, it will not help compensate for the continually increasing amount of muscle denervation [110]. Over time, this will inevitably lead to muscle thinning, muscle atrophy, muscle infiltration with fatty tissue and nerve atrophy [22, 204]. Limited earlier work has demonstrated successful muscle excitation with electrical stimulation in individuals with ALS [6, 109]. It has been speculated that FES-based therapy could conceivably attenuate or even delay the disease progression by decreasing muscle tightness and increasing muscle strength [109]. However, other work called into question the feasibility and benefits of electrical stimulation in ALS, especially in the context of muscle denervation at later stages of the disease [12, 120]. Therefore, the stage and progression of the disease may be the determining factor of success in applying FES to restore muscle movement in individuals with ALS. More research on muscle denervation in ALS and FES application in denervated muscles is, however, needed to make more specific prognosis about potential outcomes of FES application in LIS caused by a neurodegenerative disease.

The timing of introducing FES and BCI assistive technology to individuals with LIS may also affect how successful its use will be. Previous work shows that introducing BCI to people in total LIS may produce unreliable BCI control. It appears that the longer the person stays in the total LIS state, the harder it may become for them to elicit goal-directed behavior. A hypothesis called “extinction of goal-directed thinking” [33, 150] suggests that loss of motor control can be associated with cessation of voluntary cognitive activity. In addition to this, prolonged total LIS may lead to alterations in consciousness and arousal [313]. Under the “extinction of goal-directed thinking” hypothesis, it seems reasonable to assume that the loss of cognitive activity occurs gradually due to the progressive lack of sensorimotor feedback. In that regard, it may be worthwhile to investigate whether FES-based assistive technology offered to people at earlier stages of ALS could stimulate active motor control, thereby potentially leading to more reliable performance of BCI-based communication at later stages of the disease.

Finally, a combination of motor and cognitive decline in LIS may pose an additional challenge for the choice of strategy for the BCI-FES control. LIS caused by lesion or disease can lead to structural changes in target cortical circuits of face and hand leading to functional deterioration of the corresponding motor networks [295]. One could consider addressing this challenge by remapping of triggers to outputs, and, for example, using various mental strategies to spell or control a computer cursor. However, it may be questionable how intuitive and user-friendly a non-motor based mental strategy would be for FES control. Overall, severe cognitive, motor and structural impairments in LIS pose a fundamental problem for long-term use and use in total LIS for any assistive technology for communication, and it needs to be studied systematically, not only in the context of FES.

In the case where FES cannot successfully induce movements for communication, such as likely during the late stages of ALS, alternative techniques combined with BCI may still apply and benefit from the knowledge gained with existing successful FES applications. Understanding the neural musculature of body and face, the mapping between cortical neural signals and muscle movements and factors determining successful muscle activation with stimulation could inform novel assistive technology. Such technology could be based on virtual reality, including facial avatars and digital twin development, robotics, orthotics, bionic facial masks, and gloves to provide alternative means of communication and interaction with the world for the locked-in individuals.

Technical considerations of FES

In the case of translating existing BCI-FES setups to restore motor functioning of upper and lower limb, there may be a number of technical challenges. Optimal choice of stimulation parameters especially in transcutaneous FES needs particular care. Research shows that, in general, paralyzed limbs and face require higher stimulation amplitudes [182, 245, 297], which may lead to adverse effects of stimulation. Some studies show that increasing another parameter—pulse duration, may lead to muscle contraction at lower amplitudes [156] and thereby decrease user pain and discomfort [96, 297]. At the same time, the use of larger pulse durations results in lowered frequency of stimulation. The latter, however, is recommended to be set at 20–40 Hz for inducing smooth continuous movements [78, 222]. Further systematic investigation of optimal parameter configurations may be crucial for acceptability of FES and BCI-FES technologies as a therapeutic tool.

Another challenge in the case of FES applied to facial muscles is the precise electrode placement given the overlapping nature and very small size of facial muscles. Some effort has started on developing recommendations and protocols regarding electrode size and fixation for application of transcutaneous FES to facial muscles [247]. Regarding electrode type and placement, transcutaneous facial FES research has relied on existing electromyography guidelines [92]. Many studies, however, note the lack of an electromyography site “atlas” available for the facial musculature, and that relevant data is only available for three facial muscles: cheek (zygomaticus major), eyebrow (corrugator supercilia), and forehead (lateral frontalis) muscles. Some researchers are developing facial mapping techniques to identify the facial sites that induces the strongest contraction of the relevant facial muscle [96], but this practice is not yet widely used in the field.

Another potential challenge is the FES-induced muscle fatigue due to the fact the FES currents may activate muscles in a way and order that are different from naturally induced movements. Specifically, during natural voluntary movement, activation of small fatigue-resistant fibers happens first and then propagates to larger more fatigable muscles, whereas FES may activate larger muscles first or activate muscles non-selectively compromising the natural order of muscle contractions [88, 104, 290]. Animal studies show that stimulation with implanted electrodes may be able to manipulate muscle recruitment order and tackle fatigue [84, 85]. With transcutaneous FES, it has been shown that manipulating stimulation parameters can have an effect on fatigue with larger pulse frequencies increasing it [102, 105] and longer pulse durations potentially decreasing it [131]. Several studies underline the importance of muscle conditioning and training in reducing fatigue [107]. Despite these efforts, a lot remains unknown about the mechanisms of muscle fatigue and its prevention, and in many cases individual differences between participants continue to determine FES results.

Finally, it is worth considering that using percutaneous and implantable FES electrodes may hold additional benefits for targeted activation of face and body muscles compared to transcutaneous FES. Implantable FES systems may not only be more practical for long-term use, but they may also offer better positioning of the electrodes, do not suffer from issues with electrode–skin impedance on the skin and bypass skin pain receptors. Given that current percutaneous and implantable FES research, especially with facial muscles, shows promising results but remains limited, this could be a potentially noteworthy direction for explorative research on its own and in combination with BCI.

Potential of combined BCI-FES communication systems for long-term use

Several studies have demonstrated successful long-term use of individual FES and BCI systems [4, 68, 107, 108, 193]. They show promising results regarding performance and user satisfaction and should be used as guidance in development of combined BCI-FES setups. One important aspect this work can explore is long-term effects of using an assistive technology. In the BCI field, this work focuses on long-term stability of recorded neural control signal and aims to incorporate adaptive data processing methods [68, 250, 264]. Both BCI and FES research indicate the importance and challenges associated with training, motivation, and practical implementation of the system [107, 147, 196, 207]. Work with FES highlights its potential in some cases to counteract muscle denervation and promote muscle growth [140, 275]. Studying long-term use of combined BCI-FES technology is necessary to further understand the long-term effects of this technology on the users.

For long-term use, fully-implantable systems may be preferred. This is more practical and esthetically pleasing, which decreases the burden on the user, family, and caregivers. Fully-implantable systems, however, are associated with increased risks of infection, and in the case of a fully-implantable BCI-FES, a potentially large number of components may need to be implanted, leading to an extensive invasive surgery on user’s body and head. Importantly, one of the biggest concerns about the use of implantable FES systems today is increased risk of infection [144]. Overall, various factors need to be carefully considered in designing long-term BCI-FES systems in order to minimize risks yet maximize benefits for the users.

Conclusions

Recent advances in BCI technology and FES fields have led to the development of combined BCI-FES systems that have demonstrated the feasibility of restoring functional movement of paralyzed limbs in individuals with stroke or spinal cord injury. We anticipate that this research can lead the path to development of novel tools for assistive communication for locked-in individuals as a result of a brainstem stroke, trauma and perhaps neurodegenerative disease, such as ALS. This can be done in two ways: (1) by directly applying existing BCI-FES strategies for upper and lower limb reanimation to locked-in individuals to accomplish basic communication signals, or (2) by exploring a novel direction of reanimating facial movements and expressions with a combined BCI-FES approach. Facial FES research has shown promising results of restoring functional facial movements and expressions in patients with facial nerve paralysis, inducing eyeblinks, eyebrow raises, frowning, smiling. Even limited speech articulation may be conceived for this direction of research. Combined with successful decoding of attempted and intended movements and speech from neural activity of the affected individuals with BCI, we expect a combined BCI-FES approach for providing communication to locked-in individuals to emerge. FES-induced movements and facial expressions driven by neural activity could allow locked-in individuals to express themselves more effectively, which would likely have positive effects for their wellbeing and quality of life.