Participant
The study involved a single participant (the pilot) who has tetraplegia with normal vision and hearing, aged 49 at the time of the Cybathlon 2020. The pilot suffered a spinal injury (fractured C4–C5) in 1993 during a motorbike accident. Prior to the commencement of the training, the pilot was presented with information regarding the experimental protocol and was asked to read and sign an informed consent form to participate in the study, which was approved by the National Rehabilitation Hospital of Ireland research ethics committee. Before the beginning of the BCI training carried out in 2019 and 2020 (reported in this paper), the pilot took part in 10 basic BCI training sessions in 2009 and 12 training sessions for Cybathlon 2016.
Experimental paradigms
The BrainDriver BCI racing game was used in the Cybathlon BCI challenge (described at the end of this section). Our online BCI uses analogue outputs of four 2-class classifiers to relay control commands to the BrainDriver game (described in “The online BCI” section). The BrainDriver race is controlled using commands received from the BCI but does not provide continuous feedback to the pilot about the analogue output of four 2-class classifiers which are built into the BCI framework. As accurate control of the avatar in the BrainDriver race requires each of the four 2-class classifiers to be pilot-specifically calibrated, data must be collected to train the 2-class classifiers. For this purpose we used another BCI game called NeuroSensi [20, 21]. Additionally, to present the pilot with outputs from four classifiers simultaneously we used a novel paradigm referred to as the Triad game, which is introduced in this study for the first time. These are described below and shown in Fig. 1.
NeuroSensi game training for paired motor imagery tasks
The first phase of the BCI training which took place in 2019 and 2020, involved the NeuroSensi BCI game (Fig. 1A, Additional file 1: Video S1) which is played using two motor imagery commands. The NeuroSensi game has a representation of a neuronal axon on both sides of the monitor. Two seconds after the beginning of the trial, a light (representing a neural spike) appears at the beginning of one of the two axons to cue the participant to begin the corresponding motor imagery task. The light takes 6 s to travel over the ‘axon’ during the task period (Fig. 1A). In each NeuroSensi session, six runs were completed wherein different binary combinations of the three commands (left hand (L), feet (F), right hand (R)), and relax (X) were performed. The number of trials in each run acquired for BCI calibration (i.e., for calibrating/training hyper parameters of the BCI framework—described in “Calibration of the two-class classification modules” section) varied between 30–60 (equal number/class), depending on the actual session ID (more trials in the initial session, fewer trials in later sessions). The time duration of a run, therefore, varied between 240 and 480 s. The time duration of six runs involving L vs. R (LR), FR, LF, LX, FX, and XR tasks, including five 90 s inter-run pauses, varied between 20 and 30 min (Fig. 1A). Trials involving the same class recorded from different runs (i.e., LR, FR, LF, LX, FX, and XR) were pooled (e.g., for “L” class “L” trials were pooled from LR, LF, and LX runs) forming a re-structured dataset. Thus, in the re-structured dataset four different classes (L, F, R, and X) were derived. The number of trials per class for a single session varied between 45 and 90.
The three ‘task vs. task’ classifiers (LR, FR, LF) were calibrated using the corresponding trials stored in the re-structured dataset. However, in runs when the NeuroSensi game was controlled with a ‘task vs. relax’ (TX) task, i.e., in runs where the character was controlled with LX, FX, or XR task pairs, the same TX decoder was used. The TX decoder was calibrated using T vs. X trials from the re-structured dataset where T trials comprised the L, F, and R pooled trials. To improve the cross-session stability of the calibrated BCI, the final dataset for BCI calibration was prepared by pooling re-structured datasets from multiple sessions acquired prior to the calibration.
Triad game for monitoring details of the multi-class classification
The Triad game (Fig. 1C, Additional file 4: Video S4, introduced in 2020) provides real-time continuous visual feedback from each of the four 2-class classifiers. The analogue output of the three ‘task vs. task’ classifiers (LR, FR, LF) are presented using a light blue ball on the three edges of a triangle. Furthermore, the linear combination of the LR, FR, and LF classifier output is presented with an additional coloured ball indicating the composite output of these three ‘task vs. task’ classifiers. The colour of the composite output indicator ball is assigned via the analogue output of the ‘task vs. relax’ (TX) classifier. The colour of the ball indicates whether the command is decoded as the task (green) or relaxed (dark blue) condition. The Triad game provides an opportunity for online monitoring of a combination of the three ‘task vs. task’ and concurrently the ‘task vs. relax’ classification. For example, in session 17 during 2020 the triad game was regularly used by the researcher as a monitoring tool that displayed real-time analogue output from all 2-class classifiers whilst the pilot played the BrainDriver game (described in the next subsection). Furthermore, in 2020, from session 19 onwards, at the beginning of each session the Triad game was also used by the pilot to practice controlling multiple 2-class classifiers in parallel, as a warm-up exercise before the first BrainDriver race practice in the session. However, as the Cybathlon event did not permit the use of add-ons during the competition, the pilot did not use the Triad game in parallel with the BrainDriver game. In the future, the Triad game could be used for acquiring data for BCI calibration. However, this was not the case in 2019 and 2020, when the Triad game was first introduced to the pilot.
BrainDriver game to familiarize the pilot for the race in the Cybathlon BCI event
After the pilot learned to control the BCI using the NeuroSensi (in 2019 and 2020) and Triad games (2020 only), the BrainDriver race was used in both years to practice control of the avatar—a virtual race vehicle (Fig. 1B, Additional file 2: Video S2 and Additional file 3: Video S3). The actual track of the BrainDriver race comprised four different zones. There are zones with left and right curves and straight zones with streetlights turned on or off. To maintain the maximal speed of the vehicle, the pilot must produce the correct race command using the 4-class BCI, e.g., left or right arm motor imagery for left or rights turns, feet imagery for “headlight” and relax for “no-control”. If an incorrect command is presented the vehicle is inhibited which decreases speed and increases race completion times, and moreover, presents obvious negative visual feedback to the pilot which enable learnings and error correction strategy development. The pilot was instructed to relax immediately after issuing a command to allow for ‘no control’, or as an alternative strategy to continue to maintain the motor imagery command. “The online BCI” section describes how the controller limits commands and assists in dealing with variation in control performance by the BCI and pilot.
Data acquisition
The EEG was recorded from 32 EEG channels (Fig. 2B) using a g.Nautilus Research active electrode wireless EEG system (g.Scarabeo) [22] with the EEG reference electrode positioned on the left earlobe. The EEG was high-pass filtered (> 0.1 Hz), notch filtered (48–52 Hz), and digitized (A/D resolution: 24 bits, sampling rate: 250 Hz). The ground electrode was positioned over the AFz electrode location according to the international 10/20 EEG standard (Fig. 2). Communication between the real-time BCI decoder module deployed in Simulink [23] (used for EEG data acquisition and online signal processing) and each of the three games (NeuroSensi, BrainDriver, and Triad) was via the ‘user datagram protocol’ (UDP).
Calibration of the two-class classification modules
The BCI framework included a filter-bank common spatial patterns (FBCSP) [24] and mutual information (MI) based feature selection [25], a well-established framework used in BCI applications that enable discrimination between imagined movements [26] performed with the left hand (L), feet (F), right hand (R), and relax (X) conditions [27]. The FBCSP-MI module, the core of the online BCI framework (Fig. 2A), was calibrated offline as described below.
EEG signal processing
The acquired EEG dataset (“Data acquisition” section) was band-pass filtered in four non-overlapped standard EEG bands (8–12 Hz (mu), 12–18 Hz (low beta), 18–28 Hz (high beta), and 28–40 Hz (low gamma)) using high-pass and low-pass finite impulse response (FIR) filter modules (band-pass attenuation 0 dB, band-stop attenuation 60 dB). The band-pass filtered EEG was downsampled from 250 to 125 Hz. Trial-relevant time intervals between − 2 s before and 8 s after the onset of the 2 s pause [i.e., − 4 s before and 6 s after the onset of task (described in “Experimental paradigms” section)], were epoched out from the filtered EEG dataset for 21 pre-selected EEG channels (indicated with black in Fig. 2B), and stored for spatial filtering. The epoched data using a 1 s to 2 s width classification window enabled comparison of the decoding accuracy (DA) obtained in the 0 to 2 s reference baseline interval (covering the pause period) and during the 2 s to 6 s task interval (after the pause period).
Spatial filtering
The common spatial patterns (CSP) method was used to create spatial filters that increase the separability between two classes by maximizing the variance of band-pass filtered EEG signals from one class, while minimizing their variance from the other classes [28]. The linear transformation matrix defined by CSP converts the pre-processed EEG signals into a new vector space defined by the CSP filters.
Feature extraction
The number of selected CSP filter pairs for each 2-class classifier for each frequency band was set to three. The time-varying log-variance of the CSP filtered EEG was calculated using a 1 s width sliding window, with a 40 ms time lag between two windows. Thus, the offset (end-point) of the 1 s sliding window was set to cover the time interval between − 1 s before and 8 s after the onset of the pause (covering a 1 s sliding width window, the 2 s pause, and 6 s task intervals).
Feature selection
The mutual information (MI) between features and associated target class using a quantized feature space was estimated [25] to identify a subset of features that maximize classification accuracy.
Two-class classification
A regularized linear discriminant analysis (RLDA) algorithm from the RCSP toolbox [28] was applied to classify the extracted features. Linear discriminant analysis (LDA) uses a class separator boundary in a linear hyperplane to separate data into two classes. The time-varying analogue output of the classifier, i.e., the time-varying signed distance (TSD), is the time-varying distance between the location of the classifier output and the class separation boundary in the LDA hyperplane. The class assigned to each feature vector depends on the polarity of the classifier output, determined by the relation between the location of the feature vector and the class separator boundary in the hyperplane [29]. The current TSD value is calculated as described in (1):
$${TSD}_{n,t}={w}^{F}{x}_{F,n,t}-{a}_{0}$$
(1)
where, \({x}_{F,n,t}\) is the features vector at time t in the \(n\)th trial, while \({w}^{F}\) and \({a}_{0}\) are the weights (slope) and bias of the discriminant hyperplane.
In a six-fold cross-validation analysis, the time-varying DA was calculated and compared for each of the four 2-class classifiers using the highest 6, 10, 14, and 18 MI ranked features. The number of highest ranked features that provided a classifier configuration with the highest DA peak in the event-related period of the task (in a 2.4 to 8 s interval of the trial, covering a 0.4 to 6 s interval from the onset of the task) was applied to the online BCI configuration, in the case of each 2-class classifier, separately. We highlight that an optimal number of features were applied to the online BCI which were selected based on which configuration obtained the highest MI rank during the calibration method. Furthermore, as each feature was derived via a CSP weighted linear combination of the band-pass filtered EEG signals, the information content of a feature is not limited to a single EEG channel is comprised of all channels which were applied in the FBCSP framework.
Topographical analysis
To identify frequency bands and cortical areas that provide the highest contribution to the peak DA, an analysis was performed using parameters of the calibrated CSP filters and the MI weights for each of the four 2-class classifiers, separately. For the time-varying frequency analysis, the mean values of MI weights were calculated in each analyzed frequency band, and time point, separately. The obtained results were plotted in the form of subject-specific heat maps, indicating the time-varying DA contribution of the frequency bands analyzed. The location of the source activity was plotted using the ‘standardized low resolution brain electromagnetic tomography’ (sLoreta) software package [30] for each 2-class classifier in each frequency band, separately, indicating cortical areas where features provided the highest contribution for calculating maximal DA.
Combining trials for different runs and sessions
The objective was to find an online BCI configuration that provides the highest DA with a high level of stability over sessions. Thus, the BCI was calibrated using different datasets that were pooled from different combinations of existing sessions. A cross-session DA analysis was performed for each BCI configuration, wherein the time-varying DA plots were compared using datasets excluded from calibration data. The BCI configuration was selected for the subsequent sessions based on a comparison of the cross-session time-varying DA plots, frequency maps, and topographical maps using the various BCI configurations and objectives described above (i.e., long term stability paired with a maximal level of DA).
The online BCI
The core module of the online BCI involved the same FBCSP MI based 2-class classification framework (Fig. 2A) for NeuroSensi, Triad, and BrainDriver race which are described in “Calibration of the two-class classification modules” section. However, the post-processing module was different for each of the three paradigms/games.
NeuroSensi game
The NeuroSensi game (Fig. 1A, Additional file 1: Video S1) uses only one of the four binary classifiers for controlling the character (i.e., LR, FR, LF, or TX). The baseline of the corresponding TSD signal was calibrated manually and set to zero at the beginning of each run using an offset value. The amplitude of the TSD signal, using a scaling factor, was corrected to a value that enabled the controlled character to move over the controllable area during the game. The corresponding TSD, after the baseline correction, was downsampled to 25 Hz and sent by UDP to the NeuroSensi game. The avatar in the NeuroSensi game was controlled continuously using the post-processed analogue TSD control signal. Jitters in TSD signal were smoothed in the Unity game engine using a linear interpolation method based “lerp” function that smooths the transition between two values over time [31].
Triad game
As the Triad game (Fig. 1C, Additional file 4: Video S4) was controlled by the TSD output of each of the four 2-class classifiers, the baseline of each of the four TSD signals was corrected, separately, as described above for the NeuroSensi game. In addition, a smoothing filter option was applied to the baseline corrected TSD calculating the moving average within a 1 s window. The post-processed TSD was downsampled to 25 Hz and sent by UDP to the Triad game. Similarly to the NeuroSensi game, the continuous movement of the game avatars in the Triad game (i.e., one ball on each edge of the triangle plus one more ball in the middle) was smoothed by the Unity game engine using the “lerp” function [31].
Cybathlon race: BrainDriver
The avatar in the BrainDiver race (Fig. 1B, Additional file 2: Video S2 and Additional file 3: Video S3) was controlled by the online BCI framework presented in Fig. 2. The online BCI, in addition to the FBCSP-MI and TSD baseline correction modules (discussed above and presented in Fig. 2A), involves a control command decoder module composed of a multi-class decoder, a stability delay timer, a dead-band control module (Fig. 2C) and game control command translator module (Fig. 2D) followed by a UDP unit for sending commands to the BrainDriver platform (Fig. 2E).
The output of the multi-class decoder relies on the baseline-corrected outputs of the four binary (LR, FR, LF, and TX) classifiers. If the output polarity of two of the three ‘task vs. task’ classifiers (LR, FR, LF) are not conflicting and the TX classifier is indicating a task condition (“T”) (i.e., the pilot is not relaxed), the label of the decoded task is forwarded to the next module for a stability check. For example, in Fig. 2C both LR and LF classifiers output indicate the same (“L”) result and the TX classifier indicates that there is an ongoing task (“T”). Therefore, in this example, the decoded (“L”) command passes through on Command Control Gate 1.
To filter out transient responses, the decoded (“L”) command passes through Command Control Gate 2 only if the decoded (“L”) command is maintained in the same condition for a predefined (300 ms) period. If this stability check is matched, the decoded (“L”) command is translated with the game control command translator module to the game control command as shown in Fig. 2D. Finally, the game control command is sent by UDP to the BrainDriver game. An example of a track section and corresponding control commands are illustrated in Fig. 2E (details of the BrainDriver game in “Experimental paradigms” section).
To provide the opportunity for the pilot to reach a relaxed condition before the next command is decoded and to ensure sudden changes in classifier conditions do not interrupt a correct command issued to the vehicle, a dead-band system is activated. The dead-band control module involves a dead-band (DB) timer and a dead-band-break (DBB) timer module. Once a command is sent to the BrainDriver game, the DB timer countdown is activated, which blocks any new commands during the DB period. However, the DBB timer allows the pilot to correct an incorrect command by breaking the dead-band if the TX classifier after a (e.g., 3 s) DBB period detects that a command is being issued for a sufficiently long (e.g., 1 s) period whilst the dead-band is active.
For example, if the length of race zone is approximately 6–8 s and if the intended command is issued just as the avatar reaches a zone, the dead-band will ensure the avatar maintains the associated control for that entire zone, thus maximizing speed. However, if an unintended command is issued at the start of a zone the pilot can attempt to correct it by attempting to issue and produce another command for more than the predefined (e.g., 1 s) period. In such a case a portion of the speedup from that zone may be gained. The dual scenario here improves stability of control and makes the assumption that good commands are more frequent.
To find an optimal configuration that supports the pilot’s control ability maximally, the actual value of the dead-band and dead-band-break parameters were adjusted manually over sessions and runs during the training period. The dead-band was selected in a range between 2 and 8 s, and the dead-band-break timer was selected in a range between 2 and 4 s.
Statistical analysis
To evaluate differences in race completion times achieved during different periods of the long-term BCI training, paired-sample t-tests were performed on race times achieved in ten races for each period compared (see “Results” section).
To assess the potential causes of fluctuation in performance throughout training and during race days, logarithmic magnitude of power spectral density (PSD) located bilaterally over sensorimotor areas (C3, and C4) and centrally (Cz) were evaluated on different groups of races/sessions where race times differed. As the EEG dataset on the race day in 2020 was not stored due to an oversight, this analysis was performed for the dataset acquired in 2019 only. A three-way analysis of variance (ANOVA) was used to assess differences in 2019 using the following factors and levels: (1) session group (level 1: races from sessions 15 and 16, level 2: from 17 and 18, and level 3: from 19 and 20); (2) electrode site (level 1: C3, level 2: Cz, and level 3: C4); and (3) frequency (levels 1 to 15: frequencies ranging from 12 to 40 Hz in steps of 2 Hz (i.e., level 1: 12 Hz, level 2: 14 Hz, …, level 15: 40 Hz). The ANOVA was two-tailed, with a 95% confidence interval, and was conducted using the Statistical Package for Social Sciences (IBM SPSS Statistics 27.0). A Greenhouse–Geisser correction was applied when sphericity was violated and a Bonferroni correction was applied to pairwise comparisons to control for multiple comparisons.