A binary method for simple and accurate two-dimensional cursor control from EEG with minimal subject training

Paradigm design

To provide robust single-channel control, we implemented a synchronous binary approach to 2-D cursor control. Synchronous control uses a pre-defined time window for each user response so that the computer does not need to determine when a user response occurs, but only into which class each user response falls. Binary control refers to a situation under which each response must be classified into one of only two classes, as contrasted with control where a response can be classified into one of a greater number of classes or ignored altogether. Synchronous binary classification is the simplest possible classification using EEG, and we hypothesized that this simplicity would yield high cursor movement accuracy.

The binary approach works as follows. The cursor moves in discrete steps, and each step is in one of four directions (up, down, left, right) as selected by the user through his or her EEG signal. To select a direction, the user effectively answers "yes" or "no" two times in a row, performing continuous right-hand movement to answer "yes," or abstaining from such movement to answer "no." The user has a short time to give each answer, during which the resultant ERD causes a power change in the EEG signal. The computer program measures the EEG power from a single optimum channel and frequency band over the pre-defined time window of the subject's answer. If the power is above a certain threshold the software algorithm interprets the answer as a "no," and if the power is below the threshold the software algorithm interprets the answer as a "yes." The program determines the threshold value prior to the user's first game by presenting a series of "yes" or "no" prompts that the user obeys directly, and using the associated power measurements from the appropriate location/band to optimize classification accuracy. This threshold determination does not have to be repeated before each game.

Under the 2-D cursor control paradigm, a cursor moves among squares of a grid towards a target while avoiding a trap. Sequential screen shots of one cursor move are shown in Figure 1. The subject is presented with the game grid, and is allowed to blink, shift gaze, and strategize for the next move. After presentation, everything but the cursor and four adjacent squares are blacked out, and a prompt is presented in each of the possible movement directions. For all but one of the studies presented here, EEG signals were recorded with the subject making hand movements. One example is presented in which control was performed with only motor imagery. When movements are used, the subject initiates control by making continuous right hand movement. The prompts remain cyan for a short time to allow the subject to interpret the prompt in the desired movement direction, and then the prompts turn green. While the prompts are green, the subject executes the desired task. To select a direction showing a "yes" prompt, the subject continues the right hand movement. To select a direction showing a "no" prompt, the subject ceases the movement and remains motionless throughout the green prompt. In either case, the subject must fixate on the prompt, remain relaxed, and not blink to avoid artifacts while the prompt is green. Once the program determines the first response (first bit), it eliminates the two rejected directions, and repeats the prompting process. After the second response (second bit), the game grid again becomes visible, and the cursor moves to the new position. The entire process for one (two-bit) cursor move takes about 15 s. When the game is played without hand movements (as in one of our supplementary tests), the subject is asked instead to imagine a movement. When playing the game using motor imagery, the threshold-setting and control tasks are performed as normal.

While on any given movement the cursor moves in only one direction, the control is two-dimensional rather than one-dimensional because the direction of each movement can be any one of four choices in two dimensions. This is analogous to the two-dimensional control achieved by the P-300 detection method of Piccione et al. [15], which also uses a series of single cursor movements, each in one of four directions. Whereas Piccione's method relies on sequential emphasis of four stimuli to obtain the two bits of information required for each cursor move, our method obtains the first and second bits sequentially through two user selections, which together uniquely identify both the dimension and direction of each cursor move. This two-dimensional control is distinct from one-dimensional control, wherein the computer restricts the dimension of cursor movement, and the user is free to control only the direction of the movement.

The cursor control game incorporates several additional features. These include automatic recordkeeping, game scoring to hold player interest, and an optional adaptive threshold feature (which was used only for Subject F, as discussed below). Furthermore, the program avoids superfluous prompts; if the cursor is at an edge of the grid and the first prompt can uniquely determine cursor movement direction, then only one prompt is provided.

Study procedures

A Neuroscan Synamp 1 amplifier (Neuroscan Inc., El Paso, TX, USA) amplified the EEG signal from 29 electrodes. The 29 electrodes sampled at 250 Hz from FP1, F3, F7, C3A, C1, C3, C5, T3, C3P, P3, T5, O1, FP2, F4, F8, C4A, C2, C4, C6, T4, C4P, P4, T6, O2, FZ, CZA, CZ, PZA, and PZ in an elastic cap (Electro-Cap International, Inc., Eaton, OH, USA). The recordings from a maximum of five of these 29 electrodes were used for each subject's cursor control, although all 29 electrodes were used once per subject for the initial channel/bin optimization step, which did not need to be repeated thereafter. A Hewlett-Packard workstation converted the amplified analog signal to a digital signal.

We determined the optimum single electrode location and frequency band for control for each subject from offline analysis of EEG recordings. First, each subject performed the threshold-setting task (although no threshold was set at this point) wherein single predetermined yes/no prompts were presented sequentially. This threshold-setting task consisted of 30 prompts, composed of 15 "yes" and 15 "no" prompts randomly interspersed. An offline feature analysis of the resultant EEG recordings was performed to identify the location and band for which power measurements provided the greatest yes/no class separability. Once the optimum location and band were identified, these were used for all subsequent testing with the subject. Thus, this optimization step, which required a relatively large number of electrodes (all 29 were analyzed), only needed to be performed once per subject, and then a reduced number of electrodes could be used (five electrodes if using Laplacian derivation, or one electrode if not).

Once the optimum location and band were identified, each subject repeated the threshold-setting task, and the power in the optimum location/band was again computed (now using the reduced number of electrodes). These measurements were used to set an optimum threshold. For these experiments, the threshold-setting task again consisted of 30 prompts, composed of 15 "yes" and 15 "no" prompts randomly interspersed. Completion of the entire threshold-setting task took less than 5 minutes. The threshold determined from this task was used for the subsequent 2-D cursor control task. Each subject repeated the threshold-setting task multiple times to practice his or her control strategy. However, each time the task was repeated, the program discarded all previously obtained data. Thus, the threshold set by the program was based solely on the 30 prompts from the subject's most recent performance of the threshold-setting task.

Finally, each subject performed the 2-D cursor control task. The program interpreted intended cursor movement direction online in real-time by comparing measured powers to the optimum threshold. The program also tagged the EEG recordings with the interpreted yes/no answers. An electromyography (EMG) channel recorded right hand movement during the cursor control task. The EMG signal was sampled at 250 Hz from a bipolar surface electrode located over each subject's right wrist extensor muscles. Visual inspection of the EMG recording was used to quantify the control accuracy through post-hoc offline analysis.

Computational method

For all prompts in the threshold-setting and cursor control tasks, the time over which the subject gave each yes or no answer had duration 2 s. Band power measurements were computed for the final 1.5 s of this time window only, to allow for subject response time. Power was determined using the Welch estimation method with FFT length (nonequispaced fast Fourier transform) of 64 and a Hamming window with 50% overlap [17]. The sampling rate of this study was 250 Hz, and the frequency resolution was about 4 Hz. For all measurements, the EEG signal was referenced using Laplacian derivation to reduce error. This means that the EEG signal was referenced from each electrode to the average of the potentials from the nearest four orthogonal electrodes. For example, the program referenced the C3 channel to the average of C1, C3A, C5, and C3P, each of which was about 3 cm from C3, and calculated band power on C3 for the referenced signal.

To determine the optimum spatial location and frequency band for discrimination, we conducted a feature analysis by calculating Bhattacharyya distances from power measurements. Frequency bands were 4 Hz wide, corresponding to the 4 Hz resolution of the power measurement. We measured power using the Welch method for each yes/no response, for each EEG channel. Then, for each channel/bin pair, we calculated a Bhattacharyya distance based on the power measurements for all of the responses from both the "yes" and "no" classes. Higher Bhattacharyya distances corresponded to better yes/no class separability, and identified the more effective channels and frequency bands for control. We calculated each Bhattacharyya distance according to (1), where M _i and Σ _i are the mean vector and covariance matrix of class i ( = 1,2), respectively [18]. As we measured the Bhattacharyya distance for each channel and frequency bin, M _i is a scalar.

(1)

After we identified the optimum location and frequency band (only done once per subject), we used these in our threshold-setting program, which no longer needed all EEG channels. This program measured power in the optimum location/band while the subject performed the threshold-setting task. After the task was complete, a receiver operating characteristics (ROC) curve was generated by determining the true positive and false positive fractions that would result from various values of threshold. Here, "true positive fraction" refers to the fraction of intended "yes" answers that the program would interpret as "yes" answers given the particular threshold value (this is equivalent to sensitivity). "False positive fraction" is the fraction of intended "no" answers that the program would interpret as "yes" answers (this is equivalent to 1 – specificity). The threshold-setting program chose the optimal threshold as that which minimized the distance defined in (2).

(2)

Additional file 2: Overview summarizes the most important steps of the binary control computational method. The file shows examples of recorded EEG signals, and indicates how these signals can be classified based on their power spectral densities into "yes" and "no" classes. The file demonstrates the correspondence between higher Bhattacharyya distances and better class separability, and shows how choosing the optimum location/band can yield a high-quality ROC curve, from which a threshold can be set and subsequently used to achieve good control in the 2-D cursor control task.

To quantitatively assess the accuracy of the cursor control, we analyzed the recordings from the control task offline following each subject's session. We compared our program's yes/no interpretations with the recorded right wrist EMG trace to explicitly determine whether each classification and cursor move was correct.

For motor imagery, no EMG signal was available for comparison, so we assessed the accuracy of yes/no classification from one of the threshold-setting task recordings. We divided the prompts into a training set consisting of the first 7 "yes" and first 7 "no" responses, and a testing set consisting of the last 8 "yes" and last 8 "no" responses. We used the training set to calculate an optimum threshold, which we then applied to the testing set to classify its responses. Because we knew the correct classifications of the responses, we were able to quantify the classification accuracy. We also used the entire threshold-setting task to set an optimum threshold with which the subject played the cursor control game. We then asked the subject to qualitatively evaluate her control after playing the game.

Subjects and data acquisition

We tested the paradigm with four healthy subjects using hand movement. Subjects included three females and one male, with ages ranging from 24–55 years. Subject A was female, age 53 years. Subject B was female, age 55 years. Subject C was female, age 24 years. Subject D was male, age 32 years.

We also carried out several supplementary tests. Subject B performed our paradigm using motor imagery. This followed Subject B's session using real movement. Subject E, a primary lateral sclerosis (PLS) patient, performed our paradigm using hand movement. PLS is a motor neuron disease, the symptoms of which include slowly progressive spasticity of unknown cause without clinical signs of lower motor neuron loss. Pathological studies show degeneration of the corticospinal tracts. Subject E was female, age 58 years, with the disease for 11 years. She was identified as a PLS-A patient with loss of motor-evoked potentials by transcranial magnetic stimulation, and her right finger tapping rate was 3.6 taps/s, which was significantly lower than healthy controls of 5.8 taps/s [19]. Subject F performed our paradigm using hand movement, but with no Laplacian derivation referencing of the EEG channels. Subject F was male, age 23 years. We also performed a post-hoc offline analysis of data from Subject A with the Laplacian derivation removed.

None of the subjects had previous BCI experience. All subjects were right-handed according to the Edinburgh inventory [20]. All subjects gave written informed consent for the protocol, which was approved by the institutional review board.

We accomplished the real-time EEG data acquisition and processing using a Matlab-based self-developed hardware and software system. The self-developed Matlab scripts accessed the digital signal and performed the power spectral estimation. Finally, the scripts decoded the power spectral signal to drive the cursor movement.

Feature analysis

Figure 2 shows channel-frequency and head topography plots of Bhattacharyya distances for Subjects A-E using hand movement. For all subjects, including the PLS patient (Subject E), the largest Bhattacharyya distances were localized over the left sensorimotor cortex, contralateral to the hand being moved, and were located in the beta frequency band, consistent with expectations about the sensorimotor rhythm. To attempt accurate control without the need for channel/bin calibration on an individual-by-individual basis, we chose the C3 electrode and 20–24 Hz frequency band as the optimum channel/bin for Subjects A, B, C, and E, since none of their Bhattacharyya plots differed extremely from this pattern.

For Subject D, we modified our threshold-setting program to automatically choose the best channel/bin as that which yielded the smallest minimum value of the distance defined by (2). In this way, we effectively automated the feature analysis by integrating it into the threshold-setting program, eliminating the need for the calculation of Bhattacharrya distances, but requiring that all 29 electrodes be used during the threshold-setting task. Our modified program chose the C1 electrode (channel 5) and the 20–24 Hz frequency bin for optimum control for Subject D. This selection is clearly consistent with the subject's Bhattacharyya plots.

Binary 2-D cursor control with hand movement

For all four healthy subjects using hand movement, the threshold-setting task robustly classified the "yes" and "no" responses. Figure 3 shows the ROC curves generated by the threshold-setting task that immediately preceded each subject's first session of cursor control. For all curves, the optimum threshold clearly yielded a low value of the distance defined in (2).

Overall, the average of the subjects' cursor movement accuracies was 86.1%, with a standard deviation of 9.8%. This control was significantly greater than chance (p = 0.0015).

Supplementary tests

For some of the subjects, additional test were performed to help determine the robustness of the single channel system when no or weak movements were used and when the Laplacian referencing was removed.

Subject B: 2-D control with motor imagery (no hand movement)

To gain a sense of whether our method would be effective for individuals who were unable to make hand movements, we performed a test of the paradigm for motor imagery with a single subject. We asked Subject B to repeat the threshold-setting and cursor control tasks using motor imagery immediately following her performance of the cursor control task using real movement.

Figure 4 shows Bhattacharyya distance plots for Subject B using motor imagery. As expected, it was difficult to confidently determine an optimum channel/bin, so we used the same channel/bin as with real movement for the sake of parsimony.

As described above, because no EMG signal was available with which to compare classification, accuracy with motor imagery was quantified using only the threshold-setting task divided into training and testing sets. Figure 5(a) shows the ROC curve from threshold optimization using the training set. Using this threshold, the classification accuracy for the testing set was as follows: 50.0% true positive percentage (chance = 50.0%), 87.5% true negative percentage (chance = 50.0%). Because there are no cursor moves in the threshold-setting task, no correct cursor move percentage could be calculated. However, this value may be estimated by assuming that intended yes and no answers are equally likely, and that all intended moves are equally likely. Under these assumptions, the average classification accuracy is the average of the true positive and true negative fractions (true negative fraction is the fraction of intended "no" answers correctly classified as "no", which is equivalent to specificity). The average number of bits per cursor move for the 5-by-5 grid is 1.68. The estimated correct cursor movement percentage CM% is then given by (3).

(3)

From (3), the estimated cursor movement accuracy for Subject B using motor imagery was 53.3% (chance = 31.2%).

Figure 5(b) shows the ROC curve from the threshold optimization based on the entire threshold-setting task. Using the threshold set from this ROC curve, the subject played the cursor control game. Although we could not quantify cursor movement accuracy, we asked the subject to qualitatively evaluate her control. She reported that she felt she had some control over the cursor movement, although she found the task frustrating.

Subjects A and F: 2-D control with a single electrode and no referencing

To test if good single-channel control could be retained without Laplacian derivation referencing, we performed an offline analysis of the data from Subject A, using the recording from the threshold-setting task to set a threshold and the recording from the cursor control task to determine classification accuracy. As before, the electrode and frequency band were C3 and 20–24 Hz. The only change was eliminating the Laplacian derivation referencing, and using the raw signal from C3.

From this analysis, we calculated the following results. The total number of yes/no answers was 104, consisting of 34 true positives, 27 false positives, 36 true negatives, and 7 false negatives. This corresponds to a true positive percentage of 82.9% and a true negative percentage of 57.1% (chance = 50% for each). The overall correct cursor move percentage was 49.1% (chance = 31.2%).

We also performed one online test of our paradigm with true single-channel control. For Subject F, a healthy subject using real movement, we did not use Laplacian derivation referencing. Figure 6(a) shows Bhattacharyya distance plots for Subject F. We selected the 12–16 Hz frequency bin and C3 electrode for control. Using this channel/bin, the subject performed the threshold-setting and cursor control tasks. Rather than beginning a hand movement for each response and ceasing the movement to answer "no," the subject chose to perform hand movement to answer "yes," and to abstain from such movement to answer "no." The ROC curve from the threshold-setting task, using 20 prompts rather than 30, is shown in Figure 6(b). This curve showed good quality classification, with a low value of the distance defined in (2). After the threshold-setting task, the subject proceeded to play the cursor control game with an adaptive threshold feature enabled. The adaptive threshold feature allowed the program to learn as the subject played the cursor control game, by recalculating the ROC curve and optimum threshold after every yes/no answer for which there was a unique good choice. The resultant ROC curve is shown in Figure 6(c). Finally, the subject performed the cursor control task with the adaptive threshold disabled. The subject demonstrated good cursor control, with a true positive percentage of 83.3%, a true negative percentage of 89.8%, and a correct move percentage of 77.4%.

Binary 2-D cursor control with hand movement

Overall, control was good with healthy subjects using real hand movement. Accuracy for Subject B might have been even higher had the optimum channel/bin been customized for this subject, whose Bhattacharrya plots indicated better classification at slightly higher frequencies around 28–32 Hz, and whose control was less accurate than that of the other subjects. The very good results from Subject D, for whom the optimum channel/bin was customized, provide further support for the benefit of such individualized calibration. However, the downside of such calibration is that it requires all 29 EEG channels to initially be attached to the subject, even though most of the channels ultimately may not be used for control.

The high control accuracy seen with all subjects demonstrates that the binary method with hand movement is effective and robust. The method requires remarkably little user training. Each subject practiced with the threshold-setting task prior to performing the cursor control task, and Subject C also practiced briefly with the cursor control task before achieving the results given in Table 1. However, all four naïve subjects achieved the control accuracies reported in Table 1 within the first 2 hours of their experience with the paradigm.

These results are consistent with subsequent experiments using healthy subjects with real movement [21].

Supplementary tests

Comparative accuracy, speed, and ease of control

Like the method of Piccione et al. [15], our system applies a known approach to obtaining information from EEG (in our case, band power measurement; in Piccione's, evoked potential detection) in a novel paradigm to address the challenge of two-dimensional cursor control. Unlike Piccione's 2-D control method, our 2-D control method classifies each individual user response into one of two possible classes (yes or no), with two user responses per cursor step. Piccione's method classifies each user response into one of four classes (up, down, left, or right), with one user response per cursor step. At chance level, our method would equal Piccione's in accuracy for a typical cursor move. Chance level for a correct cursor movement under Piccione's method is equal to chance level for the correct classification of an individual user response, or 1/4. Chance level for a typical correct cursor movement under our method is equal to the product of the chance levels for each of the two responses being classified correctly, or (1/2)² = 1/4. We originally expected the band power signal upon which response classification is based in our paradigm to be more robust than the P-300 signal upon which response classification is based in Piccione's paradigm. Therefore, we originally hypothesized that our paradigm would yield more accurate control than Piccione's paradigm, while still providing equivalent stepwise 2-D cursor control at a roughly equivalent speed.

From our results, the average accuracy of our system was 86.1%, with one cursor move occurring approximately once every 15 s (≈ 8 bits/min). Comparatively, Piccione's system had an average accuracy of 76.2% and a bit rate of 7.59 bits/min with healthy trained subjects [15]. Based on these results, our method appears to have a higher accuracy at a slightly higher speed than Piccione's method, while providing 2-D cursor control of an equivalent nature (occurring in sequential steps as guided by user selection of a dimension and direction at each step).

The bit rate of our system, while at least comparable to Piccione's and other accepted BCIs, might still be improved through subject training to build response proficiency, allowing for shorter pauses between stages of the paradigm. Alternatively, if both bits needed for a cursor move could be collected simultaneously (e.g., by two channels – one over the sensorimotor cortex of each hemisphere), bit rate might be increased. Geng et al. describe one promising "parallel" 4-class BCI that uses two binary classifiers obtained simultaneously during hand and foot movement, although the computational method used is more complex than our threshold-based classification [16]. Further evaluation is needed to assess whether the bit rate of our system can be improved without sacrificing accuracy.

When considering the speed and efficiency of our method, it should be noted how our cursor control method is conceptually distinct from that employed in EEG-based decision trees. In such trees, real-time analysis of band power controls one dimension of cursor movement while cursor movement in the other dimension proceeds at a constant rate. This continues until the cursor reaches the edge of the screen, at which point one of multiple targets is achieved, and a single selection is thereby made. If there are two available targets (the scenario, closest to that of our paradigm, in which accuracy is expected to be highest), this entire process constitutes a single binary selection, ultimately yielding only one bit of information. In our paradigm, two complete binary selections are used to determine both the dimension and direction of each cursor movement, and only then does the cursor move one space in the determined dimension/direction. Thus, the information obtained by our paradigm for each movement of the cursor by one space is twice the amount of information that would be obtained from conducting an entire two-choice decision tree cursor control paradigm. However, our paradigm may take significantly less time per cursor move than even a single typical decision tree selection. This should be considered when judging the speed of our method.

Also noteworthy is that our method provides both the higher accuracy associated with binary selections and a straightforward means of correcting cursor movement errors: if the cursor is moved in an undesired direction, the user may move it back to the previous position on the subsequent move, or may continue movement toward the desired target using an alternate path. Contrast this with error correction in a decision tree selection, which requires a separate "undo" option in addition to the at least two other options from which the user is expected to select. Because of this "undo" option, decision trees that allow error correction must classify each selection into one of at least three possible classes, resulting in lower selection accuracy than could be achieved from a binary classification.

Comparison of our system's accuracy and speed with contemporary 2-D EEG BCI systems other than Piccione's is less straightforward, due to necessarily diverse methods of evaluating accuracy and speed. The two-band system of Wolpaw et al. [13] allows subjects to take up to 10 s to attain one target of eight. Because the two dimensions of movement are controlled simultaneously, success was measured not by the accuracy of each cursor step, but rather by whether the subject could attain the target by any path within the 10 s time limit. The average success rate of four subjects was 82%, with an average successful target acquisition time of 2.8 s. The first target attained by each subject was strongly correlated with the intended target (p < 0.001). In the system of Trejo et al. [7], which uses SSVEP control, the direction of cursor motion is recalculated every 250 ms, with control lags of 1–5 s. Because during each 250 ms movement period, movement is in only one of the four directions (left, right, up, or down), Trejo's 2-D control method is similar to both our method and that of Piccione in that it does not provide simultaneous control of the two dimensions of movement. However, because each movement period is only 250 ms, the method gives the illusion of simultaneous control of the two dimensions. Trejo's method yielded an average movement accuracy across three subjects of 76%.

While the binary control method of our system is less naturalistic than is ideal, all EEG-based control systems suffer from being somewhat awkward to use, and many extant EEG BCI systems are, like ours, externally paced. Our synchronous approach may be less naturalistic than is self-paced control, but the approach allows classification accuracy to be high despite our simple computational method. (Consider that determining when a subject is answering is a significant challenge in self-paced systems; the best asynchronous EEG switch developed by the Neil Square Society as of 2005 had a 73% mean activation rate and a 2% mean false positive error rate across four subjects [22].) Scherer et al. described a self-paced EEG BCI for 2-D navigation in a virtual environment [23]. While their system offers the potential of naturalistic pacing, it still features somewhat unnatural control methods (hand, foot, and tongue motor imagery), as well as variable accuracy and high computational demand. Thus, self-pacing is not the only obstacle to naturalistic control. We believe that our system is not unacceptably less natural to use when measured against the state of the art in EEG BCI.

We also believe that our system may be less fatiguing than are some other EEG BCI systems, particularly those that rely on visually evoked potentials. However, further testing is required to determine this conclusively.