Open Access

EEG oscillatory patterns and classification of sequential compound limb motor imagery

Contributed equally
Journal of NeuroEngineering and Rehabilitation201613:11

https://doi.org/10.1186/s12984-016-0119-8

Received: 20 January 2015

Accepted: 21 January 2016

Published: 28 January 2016

Abstract

Background

A number of studies have been done on movement imagination of motor sequences with a single limb. However, brain oscillatory patterns induced by movement imagination of motor sequences involving multiple limbs have not been reported in recent years. The goal of the present study was to verify the feasibility of application of motor sequences involving multiple limbs to brain-computer interface (BCI) systems based on motor imagery (MI). The changes of EEG patterns and the inter-influence between movements associated with the imagination of motor sequences were also investigated.

Methods

The experiment, where 12 healthy subjects participated, involved one motor sequence with a single limb and three kinds of motor sequences with two or three limbs. The activity involved mental simulation, imagining playing drums with two conditions (60 and 30 beats per minute for the first and second conditions, respectively).

Results

Movement imagination of different limbs in the sequence contributed to time-variant event-related desynchronization (ERD) patterns within both mu and beta rhythms, which was more obvious for the second condition compared with the first condition. The ERD values of left/right hand imagery with prior hand imagery were significantly larger than those with prior foot imagery, while the phase locking values (PLVs) between central electrodes and the mesial frontocentral electrode of non-initial movement were significantly larger than those of the initial movement during imagination of motor sequences for both conditions. Classification results showed that the power spectral density (PSD) based method outperformed the multi-class common spatial patterns (multi-CSP) based method: The highest accuracies were 82.86 % and 91.43 %, and the mean values were 65 % and 74.14 % for the first and second conditions, respectively.

Conclusions

This work implies that motor sequences involving multiple limbs can be utilized to build a multimodal classification paradigm in MI-based BCI systems, and that prior movement imagination can result in the changes of neural activities in motor areas during subsequent movement imagination in the process of limb switching.

Keywords

Sequential compound limb motor imageryEvent-related desynchronizationPhase locking valueBrain-computer interfaceSupport vector machine

Background

Motor imagery (MI), defined as the mental rehearsal of a motor act without any overt motor output, can modify the neuronal activity in the primary sensorimotor areas in a manner which is quite similar to motor execution [1, 2]. MI-based brain-computer interface (BCI) systems could translate the subjective movement consciousness of the users without any inducing factors such as visual stimulus producing steady-state visual evoked potential (SSVEP) or event-related potential (ERP). This provides an alternative communication and control channel for patients with limited motor function to improve the quality of their lives [3]. Therefore, MI-based BCI systems play an important role in the field of rehabilitation engineering.

Since Jasper and Penfield’s discovery of brain oscillatory activity induced by MI [4], MI-based BCI systems have gone through several decades of development. However, a major problem is the limitation of control commands in contrast to other BCI paradigms. Most research on MI-based BCI systems have focused on analyzing EEG rhythms induced by simple limb motor imagery involving a single part of the limbs such as the left hand, the right hand or the foot [5, 6]. Pfurtscheller et al. studied the reactivity of EEG rhythms in association with the imagination of three kinds of limb movements (right hand, left hand, and foot) with the addition of tongue movement [7]. To implement a two-dimensional cursor movement between arbitrary positions, P300 potential was combined with MI (left and right hands) for controlling the horizontal and vertical movements of the cursor, respectively [8]. Additionally, MI was applied as a brain switch in a hybrid BCI system by detecting the post imagery beta event-related synchronization (ERS) of foot imagery to turn a four-step electrically driven hand orthosis with two flickering lights on and off to reduce the false positive rate during the resting period [9].

On the other hand, a significant amount of work have been reported on movement imagination of motor sequences with a single limb such as sequential finger-to-thumb opposition tasks [1012], sequential dorsiflexion, and plantarflexion of the foot [13]. Lafleur et al. used positron emission tomography (PET) to measure and compare the dynamic changes in cerebral activity during the execution and imagination of sequential dorsiflexion and plantarflexion of the left ankle before and after practice [14]. A similar experiment was extended from the lower to the upper extremities to investigate the cerebral activations underlying the preparation and the execution periods associated with the actual and imagined movements of externally paced sequence of finger key press using EEG [15]. Moreover, Kranczioch et al. measured movement-related potentials to compare the functional similarity of the prediction mechanism during overt and covert action preparations of simple or complex sequential finger movements [16]. However, brain oscillatory patterns induced by sequential compound limb motor imagery (SCL-MI), particularly movement imaginations of motor sequences involving multiple limbs, have not been reported in recent years.

Limited choices of limb movement lead to the limitation of control commands in MI-based BCI systems. In addition, EEG patterns induced by simple limb MI are similar to those of sequential movement imagination of the same limb, and the latter case would activate the same functional area during sequential movement imagination of a single limb. In contrast with simple limb MI or sequential movement imagination of a single limb, different limbs are involved in SCL-MI, which can activate the neurons’ oscillation in different functional areas of the cerebral cortex within specific time intervals sequentially, and at the same time, is more in line with the normal behavior of humans. Furthermore, SCL-MI can achieve the output of multiple control instructions in MI-based BCI systems compared with the limited commands of simple limb MI.

In this study, we aimed to verify the feasibility of the application of SCL-MI to MI-based BCI systems. Additionally, the changes of EEG patterns and the inter-influence between movements in the motor sequences were also investigated. To this end, four kinds of sequential movement imagination were designed to simulate the movement of playing drums, involving one sequential movement imagination of a single limb and three kinds of SCL-MI based on the cooperation of two or three limbs. To test the separability of these four MI tasks, a multi-class common spatial patterns (multi-CSP) algorithm (successfully used in [17]) and power spectral density (PSD) were employed in feature extraction. Event-related spectral perturbation (ERSP) and phase synchronization analysis were applied to study the induced brain oscillatory patterns during SCL-MI.

Methods

Experimental procedure

The four motor sequences consist of three sub-movements. One sequential movement imagination of a single limb is [right hand]–[right hand]–[right hand] (RRR) and it represents imagining playing drums using the right hand according to the sequence. The three kinds of SCL-MI are [right hand]–[left hand]–[right hand] (RLR), [left hand]–[right foot]–[left hand] (LFL), and [right foot]–[right hand]–[left hand] (FRL) and they represent imagining playing drums using the corresponding limbs according to the sequence.

Twelve right-handed healthy subjects (7 females and 5 males, 21–26 years old) participated in this experiment. The subjects sat in a chair at a 1-meter distance in front of a computer screen. There were two task conditions. The experimental paradigm of the first condition is shown as Fig. 1. Each trial (8 s) began with a white circle that appeared at the center of the monitor for 1 s. After the disappearance of the white circle, a character indication (‘[right hand]–[right hand]–[right hand]’, ‘[right hand]–[left hand]–[right hand]’, etc.) was presented on the screen for 2 s. During this phase, the subjects were instructed to mentally prepare and remember the task, while withholding any imagination. At the 3rd second, the cue disappeared and the screen turned black for 3 s. The participants were asked to immediately perform the indicated task once the screen turned black. MI tasks were performed kinesthetically rather than visually while avoiding any motion during imagination. At the 6th second, ‘Rest’ was presented for 2 s before the next trial. During the whole experiment, the subjects were asked to listen to the drums starting simultaneously with the appearance of the white circle (60 beats per minute), that is, there was a drum sound every second to maintain the self-rhythm of the imagination. Figure 2 shows the paradigm of the second condition (12 s), which is similar to the paradigm of the first condition, and accompanied by the drums (30 beats per minute) while the imagination phase lasts for 6 s. Three sub-movements in each motor sequence were performed sequentially, while a single sub-movement was performed once within a 1-s or 2-s time interval.
Fig. 1

Experimental paradigm of the first condition. A white circle indicates the beginning of each trial, followed by a cue indicating the type of MI task. The subjects performed the indicated MI task for 3 s, and then rested for 2 s before the next trial. Two loudspeakers and the notes indicate the drum sounds during the whole experiment

Fig. 2

Experimental paradigm of the second condition. A white circle indicates the beginning of each trial, followed by a cue indicating the type of MI task. The subjects performed the indicated MI task for 6 s, and then rested for 2 s before the next trial. Two loudspeakers and the notes indicate the drum sounds during the whole experiment

The experiments for both conditions were divided into seven sections, consisting of 40 trials each for the four kinds of sequential movement imagination (10 trials for each task in 1 section). The four MI tasks were presented in a randomized order. The intersection break lasted for about 5 to 10 min. In total, there were 280 trials (70 trials per class) in the study dataset.

Prior to recording, subjects were required to take three sessions for the training, consisting of five sections each (once a day) for both conditions for their familiarization and proper understanding of the tasks.

EEG data were recorded from 64 Ag/AgCl scalp electrodes placed according to the International 10/20 System referenced to nose and grounded prefrontal lobe. The EEG signals were acquired by a Neuroscan SynAmps2 amplifier whose sampling rate is 1000 Hz and the band-pass filtering range is 0.5–100 Hz. An additional 50-Hz notch filter was used during data acquisition. Thereafter, the original EEG signals were downsampled at 200 Hz. Before further analysis, the common average reference (CAR) was adopted during pre-processing.

The study was approved by the ethical committee of Tianjin University. All subjects signed their informed consent in advance.

Event-related spectral perturbation

The event-related spectral perturbation (ERSP) method was used to inspect the spectral power changes of the induced EEG relative to the stimulus from the views of the time-frequency domain, which could supply more details about ERD/ERS patterns of different types of MI. The changes of event-related spectral power were analyzed with ERSP defined as follows:
$$ ERSP\left(f,t\right)=\frac{1}{n}{\displaystyle \sum_{k=1}^n\left({F}_k{\left(f,t\right)}^2\right)} $$
(1)

where n is the number of trials, and F k (f, t) is the spectral estimation of the kth trial at frequency f and time t [18, 19]. Short-time Fourier transform (STFT) was employed to perform time-frequency analysis of EEG data using a Hanning-tapered window in EEGLAB. The number of windows was set to 200 so that the window length is 256 points for the first condition and 512 points for the second condition. Baseline-normalized ERSP values (dB) were calculated from −3 to 5 s and from −4 to 8 s for the first and second conditions, respectively, relative to a baseline period (3 s before movement onset for the first condition and 4 s before movement onset for the second condition). In this study, the time-frequency ERD/ERS maps from three key electrode positions C3, Cz, and C4 were presented between 1 and 30 Hz for analysis because these three electrode positions are located over the right hand, the foot, and the left hand representation areas, respectively. The ERSP in the remainder of this paper refers to the baseline-normalized ERSP.

To investigate the influence of the prior sub-movement on the subsequent sub-movement, we measured the alpha ERD values of the second or third sub-movement imagination by integrating the ERSP values within the corresponding frequency bands and time intervals according to equation (2), as follows:
$$ ER{D}_{value}=\frac{1}{N}{\displaystyle \sum_{f\in F}{\displaystyle \sum_{t\in T}\left( ERSP\left(f,t\right)\right)}} $$
(2)

where F represents the ERD band and T represents the time interval (1 s or 2 s). N is the number of time-frequency bins in a selected rectangular area. The ERD values of the right and left hands were calculated for electrodes C3 and C4, respectively.

Phase synchronization analysis

Phase synchronization is encountered in weakly interacting oscillator systems and it manifests through the occurrence of a relationship between the corresponding phases’ variables [20]. The degree of synchronization between two signals could be evaluated by the phase locking value (PLV), providing a reliable measurement of phase coupling [21, 22]. At first, the original EEG signals were band-pass filtered within a specific frequency band. Then, the instantaneous phases of the signals could be extracted by Hilbert transformation, and the difference of instantaneous phases corresponding to two different signals was defined as Δφ(t) = φ x (t) − φ y (t). After obtaining the relative phases, the degree of synchronization between any two signals was evaluated on a single trial basis using PLV, defined as
$$ PLV(t)=\left|\left\langle {e}^{j\varDelta \varphi (t)}\right\rangle \right| $$
(3)

where 〈  〉 denotes the temporal average over a time interval. Thereafter, the mean PLV was calculated by averaging all trials for each kind of movement imagination. In this way, the PLV ranges from 0 (no synchronization) to 1 (phase synchronization), providing an indication of the degree of interaction between the two underlying systems.

In this study, to investigate the involvement of the mesial frontocentral cortex during the imagination of sequential movement, phase synchronization was used to evaluate the underlying relationship between central electrodes (C3, Cz, and C4) and the mesial frontocentral electrode (Fcz) [2325] during imagination of each sub-movement. The signals from these four electrodes were band-pass filtered between 8 and 13 Hz to focus on the EEG activities of the mu rhythm. PLVs were calculated according to equation (3) on three pairs of electrodes (C3-Fcz, Cz-Fcz, and C4-Fcz) for sub-movement imagination (within 1 or 2 s) in each sequence for further investigation, that is, studying the PLVs over C3-Fcz, Cz-Fcz, and C4-Fcz for the right hand, right foot, and left hand movement imaginations, respectively.

Classification

Multi-class common spatial patterns based method

A common spatial patterns (CSP) algorithm has been widely used to extract the features of two MI classes based on multi-channel EEG information. Due to the property of CSP for binary situation, it has to be modified into a multi-class common spatial patterns (multi-CSP) algorithm to be suitable for the circumstance of multi-class MI tasks. Similar to the steps in [26], we first obtained the average covariance matrix Σ i of each MI pattern, i {1, 2, 3, 4}. The whitening matrix can be obtained by
$$ P={\varLambda}^{\raisebox{1ex}{$-1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}{U}_0^T $$
(4)
where U 0 is the matrix of eigenvectors and Λ is the diagonal matrix of eigenvalues from
$$ \varSigma ={\displaystyle \sum_{i=1}^4{\varSigma}_i}={U}_0\varLambda {U}_0^T $$
(5)
Thereafter, to acquire the spatial filter matrix relevant to the first class, we let \( {\varSigma}_1^{\hbox{'}}={\displaystyle \sum_{i=2}^4{\varSigma}_i} \) according to the strategy of one-versus-rest, and if Σ 1 and Σ 1 ' can be translated as
$$ {Y}_1=P{\varSigma}_1{P}^T\kern1em {Y}_1^{\hbox{'}}=P{\varSigma}_1^{\hbox{'}}{P}^T $$
(6)
then Y 1 and Y 1 ' share common eigenvectors
$$ {Y}_1={U}_1{\varLambda}_1{U}_1^T\kern1em {Y}_1^{\hbox{'}}={U}_1{\varLambda}_1^{\hbox{'}}{U}_1^T $$
(7)
With the projection matrix W 1 = U 1 T P consisting of spatial filters corresponding to the first class, the other three projection matrices could be obtained similarly.

The support vector machine (SVM) was adopted here for the classification of four kinds of sequential MI tasks. In this study, we used the LIBSVM software package to solve the multi-class classification problem. The multi-channel (64) EEG data were band-pass filtered between 8 and 30 Hz [26], including the whole MI period. By a tenfold cross-validation strategy, the training set served as the input of multi-CSP algorithm to achieve spatial filter matrices. For each direction of an MI task, only the eigenvectors corresponding to the first l eigenvalues could be used as spatial filters. The log variances of the spatially filtered EEG data were then used as the extracted features. The final classification accuracy \( acc=\frac{1}{10}{\displaystyle \sum_{k=1}^{10}ac{c}^{(k)}} \) where acc (k) is the classification accuracy of the testing set for the kth fold. The filter combination with the highest classification accuracy was the optimal combination.

Power spectral density based method

We extracted the power spectral density (PSD) as the feature for each MI task using the EEG data during imagination by sliding a short window in steps of 50 % overlaps, in which the PSD was calculated for 200 samples (1-s interval) shifted by 100 samples (0.5 s). Each segment was put into a PSD estimator using Burg's method (the order of an autoregressive model was 5). The sums of PSD in five frequency bands (delta: 0.5-4 Hz; theta: 4–8 Hz; alpha: 8–13 Hz; beta: 14–30 Hz; gamma: 30–50 Hz) were extracted for each segment, resulting in 5*m (m is the number of segments) features per channel for 62 channels (except HEO, VEO).

Support vector machine-recursive feature elimination (SVM-RFE), the most popular recursive feature-elimination algorithm, was applied to select the features for elimination. The RFE selection method is a recursive process that ranks features according to some measure of their importance [27]. However, in this study, the features coming from the same channel were regarded as a group; thus, we measured the importance of N channels. After constructing a final ranking of all channels, the features of the first n channels were taken as the input of SVM to obtain the corresponding classification accuracy using a tenfold cross-validation strategy. The channel combination with the highest classification accuracy was the optimal combination.

Results

Time-frequency analysis

Figure 3 shows the time-frequency maps of the four MI tasks for the first condition from one subject at electrode positions C3, Cz, and C4. The maps present obvious ERD patterns in both mu (11–13 Hz) and beta (22–24 Hz) rhythms for all the MI tasks. However, the phenomenon of ERD changes over time is not clear. Taking a close look at the ERD patterns at electrode C4 during imagination of LFL, slightly weaker ERD can be found within the 2nd second. There are no significant differences between the ERD patterns at C3 and C4 for the sequences of RLR and FRL. Besides the induced mu and beta ERD, there are also long-lasting ERS patterns (around 10 Hz) observed starting from almost 1 s after the MI onset at all these electrodes.
Fig. 3

Examples of time-frequency maps for the first condition from one subject at three electrode locations. RRR, RLR, LFL, and FRL indicate [right hand]–[right hand]–[right hand], [right hand]–[left hand]–[right hand], [left hand]–[right foot]–[left hand], and [right foot]–[right hand]–[left hand], respectively. Blue represents ERD and red represents ERS. The first and second vertical lines represent the onset and end of MI, respectively. Time 0 represents the time point of cue offset

Figure 4 shows the time-frequency maps of the four MI tasks for the second condition from the same subject at electrode positions C3, Cz, and C4. The maps show obvious ERD and ERS patterns within the same frequency bands as the first condition. In addition, the ERD patterns present better time-variant features, with apparent contralateral dominances for left/right hand imagery during the sequences. Compared with the sequence of RRR, quite different ERD patterns can be found for the other three sequences with discontinuous ERD over time at electrodes C3 and C4. ERD blocking is present at electrode C3, while the ERD feature reappears within the corresponding time interval at electrode C4 during the imagination of RLR. A similar phenomenon can be observed for the sequences of LFL and FRL. At the same time, no ERD feature induced by the sub-movement of the right foot is present at electrode Cz for both conditions.
Fig. 4

Examples of time-frequency maps for the second condition from the same subject at three electrode locations. RRR, RLR, LFL, and FRL indicate [right hand]–[right hand]–[right hand], [right hand]–[left hand]–[right hand], [left hand]–[right foot]–[left hand], and [right foot]–[right hand]–[left hand], respectively. Blue represents ERD and red represents ERS. The first and second vertical lines represent the onset and end of MI, respectively. Time 0 represents the time point of cue offset

ERD analysis

The ERD values within the alpha band during imagination of the second or third sub-movement were measured to evaluate the influence of the prior sub-movement on the subsequent sub-movement. According to Fig. 5a, right hand imagery with prior hand (left or right)/foot imagery is represented by RHPH/RHPF, while left hand imagery with prior hand (left or right)/foot imagery is represented by LHPH/LHPF. Therefore, with respect to the sub-movement of the right hand, we can obtain the ERD values of RHPH-I, RHPH-II, RHPH-III, and RHPF-I, while the ERD values of LHPH-I, LHPH-II, and LHPF-I can be obtained for the sub-movement of the left hand. Paired t-test was used to inspect the difference between the ERD values. The results of the ERD values at electrodes C3 and C4 for both conditions are shown in Fig. 5b. The ERD values of RHPH-I, RHPH-II, and RHPH-III are significantly larger than those of RHPF-I at electrode C3, while the ERD values of LHPH-I and LHPH-II are significantly larger than those of LHPF-I at electrode C4 for both conditions. These results indicate that the ERD values of left/right hand imagery with prior hand imagery are significantly larger than those with prior foot imagery during sequential movement imagination. For both conditions, there is no difference between the ERD values of RHPH-I, RHPH-II, and RHPH-III at electrode C3; there is also no difference between the ERD values of LHPH-I and LHPH-II at electrode C4, indicating that there is no difference between the ERD values of hand imagery with prior left or right hand imagery during sequential movement imagination.
Fig. 5

Illustration of ERD analysis. a Each row in the left diagram represents three sub-movements in each sequence. The diagram on the right contains the definition of the corresponding sub-movement in each sequence against the diagram on the left. “-” means no definition. b The comparison of ERD values from the 12 subjects at electrodes C3 (left column) and C4 (right column) for the first condition (upper panel) and second condition (lower panel). Dark red represents right hand imagery with prior hand imagery, while light red represents right hand imagery with prior foot imagery. Dark blue represents left hand imagery with prior hand imagery, while light blue represents left hand imagery with prior foot imagery. Conditions 1 and 2 indicate that the time intervals are 1 s and 2 s, respectively, for sub-movement imagination. Condition pairs that significantly differ from each other are indicated by a single asterisk (p < 0.05) or two asterisks (p < 0.01)

PLV analysis

Phase synchronization was used to evaluate the involvement of the mesial frontocentral cortex for each sub-movement imagination during sequential movement imagination. While analyzing PLVs for each sub-movement imagination in all sequences, we found the differences of PLVs between the initial movement (the first sub-movement) and the non-initial movement (the second or third sub-movement). In the following analysis, sub-movements in each sequence were divided into two groups: initial and non-initial movements. For right hand imagery, the initial movement consists of two initial imaginations during the sequence imagination of RRR and RLR, while the non-initial movement consists of four non-initial imaginations during the sequence imagination of RRR, RLR, and FRL. Similarly, the initial and non-initial imaginations of the left hand were integrated as a group, respectively. A homogeneity test of variance and the following two independent sample t-test were employed to investigate the differences of PLVs between the initial and non-initial movement imaginations for both conditions. As shown in Fig. 6a, ERD patterns around 13 Hz could be found at electrode Fcz during the imagination of RRR for both conditions. Figure 6b shows the comparative results of PLVs over C3-Fcz, Cz-Fcz, and C4-Fcz for both conditions. The PLVs over C3-Fcz during non-initial imaginations are significantly larger than those during initial imaginations for the right hand in both conditions. Meanwhile, the PLVs over Cz-Fcz for right foot imagery and the PLVs over C4-Fcz for left hand imagery present the same characteristic. The above results indicate that the PLVs over corresponding electrode pairs of the non-initial movement are significantly larger than those of the initial movement.
Fig. 6

The time-frequency maps at electrode position Fcz during imagination of RRR from subject S8 (a) and the comparison of PLVs over C3-Fcz, Cz-Fcz, and C4-Fcz from the 12 subjects (b) for the first condition (left column) and second condition (right column). Conditions 1 and 2 indicate that the time intervals are 1 s and 2 s, respectively, for sub-movement imagination. RRR indicates [right hand]–[right hand]–[right hand]. Blue represents ERD and red represents ERS. The first and second vertical lines represent the onset and end of MI, respectively. Time 0 represents the time point of cue offset. C3-Fcz(R), Cz-Fcz(F), and C4-Fcz(L) indicate calculating the PLVs over C3-Fcz, Cz-Fcz, and C4-Fcz for the right hand, right foot, and left hand imagery during sequences, respectively. The black and gray bars represent initial and non-initial movements, respectively. Condition pairs that significantly differ from each other are indicated by an asterisk (p < 0.05) or two asterisks (p < 0.01)

Classification performance

To validate the separability of the four types of sequential movement imagination in this study, we used multi-CSP-based and PSD-based methods, and then analyzed the highest classification accuracies with optimal combination in each method for both conditions. The highest classification accuracies were obtained with optimal filter combination using the 64-channel EEG data in the multi-CSP-based method while those were obtained with optimal channel combination in the PSD-based method. Figure 7 illustrates the classification accuracies of multi-CSP-based and PSD-based methods for both conditions. The classification accuracies of the PSD-based method are significantly higher than those of the multi-CSP-based method, whose mean classification accuracies are lower than 60 % for both conditions.
Fig. 7

The classification accuracies of multi-CSP-based and PSD-based methods for both conditions from the 12 subjects. Conditions 1 and 2 indicate that the time intervals are 1 s and 2 s, respectively, for sub-movement imagination. The black and gray bars indicate the classification accuracies of the multi-CSP-based and the PSD-based methods, respectively. Condition pairs that significantly differ from each other are indicated by a single asterisk (p < 0.05) or two asterisks (p < 0.01)

The averaged results developed over channels selected with SVM-RFE in the PSD-based method for both conditions are presented in Fig. 8. Results for both conditions show a similar trend reaching a maximum around 20 channels, and starting to decrease with more than 30 channels. The highest classification accuracy could be obtained with small channel subsets for both conditions, indicating a sharp decrease in channel numbers after feature optimization. Meanwhile, the classification accuracies of the second condition are higher than those of the first condition.
Fig. 8

The average result of SVM-RFE in PSD-based method from the 12 subjects. The horizontal and vertical axes represent the number of channels selected and the classification accuracies, respectively. The blue and red lines represent the first and second conditions, respectively. Conditions 1 and 2 indicate that the time intervals are 1 s and 2 s, respectively, for sub-movement imagination

Table 1 shows the classification accuracies of the four MI tasks using the PSD-based method in both conditions for the 12 subjects. The highest accuracies are 82.86 % and 91.43 %, and the mean values are 65 % and 74.14 % for the first and second conditions, respectively. The classification accuracies of the second condition are significantly higher than those of the first condition, exceeded by ~9 % in the mean classification accuracy, at the 5 % significance level using t-test (p = 0.001).
Table 1

Classification accuracies (%) of the PSD-based method in both conditions from the 12 subjects

PSD-based method

S1

S2

S3

S4

S5

S6

S7

S8

S9

S10

S11

S12

mean

Condition 1

63.57

69.29

72.86

43.57

73.57

73.57

56.43

82.86

62.50

61.79

67.14

52.86

65.00

Condition 2

78.21

76.43

73.93

68.21

70.36

80.71

76.07

91.43

70.71

75.36

73.21

55.00

74.14

Conditions 1 and 2 indicate that the time intervals are 1 s and 2 s, respectively, for sub-movement imagination. The highest accuracy for each condition is in bold font

Discussion

One disadvantage of MI-based BCI is the limitation of control commands. Several previous studies have been devoted to expanding the instruction set of MI-based BCI using simultaneous imagination of different limbs [17, 22, 28]. Although MI-based BCI has been frequently employed in rehabilitation engineering [2931], rehabilitation training using MI-based BCI with sequential and multiple limbs involved in movement could be a trend in the future. In terms of SCL-MI, enough time should be ensured to complete each sub-movement imagination considering limb switching between different limbs. Moreover, information transfer rate (ITR) is also an important aspect to be considered for a BCI system. Therefore, we set two task conditions with two different time intervals in this study. Although it is difficult to compress the time needed during SCL-MI, there are more candidate instructions (except those used in this study) that could be employed in MI-based BCI systems as compared with simultaneous compound limb MI.

The results in time-frequency maps mainly show the difference of ERD changes between two conditions brought by the changes in time interval during SCL-MI. From the time-frequency maps of the second condition, the ERD patterns of sub-movements imagination were found clearly occurring within corresponding time intervals over time at electrodes C3 and C4, due to the characteristic of contralateral dominance with hand imagery [32]. Such a phenomenon indicates that the changes of ERD patterns over time are in line with the changes of limbs during SCL-MI. However, the changes of ERD patterns in the first condition are less obvious than those for the second condition during SCL-MI. One explanation could be that a time interval of 1 s is not enough for EEG power returning to baseline after sub-movement imagination. At the same time, with respect to the sequences of RLR and FRL, it may also result from the mixed effect from the prior hand imagery and subsequent hand imagery on the same electrode. The time-frequency results reflect the importance of time-parameter setting for the imagination of each sub-movement. On the other hand, no ERD phenomenon was observed in the foot area during sub-movement imagination of the right foot, which may be due to its special location in the mesial wall.

In terms of the inter-influence between sub-movements, the ERD analysis reflects that different prior sub-movement imaginations contribute to different influences on the subsequent sub-movement imagination. The statistical analysis reveals significantly larger ERD values of left/right hand imagery with prior hand imagery, indicating that the prior hand imagery results in more neurons activated during the subsequent hand imagery because an increased ERD represents the involvement of more cell assemblies in information processing [32]. In addition, we found that there is no significant difference in ERD values at electrode C3 between the first and second sub-movements, and between the second and third sub-movements during sequence imagination of RRR for both conditions. In this regard, the main effect of the prior sub-movement may be caused by movement type. However, the MI in this study is based on drum movements where hand movement is more prominent than foot movement. Probably, a dominant movement could facilitate or a non-dominant movement could inhibit the activation of contralateral hand area with the subsequent hand imagery during sequential movement imagination. Hence, we still need further studies where both hands and foot imagery have equal priority or foot imagery is more prominent than hand imagery to justify the above speculation. Moreover, the process switching from hand imagery to hand imagery seems to be more efficient than switching from foot imagery to hand imagery in the four motor sequences of the present study as more efficient task performance shows larger alpha desynchronization in semantic tasks [32, 33]. On the other hand, ERD patterns were observed at electrode Fcz over the mesial frontocentral cortex, suggesting that the mesial frontocentral cortex was activated during sub-movement imagination because ERD reflects the activity in motor areas during MI [32]. Different from the source analysis, functional connectivity could reveal mutual interaction among multiple cortical regions. As one method to calculate functional connectivity, phase synchronization has been measured between electrodes over the contralateral hand area and the mesial frontocentral cortex during executed and imagined hand movements [34, 35], suggesting that the mesial frontocentral cortex is activated and synchronized with the contralateral hand area. Here, we analyzed phase synchronization to evaluate the involvement of the mesial frontocentral cortex depending on the coupling degree between the corresponding electrodes for each sub-movement during SCL-MI. Compared with initial movement, a significantly larger PLV reveals a closer coupling relationship between central electrodes and the mesial frontocentral electrode for non-initial movement, which probably implies a higher involvement of the mesial frontocentral cortex during the information processing of non-initial movement imagination. It is inferred that the coupling degree between the hand/foot representation areas and the mesial frontocentral cortex may be at a lower level during the initiation of sequence movement imagination, and then gradually becomes strengthened during the subsequent sub-movement imagination. In terms of non-initial movement, the mesial frontocentral cortex has been activated during the prior sub-movement imagination so that it may be easier to be aroused during the subsequent sub-movement imagination. Probably, this is the reason for stronger phase synchronization during non-initial movement imagination. The results of PLV analysis also reflect the inter-influence between sub-movements, in particular, that prior sub-movement may facilitate the involvement of the mesial frontocentral cortex for the subsequent sub-movement during the whole period of sequential movement imagination. As a result, not only the neural activities of sensorimotor areas but also those of the mesial frontocentral cortex could be influenced during limb switching, indicating that the mental changes associated with the imagination of sub-movement in the motor sequence are different from those of simple limb MI with the same limb.

Considering the different frequency bands used in the two classification methods, we also computed and compared the classification accuracies within the same frequency band. The mean classification accuracy is 39.46 % (SD ± 6.13) for the first condition and 42.50 % (SD ± 8.98) for the second condition in 0.5–50 Hz using the multi-CSP-based method. The mean classification accuracy is 63.17 % (SD ± 10.36) for the first condition and 70.39 % (SD ± 9.87) for the second condition in 8–30 Hz using the PSD-based method. The classification accuracies in 8–30 Hz are better than those in 0.5–50 Hz for the multi-CSP-based method. The frequency band 8–30 Hz is chosen because it encompasses the alpha and beta frequency bands, which have been shown to be most important for movement classification [26]. However, besides alpha and beta bands, other frequency bands could also be helpful in the classification. For the PSD-based method, the classification accuracies in 0.5–50 Hz are better than those in 8–30 Hz. A broader band (0.5–50 Hz) could be used to select the optimal feature from a high-dimensional feature space using SVM-RFE in the PSD-based method. It is implied that the selection of the frequency bands for both methods in this paper is available to extract features in the classification of four kinds of SCL-MI. In the meantime, the performance of the PSD-based method outperformed that of the multi-CSP-based method in both frequency bands. CSP is an algorithm based on spatial filtering of raw signals, while the PSD-based method extracts EEG features over time. It can be inferred that the changes of temporal effect are stronger than those of spatial effect during SCL-MI. SCL-MI presents time-variant EEG patterns on frequency and spatial domains due to limb switching. As a result, multi-CSP that is insensitive to temporal effect seems to be inappropriate for the classification of SCL-MI, while the PSD-based method is more suitable for capturing time-variant information and for discriminating the four types of sequential movement imagination. In addition, the classification results of multi-CSP-based and PSD-based methods should serve as a guide for the next development of algorithms applied for SCL-MI in the future.

For SVM-RFE, after a given point in the feature selection loop, all useless or redundant channels have already been removed, and the algorithm begins to eliminate channels that carry nonredundant information [36]. Therefore, the decrease in classification accuracies when evaluating smaller subsets with less than ~20 channels is typical of feature selection methods. SVM-RFE selects small feature subsets (optimal channel combination) with better discrimination capabilities than using a whole feature set, indicating that SVM-RFE is an effective method for reducing feature dimension to reach the highest accuracy with the least channels. From the higher classification accuracies obtained in the second condition compared with those in the first condition, we can infer that SCL-MI presents much richer information of time-variant EEG patterns for the second condition. A time interval of 2 s is enough for the changes of EEG patterns during limb switching just as the ERD changes revealed in time-frequency maps. The four sequential MI tasks could not be discriminated effectively with the smaller time interval, which also indicates the significance of the time required for sub-movement imagination. Overall, the classification accuracies in this study are acceptable, but have yet to be improved with more suitable algorithms in the future. In conclusion, multi-class SCL-MI and the feature extraction method studied in this paper could be expected to provide technical support to expand instructions of MI-based BCI systems effectively. Furthermore, as an important tool in the field of rehabilitation engineering, it would seem significant to consider the tendency and availability to complete sequences as a whole rather than one independent movement at a time in MI-based BCI systems.

Conclusion

This study provides new information about ERD changes with different time intervals during SCL-MI. Aside from this finding, prior sub-movement imagination could affect the neural activities in the motor areas during the subsequent sub-movement imagination, suggesting that inter-influence exists between sub-movements in the motor sequence. On the basis of classification performance, multiple types of sequential movement imagination appear to have potential for use as a novel paradigm in multi-class MI-based BCI system.

Declarations

Acknowledgements

The authors wish to thank Prof. Xiaobo Xie from Institute of Biomedical Engineering, Chinese Academy of Medical Science & Peking Union Medical College and Prof. Yong Hu from Department of Orthopedic Surgery, Hong Kong University, Hong Kong, for the contributions of the experiment.

This paper was supported by National Natural Science Foundation of China (Grant No. 91520205, 81571762 and 31500865), Tianjin Key Technology R&D Program (No. 15ZCZDSY00930).

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University
(2)
Tianjin Key Laboratory of Biomedical Detecting Techniques and Instruments, Tianjin University

References

  1. Pfurtscheller G, Nerper C. Motor imagery and direct brain–computer communication. Proceeding of the IEEE. 2001;89:1123–34.View ArticleGoogle Scholar
  2. Munzert J, Lorey B, Zentgraf K. Cognitive motor processes: the role of motor imagery in the study of motor representations. Brain Res Rev. 2009;60:306–26.PubMedView ArticleGoogle Scholar
  3. Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM. Brain–computer interfaces for communication and control. Clin Neurophysiol. 2002;113:767–91.PubMedView ArticleGoogle Scholar
  4. Jasper H, Penfield W. Electrocardiograms in man: effect of voluntary movement upon the electrical activity of the precentral gyrus. Eur Arch Psychiatry Clin Neurosci. 1949;183:163–74.Google Scholar
  5. Jeon Y, Nam CS, Kim YJ, Whang MC. Event-related (De)synchronization (ERD/ERS) during motor imagery tasks: Implications for brain–computer interfaces. Int J Ind Ergon. 2011;41:428–36.View ArticleGoogle Scholar
  6. Friedrich EV, Neuper C, Scherer R. Whatever works: a systematic user-centered training protocol to optimize brain-computer interfacing individually. PLoS One. 2013;8:e76214.PubMedPubMed CentralView ArticleGoogle Scholar
  7. Pfurtscheller G, Neuper C, Schlögl A, Lopes da Silva FH. Mu rhythm (de)synchronization and EEG single-trial classification of different motor imagery tasks. NeuroImage. 2006;31:153–9.PubMedView ArticleGoogle Scholar
  8. Li Y, Long J, Yu T, Yu Z, Wang C, Zhang H, et al. An EEG-based BCI system for 2-D cursor control by combining Mu/Beta rhythm and P300 potential. IEEE Trans Biomed Eng. 2010;57:2495–505.PubMedView ArticleGoogle Scholar
  9. Pfurtscheller G, Solis-Escalante T, Ortner R, Linortner P, Müller-Putz GR. Self-paced operation of an SSVEP-Based orthosis with and without an imagery-based "brain switch:" a feasibility study towards a hybrid BCI. IEEE Trans Neural Syst Rehabil Eng. 2010;18:409–14.PubMedView ArticleGoogle Scholar
  10. Lacourse MG, Orr ELR, Cramer SC, Cohen MJ. Brain activation during execution and motor imagery of novel and skilled sequential hand movements. NeuroImage. 2005;27:505–19.PubMedView ArticleGoogle Scholar
  11. Holper L, Scholkmann F, Shalóm DE, Wolf M. Extension of mental preparation positively affects motor imagery as compared to motor execution: a functional near-infrared spectroscopy study. Cortex. 2012;48:593–603.PubMedView ArticleGoogle Scholar
  12. Roosink M, Zijdewind I. Corticospinal excitability during observation and imagery of simple and complex hand tasks: Implications for motor rehabilitation. Behav Brain Res. 2010;213:35–41.PubMedView ArticleGoogle Scholar
  13. Jackson PL, Lafleur MF, Malouin F, Richards CL, Doyon J. Functional cerebral reorganization following motor sequence learning through mental practice with motor imagery. NeuroImage. 2003;20:1171–80.PubMedView ArticleGoogle Scholar
  14. Lafleur MF, Jackson PL, Malouin F, Richards CL, Evans AC, Doyon J. Motor learning produces parallel dynamic functional changes during the execution and imagination of sequential foot movements. NeuroImage. 2002;16:142–57.PubMedView ArticleGoogle Scholar
  15. Caldara R, Deiber MP, Andrey C, Michel CM, Thut G, Hauert CA. Actual and mental motor preparation and execution: a spatiotemporal ERP study. Exp Brain Res. 2004;159:389–99.PubMedView ArticleGoogle Scholar
  16. Kranczioch C, Mathews S, Dean P, Sterr A. Task complexity differentially affects executed and imagined movement preparation: evidence from movement-related potentials. PLoS One. 2010;19:e9284.View ArticleGoogle Scholar
  17. Yi W, Qiu S, Qi H, Zhang L, Wan B, Ming D. EEG feature comparison and classification of simple and compound limb motor imagery. J Neuroeng Rehabil. 2013;10:106.PubMedPubMed CentralView ArticleGoogle Scholar
  18. Delorme A, Makeig S. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics. J Neurosci Methods. 2004;134:9–21.PubMedView ArticleGoogle Scholar
  19. Makeig S. Auditory event-related dynamics of the EEG spectrum and effects of exposure to tones. Electroencephalogr Clin Neurophysiol. 1993;86:283–93.PubMedView ArticleGoogle Scholar
  20. Stavrinou ML, Moraru L, Cimponeriu L, Della-Penna S, Bezerianos A. Evaluation of cortical connectivity during real and imagined rhythmic finger tapping. Brain Topogr. 2007;19:137–45.PubMedView ArticleGoogle Scholar
  21. Lachaux JP, Rodriguez E, Martinerie J, Varela FJ. Measuring phase synchrony in brain signals. Hum Brain Mapp. 1999;8:194–208.PubMedView ArticleGoogle Scholar
  22. Zhou Z, Wan B, Ming D, Qi H. A novel technique for phase synchrony measurement from the complex motor imaginary potential of combined body and limb action. J Neural Eng. 2010;7:046008.PubMedView ArticleGoogle Scholar
  23. Macar F, Vidal F, Casini L. The supplementary motor area in motor and sensory timing: evidence from slow brain potential changes. Exp Brain Res. 1999;125:271–80.PubMedView ArticleGoogle Scholar
  24. Wheaton LA, Nolte G, Bohlhalter S, Fridman E, Hallett M. Synchronization of parietal and premotor areas during preparation and execution of praxis hand movements. Clin Neurophysiol. 2005;116:1382–90.PubMedView ArticleGoogle Scholar
  25. Kononowicz TW, van Rijn H. Slow potentials in time estimation: the role of temporal accumulation and habituation. Front Integr Neurosci. 2011;5:48.PubMedPubMed CentralView ArticleGoogle Scholar
  26. Ramoser H, Müller-Gerking J, Pfurtscheller G. Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans Rehab Eng. 2000;8:441–6.View ArticleGoogle Scholar
  27. Guyon I, Weston J, Barnhill S, Vapnik V. Gene selection for cancer classification using support vector machines. Mach Learn. 2002;46:389–422.View ArticleGoogle Scholar
  28. Doud J, Lucas P, Pisansky T, He B. Continuous three-dimensional control of a virtual helicopter using a motor imagery based brain–computer interface. PLoS One. 2011;6:e26322.PubMedPubMed CentralView ArticleGoogle Scholar
  29. Prasad G, Herman P, Coyle D, McDonough S, Crosbie J. Applying a brain-computer interface to support motor imagery practice in people with stroke for upper limb recovery: a feasibility study. J Neuroeng Rehabil. 2010;7:60.PubMedPubMed CentralView ArticleGoogle Scholar
  30. Broetz D, Braun C, Weber C, Soekadar SR, Caria A, Birbaumer N. Combination of brain-computer interface training and goal-directed physical therapy in chronic stroke: a case report. Neurorehabil Neural Repair. 2010;24:674–9.PubMedView ArticleGoogle Scholar
  31. Daly JJ, Cheng R, Rogers J, Litinas K, Hrovat K, Dohring M. Feasibility of a new application of noninvasive Brain Computer Interface (BCI): a case study of training for recovery of volitional motor control after stroke. J Neurol Phys Ther. 2009;33:203–11.PubMedView ArticleGoogle Scholar
  32. Pfurtscheller G, Lopez-da-Silva FH. Event-related EEG/MEG synchronization and desynchronization: basic principles. Clin Neurophysiol. 1999;110:1842–57.PubMedView ArticleGoogle Scholar
  33. Klimesch W. EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain Res Rev. 1999;29:169–95.PubMedView ArticleGoogle Scholar
  34. Wang Y, Hong B, Gao X, Gao S. Phase synchrony measurement in motor cortex for classifying single-trial EEG during motor imagery. New York: Proc. 28th Annu. Int. IEEE EMBS Conf; 2006. p. 75–8.Google Scholar
  35. Sweeney-Reed CM, Nasuto SJ. Detection of neural correlates of self-paced motor activity using empirical mode decomposition phase locking analysis. J Neurosci Methods. 2009;184:54–70.PubMedView ArticleGoogle Scholar
  36. Granittoa PM, Furlanellob C, Biasiolia F, Gasperi F. Recursive feature elimination with random forest for PTR-MS analysis of agroindustrial products. Chemometr Intell Lab. 2006;83:83–90.View ArticleGoogle Scholar

Copyright

© Yi et al. 2016

Advertisement