Skip to main content

Toward a hybrid brain-computer interface based on repetitive visual stimuli with missing events

Abstract

Background

Steady-state visually evoked potentials (SSVEPs) can be elicited by repetitive stimuli and extracted in the frequency domain with satisfied performance. However, the temporal information of such stimulus is often ignored. In this study, we utilized repetitive visual stimuli with missing events to present a novel hybrid BCI paradigm based on SSVEP and omitted stimulus potential (OSP).

Methods

Four discs flickering from black to white with missing flickers served as visual stimulators to simultaneously elicit subject’s SSVEPs and OSPs. Key parameters in the new paradigm, including flicker frequency, optimal electrodes, missing flicker duration and intervals of missing events were qualitatively discussed with offline data. Two omitted flicker patterns including missing black/white disc were proposed and compared. Averaging times were optimized with Information Transfer Rate (ITR) in online experiments, where SSVEPs and OSPs were identified using Canonical Correlation Analysis in the frequency domain and Support Vector Machine (SVM)-Bayes fusion in the time domain, respectively.

Results and conclusions

The online accuracy and ITR (mean ± standard deviation) over nine healthy subjects were 79.29 ± 18.14 % and 19.45 ± 11.99 bits/min with missing black disc pattern, and 86.82 ± 12.91 % and 24.06 ± 10.95 bits/min with missing white disc pattern, respectively. The proposed BCI paradigm, for the first time, demonstrated that SSVEPs and OSPs can be simultaneously elicited in single visual stimulus pattern and recognized in real-time with satisfied performance. Besides the frequency features such as SSVEP elicited by repetitive stimuli, we found a new feature (OSP) in the time domain to design a novel hybrid BCI paradigm by adding missing events in repetitive stimuli.

Background

Brain Computer Interface (BCI) is a communication technology that bypasses human’s normal output pathways of muscle and peripheral nervous, providing a direct connection between human brain and external devices. For people with severe motor disabilities such as spinal cord injury (SCI), BCI is a promising tool for communication and control [14]. Most of BCI researches rely on the conscious modulation of noninvasively scalp electroencephalography (EEG) to either external stimuli or internal motorsensory tasks, namely, exogenous or endogenous BCI, respectively. EEG signals including sensorimotor rhythms (SMR), P300, event-related desynchronization (ERD), event-related synchronization (ERS), and steady-state visual evoked potentials (SSVEPs) have been used to design different BCI paradigms. Specifically, SSVEP and P300-based BCI usually have the advantages of high information transfer rate (ITR), high tolerance to artifacts and robust performance across users with minimal training requirement [520]. SSVEPs are periodically visual cortical responses evoked by repetitive stimuli with a constant frequency [5]. In previous studies, several methods such as canonical correlation analysis (CCA), Fourier Transform (FT) and common feature analysis (CFA) were employed to extract SSVEP features in EEG [713], where CCA is the most commonly used. Zhang et al. proposed an improved CCA algorithm called L1-regularized multiway CCA (L1-MCCA) to better recognize SSVEP. P300 is a positive potential recorded approximately 300 ms after the stimuli. Generally, P300 can be elicited with oddball, single stimulus, and omitted-stimulus paradigms [14]. Most studies developed P300-based BCI based on oddball paradigm [1524]. Classifiers such as linear discriminant analysis (LDA) [1517], support vector machine (SVM) [18] and Bayesian fusion [19, 20] were proved to be efficient in P300 classification.

In the recent decade, hybrid BCIs that combing two different physiological signals have been developed to improve BCI performances. Specifically, pure hybrid BCI which incorporate two different BCI paradigms is becoming the hot topic in the field [2134]. Ferrez et al. proposed a hybrid BCI paradigm based on motor imagery (ERD/ERS) and error-related potentials [27]. Allison [28] and Brunner [29] combined motor imagery and SSVEP to improve the accuracy of hybrid BCI system. Long et al. developed a hybrid BCI based on motor imagery and P300 to control a 2-D cursor [30]. In recent years, hybrid BCI based on SSVEP and P300 received the most attention. Edlinger et al. proposed a hybrid BCI system based on SSVEP followed by P300 [31]. P300-based BCI was used to control a virtual smart home environment, while SSVEP was implemented as a toggle switch to initiate and stop P300-based BCI. Panicker et al. carried out similar study, where SSVEP was also used as a switch of P300-based BCI [32]. Yin et al. developed a hybrid BCI speller based on the fusion of P300 and SSVEP [21, 22]. SSVEP and P300 were simultaneously elicited in this study. Specifically, they used oddball paradigm to elicit P300. Xu et al. [34] proposed a hybrid BCI speller paradigm combing P300 and SSVEP blocking feature. However, the mechanism of hybrid BCI based on SSVEP and P300 remains unclear.

On the other hand, another type of P300, elicited with omitted-stimulus paradigms, defined as Omitted Stimulus Potentials (OSP), was rarely concerned in BCI. Previous studies indicated that OSPs are event-related potentials time-locked to the absence of a stimulus in a regular series with acoustic and visual stimuli [3537]. These “slow” OSPs were elicited with a stimulus of low frequency (<2Hz). The particular phenomenon also existed in ganglion cells of salamander and mouse [38]. However, SSVEP can be elicited only with repetitive stimulus of high frequency (>6Hz) [5, 39]. In this case, it is impossible to simultaneously elicit SSVEP and OSP. Bullock et al. reported a new omitted stimulus potential called “fast” OSP recorded in the retina of fish and reptiles [40, 41]. Later studies recorded “fast” OSPs on the human scalp EEGs, using conditioning trains of light flashes with frequency >5Hz [42]. Similar phenomenon was also reported in human beings with auditory stimuli [43]. Compared with “slow” OSPs, “fast” OSPs can be elicited without attention. Thus, it is possible to simultaneously elicit SSVEP and OSP. However, how to construct an efficiently hybrid BCI based on SSVEP and OSP, remains unknown.

In this study, we systematically investigated a hybrid BCI paradigm based on SSVEP and OSP for the first time. Repetitive visual stimuli with missing events were presented to simultaneously elicit SSVEP and OSP. Key parameters including flicker frequency, missing flicker duration, optimal electrodes, missing flicker patterns, averaging times and intervals of missing events were determined to obtain satisfied BCI performance. SSVEPs and OSPs were identified using Canonical Correlation Analysis in frequency domain and Support Vector Machine (SVM) -Bayes fusion in time domain, respectively. Finally, online experiments over nine healthy subjects validated the performance of the proposed BCI paradigm. This paper was organized into four sections with this section as introduction. The materials and methods, including paradigm design, experimental setup, feature extraction of SSVEPs and OSPs were described in "Methods" section. In "Results" section, we discussed the optimization of key parameters in the proposed paradigm and presented the online results over nine healthy subjects. Discussions and conclusions were given in "Discussion" section.

Methods

Stimulation paradigm

In this study, we proposed a novel hybrid BCI paradigm based on repetitive stimuli with missing events. It was constructed with four visual stimulators which can elicit SSVEP and OSP simultaneously and displayed on a DELL U2312HM screen with refresh rate of 60 Hz. During the experiments, the subjects’ viewing distance to the screen was 70 cm. Four stimulators were uniformly spaced in left, right, up and down directions to the center of the monitor. The stimulus luminance was 150 cd/m2 for the white discs and 0.7 cd/m2 for the black ones (Michelson contrast of 98.8 %). Each stimulator was a disc with diameter of 6.5 degrees and the distance from the outer edge of each stimulator to the center of the monitor was 10.3 degrees, as shown in Fig. 1a. Therein, disc 1 and 3 flickered at the same frequency with different onset time of missing event, so did disc 2 and 4, as shown in Fig. 1b. The high and low level of the sequence indicate white and black for the corresponding discs. Since the background color of the screen is black, “missing white discs” looks like the disc disappeared and “missing black discs” looks like the disc stopped. A typical EEG response in the time domain to the repetitive stimuli with missing events was shown in Fig. 1c. Red dotted line indicated the onset time of the missing events. Presentation of the stimulators is controlled by the Psychophysics Toolbox 3.0 [44, 45].

Fig. 1
figure 1

Schematic diagram of the experimental setup. a Distribution of four stimulators on the screen. The cross indicating the center of the monitor was not presented on the screen; b Stimulus sequence for four discs in a; c Typical EEG responses in the time domain to the repetitive stimuli with missing events

Subjects and recordings

Nine right-handed healthy subjects (8 male, 1 female, age 23–26) participated in the experiments. They were requested to sit in a comfortable armchair in an ordinary lighting office room with no electromagnetic shielding. All of them had normal or corrected-to-normal eyesight and experienced BCIs before. Similar to the traditional SSVEP-based BCIs, subjects in the experiments were asked to binocularly view the screen and fixate on the center of the target stimulator.

In previous SSVEP studies, EEGs were usually acquired at occipital sites [7, 46, 47]. In order to find the optimal electrodes which can best record SSVEP-OSP features in EEGs, ten electrodes including O1, O2, Oz, PO3, POz, PO4, PO7, PO8, Pz and Cz in International 10–10 System were adopted. EEGs were collected using a g. USBamp (g.tec Inc., Austria) system with sampling rate 1200 Hz. Signals were referenced to a unilateral earlobe and grounded at Fpz. Online band-pass filter of 0.01-100 Hz and notch filter between 48–52 Hz were utilized to remove artifacts and power line interference. All electrodes impedances were kept below 5 \( \mathrm{k}\Omega \) during experiments. Each subject carried out the experiments composed of classifier training and online testing. During classifier training period, four experimental tasks were executed where Task 1 to Task 4 was to fixate on the four visual stimulators (disc 1–4) on the screen. Each task contained seven runs and each run included sixteen trials. Subjects were instructed to fixate on a specific stimulator throughout the task. Task 1–4 were performed one by one in random order. In each run, dark screen was first displayed for 2 s, and then the target prompt was presented for 0.5 s. After that, four stimulators were simultaneously presented for 2.5 s. Two adjacent trials were isolated by dark screen and the interval time was fixed to 0.5 s. Before the ending of each run, dark screen was displayed for 2 s and the subject was asked to relax. Each run lasted for 60 s. The classifier training period for each subject lasted for about 30 min. The timing of the experimental sequence and behavior task for classifier training was shown in Fig. 2a. In online testing, ten runs were included and each run had 16 trials. The timing of the experimental sequence and behavior task for each run in online testing (see Fig. 2b) was identical with that of classifier training period except that the classification result calculated by the trained classifier was displayed after the presentation of visual stimulators (2.5 s), which provides a feedback (success or failure) for the last task.

Fig. 2
figure 2

The timing of the experimental sequence and behavior task for a classifier training and b online testing

Extraction of SSVEPs using canonical correlation analysis

Canonical Correlation Analysis (CCA) is a nonparametric multivariable method [48] to quantify the correlation between two signals and has been used in in extraction of SSVEPs in EEG signals. Compared with tradition methods, CCA can identify SSVEPs at satisfied accuracies without subject- specific training [4951]. In this paper, we adopted CCA to measure the correlation between stimulus frequencies and EEGs acquired from selected electrodes. When the subject fixated on a specific stimulator, an enhanced correlation between its stimulus frequency and EEG signals was expected to occur, leading to a largest correlation coefficient among all stimulators. The acquired EEG signals were pre-processed with a band-pass filter of 1–45 Hz to remove baseline excursion and high frequency noises. Details about CCA can be referenced in our previous study [7].

Recognition of OSPs based on Support Vector Machine (SVM) and Naive Bayes

With CCA, the flicker frequency of the target stimulator was identified. Since there are 2 stimulators flickering at the same frequency with different onset time of missing event in the proposed BCI paradigm, the target stimulator cannot be identified until the OSP features were recognized. Previous study demonstrated SVM followed by naive Bayes can solve classification problem of time series such as event related potentials (ERPs) with satisfied performance [18]. In this study, we utilize the similar scheme to detect OSP in the time domain.

Support vector machine

Support vector machine is an efficient tool for solving supervised classification problems due to its generalization performance and established empirical performance [52]. The basic idea of classification with SVM is to project the sample space into a high-dimensional eigenspace and find an optimal separating hyperplane (OSH) for a given feature set, which maximizes the margin between the training data and the decision boundary. The construction of OSH can be described as the following quadratic optimization problem:

$$ \underset{\mathbf{w}}{ \min}\frac{1}{2}\left({\mathbf{w}}^T\mathbf{w}\right)\kern0.7em s.t.\kern1em {d}^s\left(\left({\mathbf{w}}^T{\mathbf{x}}^s\right)+b\right)\ge 1 $$
(1)

where d s {−1, 1} represents the sth desired output, x sR p is the sth input sample of the training data set {x s, d s} S s = 1 and s is the number of training vectors. In practice, the OSH probably does not exist. Hence, the slack parameters ξ s  ≥ 0, s = 1, 2, S are introduced. The optimization problem now becomes:

$$ \begin{array}{c}\hfill \underset{\mathbf{w},\xi }{ \min}\frac{1}{2}\left({\mathbf{w}}^T\mathbf{w}\right)+C{\displaystyle \sum_{s=1}^S{\xi}_s}\hfill \\ {}\hfill s.t.\kern1em {d}^s\left(\left({\mathbf{w}}^T{\mathbf{x}}^s\right)+b\right)\ge 1-{\xi}_s;{\xi}_s\ge 0,s=1,2,\cdots S\hfill \end{array} $$
(2)

where C stands for the misclassification penalty term and can be considered as the regularization parameter. A larger C indicates higher penalty to the training errors. By introducing Lagrange multipliers α s , the OSH is computed as a decision surface:

$$ f\left(\mathbf{x}\right)=\operatorname{sgn}\left({\displaystyle \sum_{s=1}^S{d}^s{\alpha}_sK\left(\mathbf{x},{\mathbf{x}}^s\right)+b}\right) $$
(3)

where sgn()  {±1}, x are support vectors and K() is the kernel function. Here, we employ the linear function defined as:

$$ K\left(\mathbf{x},{\mathbf{x}}^s\right)=x\cdot {x}^s $$
(4)

The regularization parameter C was optimized using 10-fold cross-validation. SVM was then trained using the regularization parameter C with the best validation performance.

Naive bayes

Naive Bayes is a probabilistic classifier based on Bayes theorem with strong independence assumptions between the features. It is widely used in EEG classification and has satisfied performance without large amount of training samples [53]. In Bayes theorem, posteriori probability is calculated based on the priori probability:

$$ P\left(C\Big|A\right)=P\left(A\Big|C\right)P(C)/P(A) $$
(5)

where A is the set of attributes {A 1, A 2, , A n } and C is the set of hypotheses {c 1, c 2, , c m }. The classifier output is given by

$$ {c}_{nb}= \arg \underset{c_i\in C}{ \max }P\left({c}_i\Big|A\right) $$
(6)

According to Bayes theorem, it can be written as

$$ {c}_{nb}= \arg \underset{c_i\in C}{ \max}\left[P\left(A\Big|{c}_i\right)P\left({c}_i\right)/P(A)\right] $$
(7)

Due to the independence assumptions of the attributes, the joint probability can be calculated with

$$ P\left(A\Big|{c}_i\right)={\displaystyle \prod_{j=1}^nP\left(A{}_j\Big|{c}_i\right)} $$
(8)

Thus the output of Naive Bayes is:

$$ {c}_{nb}= \arg \underset{c_i\in C}{ \max}\frac{{\displaystyle \prod_{j=1}^nP\left(A{}_j\Big|{c}_i\right)}P\left({c}_i\right)}{P(A)} $$
(9)

Recognition of OSPs

Previous study demonstrated that OSP occurred 125–375 ms after the onset of missing events [42]. Since OSP is time-locked relative to the missing events, it will be strengthened in case that the signal is superimposed and averaged with starting line at the onset time of target stimulus. For non-target stimulus, OSPs will be weakened or even lost after averaging. The averaged segments are input into SVM and Naive Bayes to recognize targeted OSPs with FieldTrip toolbox [54], as shown in Fig. 3.

Fig. 3
figure 3

Flowchart of OSP recognition using SVM and Naive Bayes

In time windows T = {Τ 1, …, Τ 10} = {{125ms − 150ms}, …, {350ms − 375ms}}, EEG signals acquired from electrodes L = {O1, Oz, O2, POz, PO4, PO8} construct a feature sets

x i (L, T i ) = [x(l 1, T i ), …, x i (L, T i ) = [x(l 1, T i ), …, x(l 6, T i )] i = 1, 2, …, 10 where x i (l,T i ) stands for the EEG segment in time window T i at electrode l. The feature sets are input into SVMs to obtain the classification results d k ,k = 1,2,…,10. The results are further input into Naive Bayes to get the probability of fixation on a specific stimulator:

$$ P={\displaystyle \prod_{k=1}^{10}{p_{k01}}^{\left(1-{d}_k\right)}{p_{k11}}^{d_k}}/{\displaystyle \prod_{k=1}^{10}{p_{k00}}^{\left(1-{d}_k\right)}{p_{k10}}^{d_k}} $$
(10)

Where \( \begin{array}{cc}\hfill {p}_{kmn}=p\left({d}_k=m\left|c=n\right.\right)\hfill & \hfill m,n=0,1\hfill \end{array} \). c = 1 indicates the target stimulator. p k01, p k11, p k00, p k10 is obtained during classifier training. When the classifier is performed on the segments superimposed with different onset time of missing events, the target stimulator with maximum probability of fixation P is recognized. As shown in Fig. 2a, each subject was asked to perform 4 tasks (fixating on the four visual stimulators) during classifier training. Each task contained 7 runs and each run included 16 trials. So, there were 6 × 4 × 7 × 16 = 2688 samples during classifier training for each subject. The dimensionality of each feature vector was 10 × 30.

Information transfer rate

In order to evaluate the proposed BCI paradigm, Information Transfer Rate (ITR) was used to measure the achievable information rate per unit time, given the detection accuracy and the time required for target identification:

$$ \mathrm{I}\mathrm{T}\mathrm{R}=\frac{60}{DTI}\left[{ \log}_2N+Acc{ \log}_2Acc+\left(1-Acc\right){ \log}_2\left(\frac{1-Acc}{N-1}\right)\right] $$
(11)

where N is the number of stimulators, Acc is the mean detection accuracy averaged over all stimulators and DTI is the decision transfer interval (i.e., the sum of single detection time and interval between detections). Furthermore, one-way ANOVA was performed in SPSS19 (SPSS Inc., Chicago, Illinois, USA) for statistical comparison of system performance across all the subjects. The significance level for all statistical analyses was set at p < 0.05.

Results

Stimulus frequencies

Due to the limitation of refresh rate of the computer screen (60 Hz), stimulus frequencies including 30 Hz, 20 Hz, 15 Hz, 12 Hz, 10 Hz and 8.6 Hz were selected to elicit SSVEP-OSP features. In order to clarify the frequency effect, one disc flickered at different frequencies for 1 s followed by absence of black disc (namely, the disc was paused on white) for another 1 s. Subjects were asked to decrease eye-blinking frequency and avoid body movements during stimulus presentation. EEGs including 1 s before and after onset of missing flickers were superimposed and averaged over 16 trials. Typical results of two electrodes at occipital sites (PO8 and Oz) are shown in Fig. 4. Moments “0” of x-axis indicate the onset time of missing flickers (absence of black disc). For both electrodes across all stimulus frequencies, OSP features, i.e., three transient components of P1, N2 and P2 with latencies of 160, 210 and 290 ms after onset of missing flickers can be observed. Specifically, positive peak of OSP at P2 has higher amplitude with stimulus frequencies of 10 to 20 Hz, which is consistent with previous study [42]. On the other hand, stronger SSVEP responses appear at stimulus frequencies of 8.6, 10, 12 and 15 Hz. Previous studies demonstrated that SSVEP responses can be also observed at the second harmonic frequencies [7]. It would lead to confusion of SSVEP in case we select 10 and 20 Hz as the stimulus frequencies in the proposed paradigm. As a result, 10 and 12 Hz were chosen because both SSVEP and OSP features are prominent enough to be extracted in the time or frequency domain.

Fig. 4
figure 4

Comparison of SSVEPs and OSPs at PO8 and Oz elicited by different stimulus frequencies

Optimal electrodes

Previous studies demonstrated that SSVEPs are prominent at occipital sites [7, 46, 47]. In order to locate the optimal electrodes for OSP feature, the visual stimulator was presented at frequency of 12 Hz with white disc missing for 1/12 s. Time series and topographies of EEGs after the onset of omitted stimulus are shown in Fig. 5. Specifically, OSPs with higher amplitude was recorded at O1, Oz, O2, POz, PO4 and PO8 and thus selected as optimal electrodes to extract OSPs.

Fig. 5
figure 5

Comparison of SSVEP-OSP at O2 with different duration of missing events

Duration of missing events

Previous studies indicated missing events in repetitive stimuli may generate OSPs [38, 42]. Missing events with longer duration may elicit complete OSPs in the time domain whereas SSVEP features disappeared during the missing events. An ideal situation is that OSPs are elicited by missing events with very short duration (i.e., 1/Stimulus Frequency) and the negative effect to SSVEP can be minimized. In order to testify the hypothesis, repetitive stimuli with frequency of 10 Hz were used to elicit SSVEP-OSP features. The stimuli was first presented for 1 s, then paused on white for 0.1 or 1 s and presented for another 1 s. The pre-processed data (band-pass and notch filter) were superimposed and averaged over 8 single trials to extract SSVEP-OSP features. Results are shown in Fig. 6. Moment “0” of x-axis indicated the onset time of missing events. Similar SSVEP and OSP features can be elicited in both conditions. Especially, the latencies of OSP relative to the onset time of missing event (“0” of x-axis) remain the same. In the online experiment, we set the duration of missing events to minimum value (1/Stimulus Frequency), which is enough to elicit OSPs while keeping SSVEPs.

Fig. 6
figure 6

Time series and topographies of EEGs from 10-channel after the onset of omitted stimulus. Red color indicates OSPs with higher amplitude

Missing flicker patterns

Two missing flicker patterns are available in the proposed BCI paradigm: missing white and black discs. Since the background color of the screen is black during visual stimulation, “missing white discs” looks like the stimulator disappeared and “missing black discs” looks like the stimulator stopped. We qualitatively compare the SSVEP-OSP features elicited by these two missing flicker patterns. Visual stimulus was presented at frequency of 10 Hz with two different missing flicker patterns with duration of 0.1 s. EEGs 1 s before and after onset of missing pauses were superimposed and averaged over 8 trials (see Fig. 7). Moment “0” of x-axis indicated the onset time of missing flicker. Compared with missing black disc pattern, the elicited OSP with missing white disc pattern was superposed on the SSVEP with higher amplitude. Generally, OSPs can be elicited with both missing flicker patterns. Further quantitative comparisons will be made based on accuracy and information transfer rate (ITR) in online experiments.

Fig. 7
figure 7

Comparison of different missing flicker patterns

Interval of missing events

According to equation 11, interval of missing events affects ITR in two ways: increasing interval of missing events may (1) result in longer DTI with a negative effect on ITR; (2) strengthen SSVEP features and increase Acc with a positive effect on ITR. With stimulus frequency 12 Hz, typical EEG responses to different interval of missing events are shown in Fig. 8. Therein, interval of missing events 1 s, 667 ms, 417 and 250 ms correspond to 12, 8, 5 and 3 flickers (white-black-white), respectively. Though the amplitude of OSP as well as SSVEP features attenuated with the decrease of interval of missing events, previous studies demonstrated that OSPs can be elicited by two repetitive flickers with one missing events [38, 41, 42]. On the other hand, SSVEP features were still recognizable in the time domain. Thus, we set the interval of missing events to 3 flickers, i.e., two repetitive flickers followed by one missing events.

Fig. 8
figure 8

EEG responses with different interval of missing events

BCI performances

Experiments were carried out with optimized parameters discussed above. The onset time of first missing event of disc 1, 2, 3, and 4 occurred at 467, 450, 633 and 650 ms, respectively. Previous studies reported ERPs such as P300 can be well extracted with averaging 8 times [14, 23, 32, 55]. In our study, we set the averaging times to 2, 4, 6, and 8 for both offline training and online testing. Results in Table 1 demonstrated each subject’s highest ITR under optimal averaging times and mean accuracy, with two different missing events patterns. The online accuracy and ITR (mean ± standard deviation) over nine healthy subjects were 79.29 ± 18.14 % and 19.45 ± 11.99 with missing black disc pattern, and 86.82 ± 12.91 % and 24.06 ± 10.95 with missing white disc pattern, respectively. With missing black and white disc pattern, six and seven out of nine subjects exceeded the level of 80 % mean accuracy and 15 bits/min ITR. No significant difference was found between two missing flicker patterns [Accuracy: F(1,16) = 0.727, p = 0.406; ITR: F(1,16) = 1.058, p = 0.319, one-way ANOVA].

Table 1 Online accuracy and ITR statistics with optimal parameters

In order to demonstrate the advantage of OSP to BCI performances, we further compared the proposed SSVEP-OSP paradigm and traditional SSVEP paradigm with two stimulators. The comparison is fair since both paradigms utilized two frequencies to design visual stimulators. One of the main advantages of the proposed SSVEP-OSP paradigm is that more stimulators can be presented with certain stimulus frequencies. Results shown in Fig. 9 suggested significantly higher ITR with SSVEP-OSP paradigm than that with traditional SSVEP paradigm [F(1,34) = 8.882, p = 0.005, one-way ANOVA]. All the subjects achieved better ITRs with SSVEP-OSP paradigm except one subject (S7).

Fig. 9
figure 9

ITR comparison between SSVEP-OSP paradigm and SSVEP paradigm

Moreover, we carried out experiments to demonstrate the effect of interval of missing events on the performance of the proposed paradigm. Interval of missing events was set to 12, 8, 5, 3 and 1 flicker(s) (white-black-white), respectively. Results were shown in Fig. 10. With increasing interval of missing events, the detection accuracy of SSVEP increased significantly (from 84.38 – 100 %). When the interval of missing events decreased to 1 flicker, the OSP pattern cannot be elicited. As a result, the detection accuracy of OSP dropped to random level (54.81 %, two stimulators) and ITR also decreased significantly (10.76 bits/min). Highest ITR (29.14 bits/min) was achieved when the interval of missing events was 3 flickers.

Fig. 10
figure 10

Performance with different interval of missing events

Discussion

The novel hybrid BCI paradigm proposed in this study is based on simultaneous elicitation of SSVEP and OSP using repetitive visual stimuli with missing events. As suggested in previous studies, OSP is one type of P300 that elicited with omitted-stimulus paradigm [14, 56]. Here, we use “OSP” rather than “P300” for differentiation because most studies developed P300-based BCI based on oddball paradigm [5, 8, 2124, 34, 55]. Typical hybrid BCIs based on SSVEP and oddball-P300 were proposed in [21, 22, 34, 55]. Xu et al. [34] developed a hybrid BCI paradigm combining P300 and SSVEP blocking feature. Therein, the SSVEP blocking feature is similar with the OSP in our study. However, the authors focused on the comparison of hybrid BCI and P300-BCI. The influence of several key parameters on the BCI performances, such as stimuli frequency of SSVEP and duration of SSVEP-block, was not concerned. These questions were discussed in detail in our study. Combaz et al. [55] proposed a hybrid BCI paradigm combining oddball-P300 and SSVEP. Background flickers at 12 or 15Hz were used to elicit SSVEP continuously and traditional oddball stimulus was presented 500 ms after SSVEP started. Two types of visual stimulus, i.e., repetitive stimulus with fixed frequency followed by oddball stimulus, were used to elicit SSVEP and P300 in time order, which is different from the proposed paradigm in our study. On the other hand, previous studies indicated that oddball-P300 required attention [5759]. Fast OSPs in our study required fixation but not attention [42], which is superior to oddball-P300 paradigm. In the code-modulated visual evoked potential (c-VEP) paradigm, a pseudorandom sequence modulated by different time lag was used to construct target stimuli [60, 61]. A VEP template for the specified target stimulator can be obtained by averaging the EEG data from multiple stimulus cycles. All the target stimuli have similar VEP templates with different time lag. CCA, a template matching method in the time domain, was used for target identification. Comparatively, EEG features in both time and frequency domain were used to detect different target stimulators in our hybrid BCI paradigm, where SSVEP and OSP were identified in the frequency and time domain, respectively. One interesting point is that the VEP template in c-VEP paradigm is much more complicated than the SSVEP template in our hybrid paradigm, though CCA were used for template matching in both paradigms.

In the proposed hybrid BCI paradigm, online experiments across nine healthy subjects suggested that mean accuracy and ITR with missing black disc pattern is comparatively higher than those with missing white disc pattern, but the effect is not statistically significant [Accuracy: F(1,16) = 0.727, p = 0.406; ITR: F(1,16) = 1.058, p = 0.319, one-way ANOVA]. One possible explanation is that the elicited OSP with missing white disc pattern has higher amplitude and can be better recognized with the proposed algorithm. As a preliminary study of hybrid BCI paradigm based on SSVEP and OSP, one possible limitation is that we only present four visual stimulators on the screen, which limits the ITR of the proposed BCI paradigm. When comparing the proposed SSVEP-OSP paradigm with traditional SSVEP paradigm, we used the same time to calculate ITR for both paradigms. For subject S1 and S2, the optimal averaging times is 4, which corresponds to 2 s to output a command for both SSVEP-OSP and traditional SSVEP paradigm (1 s for visual stimulus and 1 s between two trials). For subject S3 to S9, the optimal averaging times is 6, which corresponds to 2.5 s to output a command (1.5 s for visual stimulus and 1 s between two trials). In our previous study, oscillating Newton’s rings were presented 4 s as a single trial to elicit SSVEP [7]. Other researchers also reported 1.75-4 s to detect SSVEP with satisfied performance [911]. So the time 2–2.5 s to output a command in the present study is fair for SSVEP comparison.

Conclusions

In summary, we proposed a novel hybrid BCI based on SSVEP and OSP in this study. The proposed BCI paradigm is composed of four discs flickering from black to white with different stimulation frequency and/or different onset time of missing event. The online accuracy and ITR (mean ± standard deviation) over nine healthy subjects were 79.29 ± 18.14 % and 19.45 ± 11.99 bits/min with missing black disc pattern, and 86.82 ± 12.91 % and 24.06 ± 10.95 bits/min with missing white disc pattern, respectively. The main contribution of this study is to simultaneously elicit SSVEP and OSP by repetitive stimuli with missing events, and detect them in frequency and time domain in real-time, respectively. We discussed the optimal parameters in the new paradigm and proposed feasible algorithms for feature extraction and recognition. Besides the frequency features such as SSVEP elicited by repetitive stimuli in previous studies, we explored a new feature (OSP) in the time domain by adding missing events in repetitive stimuli. By adjusting the onset time of first missing events and interval of missing events, the recognizable target stimulators of the proposed hybrid BCI paradigm can be enhanced significantly. If the mean detection accuracy can maintain above 80 %, there will be large improvement on the ITR of the SSVEP-OSP paradigm. These are what we will focus on next.

Abbreviations

BCI, brain computer interface; CCA, canonical correlation analysis; ERD, event-related desynchronization; ERS, event-related synchronization; ITR, information transfer rate; OSP, omitted stimulus potential; SMR, sensorimotor rhythm; SSVEP, steady state visually evoked potential; SVM, support vector machine

References

  1. Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM. Brain-computer interfaces for communication and control. Clin Neurophysiol. 2002;113:767–91.

    Article  PubMed  Google Scholar 

  2. Do AH, Wang PT, King CE, Chun SN, Nenadic Z. Brain-computer interface controlled robotic gait orthosis. J Neuroeng Rehabil. 2013;10:111.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Kaufmann T, Herweg A, Kubler A. Toward brain-computer interface based wheelchair control utilizing tactually-evoked event-related potentials. J Neuroeng Rehabil. 2014;11:7.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Treder MS, Bahramisharif A, Schmidt NM, van Gerven MAJ, Blankertz B. Brain-computer interfacing using modulations of alpha activity induced by covert shifts of attention. J Neuroeng Rehabil. 2011;8:24.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Middendorf M, McMillan G, Calhoun G, Jones KS. Brain-computer interfaces based on the steady-state visual-evoked response. IEEE Trans Neural Syst Rehabil Eng. 2000;8:211–4.

    Article  CAS  Google Scholar 

  6. Gao S, Wang Y, Gao X, Hong B. Visual and auditory brain-computer interfaces. IEEE Trans Biomed Eng. 2014;61:1436–47.

    Article  PubMed  Google Scholar 

  7. Xie J, Xu GH, Wang J, Zhang F, Zhang YZ. Steady-state motion visual evoked potentials produced by oscillating Newton’s rings: implications for brain-computer interfaces. PLoS One. 2012;7:e39707.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Vialatte FB, Maurice M, Dauwels J, Cichocki A. Steady-state visually evoked potentials: focus on essential paradigms and future perspectives. Prog Neurobiol. 2010;90:418–38.

    Article  PubMed  Google Scholar 

  9. Zhang Y, Zhou G, Jin J, Wang M, Wang X, Cichocki A. L1-regularized multiway canonical correlation analysis for SSVEP-based BCI. IEEE Trans Neural Syst Rehabil Eng. 2013;21:887–96.

    Article  CAS  PubMed  Google Scholar 

  10. Diez P, Mut V, Perona E, Leber E. Asynchronous BCI control using high-frequency SSVEP. J Neuroeng Rehabil. 2011;8:39.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Zhang Y, Zhou G, Jin J, Wang X, Cichocki A. Frequency recognition in SSVEP-based BCI using multiset canonical correlation analysis. Int J Neural Syst. 2014;24:1072–92.

    Google Scholar 

  12. Lin Y, Wang Y, Jung T. Assessing the feasibility of online SSVEP decoding in human walking using a consumer EEG headset. J Neuroeng Rehabil. 2014;11:119.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Zhang Y, Zhou G, Jin J, Wang X, Cichocki A. SSVEP recognition using common feature analysis in brain-computer interface. J Neurosci Methods. 2015;244:8–15.

    Article  PubMed  Google Scholar 

  14. Tarkka IM, Stokic DS. Source Localization of P300 from Oddball, Single Stimulus, and Omitted-Stimulus Paradigms. Brain Topogr. 1998;11:141–51.

    Article  CAS  PubMed  Google Scholar 

  15. Blankertz B, Lemm S, Treder M, Haufe S, Müller KR. Single-trial analysis and classification of ERP components - a tutorial. Neuroimage. 2011;56:814–25.

    Article  PubMed  Google Scholar 

  16. Zhang Y, Zhou G, Jin J, Zhao Q, Wang X, Cichocki A. Aggregation of sparse linear discriminant analysis for event-related potential classification in brain-computer interface. Int J Neural Syst. 2014;24:263–74.

    Google Scholar 

  17. Hoffmann U, Vesin JM, Ebrahimi T, Diserens K. An efficient P300-based brain-computer interface for disabled subjects. J Neurosci Methods. 2008;167:115–25.

    Article  PubMed  Google Scholar 

  18. Huang YH, Erdogmus D, Pavel M, Mathan S, Hild II KE. A framework for rapid visual image search using single trial brain evoked responses. Neurocomputing. 2011;74:2041–51.

    Article  Google Scholar 

  19. Lan T, Huang C, Erdogmus D. A comparison of temporal windowing schemes for single-trial ERP detection. Antalya, Turkey: Proceedings 4th Int IEEE EMBS Conf Neural Eng; 2009.

    Book  Google Scholar 

  20. Zhang Y, Zhou G, Jin J, Zhao Q, Wang X, Cichocki A. Sparse Bayesian classification of EEG for brain-computer interface. IEEE Trans. Neural Netw. Learn Syst. 2015. doi:10.1109/TNNLS.2015.2476656.

  21. Yin E, Zhou Z, Jiang J, Chen F, Liu Y, Hu D. A novel hybrid BCI speller based on the incorporation of SSVEP into the P300 paradigm. J Neural Eng. 2013;2013:026012.

    Article  Google Scholar 

  22. Yin E, Zeyl T, Saab R, Chau T, Hu D, Zhou Z. A hybrid brain-computer interface based on the fusion of P300 and SSVEP scores. IEEE Trans Neural Syst Rehabil Eng. 2015;23:693–701.

    Article  PubMed  Google Scholar 

  23. Riccio A, Simione L, Schettini F, Pizzimenti A, Inghilleri M, Belardinelli MO, Mattia D, Cincotti F. Attention and P300-based BCI performance in people with amyotrophic lateral sclerosis. Front Hum Neurosci. 2013;7:732–40.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Li YQ, Pan JH, Wang F, Yu ZL. A hybrid BCI system combing P300 and SSVEP and its application to wheelchair control. IEEE Trans Biomed Eng. 2013;60:3156–66.

    Article  PubMed  Google Scholar 

  25. Pfurtscheller G, Allison BZ, Brunner C, Bauernfeind G, Solis-Escalante T, Scherer R, Zander TO, Mueller-Putz G, Neuper C, Birbaumer N. The hybrid BCI. Front Neurosci. 2010;4:30–40.

    PubMed  Google Scholar 

  26. Yao L, Meng J, Zhang D, Sheng X, Zhu X. Combining motor imagery with selective sensation toward a hybrid-modality BCI. IEEE Trans Biomed Eng. 2014;61:2304–12.

    Article  PubMed  Google Scholar 

  27. Ferrez PW, Millian JDR. Error-related EEG potentials generated during simulated brain-computer interactions. IEEE Trans Biomed Eng. 2008;55:923–9.

    Article  PubMed  Google Scholar 

  28. Allison BZ, Brunner C, Kaiser V, Mullerputz GR, Neuper C, Pfurtscheller G. Toward a hybrid brain-computer interface based on imagined movement and visual attention. J Neural Eng. 2010;7:026007.

    Article  CAS  Google Scholar 

  29. Brunner C, Allison BZ, Krusienski DJ, Mullerputz GR, Pfurtscheller G, Neuper C. Improved signal processing approaches in an offline simulation of a hybrid brain-computer interface. J Neurosci Methods. 2010;188:165–73.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Long JY, Li YQ. Target selection with hybrid feature for BCI-Based 2-D cursor control. IEEE Trans Biomed Eng. 2012;59:132–40.

    Article  PubMed  Google Scholar 

  31. Edlinger G, Holzner C, Guger C. A hybrid brain computer interface for smart home control. Lect Note Comput Sci. 2011;6762:417–26.

    Article  Google Scholar 

  32. Panicker RC, Puthusserypady S, Sun Y. An asynchronous P300 BCI with SSVEP-based control state detection. IEEE Trans Biomed Eng. 2011;58:1781–8.

    Article  PubMed  Google Scholar 

  33. Yong XY, Fatourechi M, Ward RK, Birch GE. Automatic artefact removal in a self-paced hybrid brain-computer interface system. J Neuroeng Rehabil. 2012;9:50.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Xu MP, Qi HZ, Wan BK, Yin T, Liu ZP, Ming D. A hybrid BCI speller paradigm combing P300 potential and the SSVEP blocking feature. J Neural Eng. 2013;10:026001.

    Article  PubMed  Google Scholar 

  35. Rusinov VS. General and localized alternations in the electroencephalogram during the formation of conditioned reflexes in man. Electroencephalog Clin Neurophysiol. 1960;13:309–19.

    Google Scholar 

  36. Klinke R, Fruhstorfer H, Finkenzeller P. Evoked responses as a function of external and stored information. Electroencephalog Clin Neurophysiol. 1968;25:119–22.

    Article  CAS  Google Scholar 

  37. Jones SJ. AEPs at the onset and offset of repetitive sound modulation, due to mismatch with the contents of an auditory sensory store. Electroencephalog Clin Neurophysiol. 1992;84:149–56.

    Article  CAS  Google Scholar 

  38. Schwartz G, Harris R, Shrom D, Berry MJ. Detection and prediction of periodic patterns by the retina. Nature Neurosci. 2007;10:552–4.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  39. Gao XR, Xu DF, Chen M, Gao SK. A BCI-based environmental controller for the motion-disabled. IEEE Trans Neural Syst Rehabil Eng. 2003;11:137–40.

    Article  PubMed  Google Scholar 

  40. Bullock TH, Hofmann MH, Nahm FK, New JG, Prechtl JC. Event-related potentials in the retina and optic tectum of fish. J Neurophysiol. 1990;64:903–14.

    CAS  PubMed  Google Scholar 

  41. Prechtl JC, Bullock TH. Event-related potentials to omitted visual stimuli in a reptile. Electroencephalogr Clin Neurophysiol. 1994;91:54–66.

    Article  CAS  PubMed  Google Scholar 

  42. Bullock TH, Karamursel S, Achimowicz JZ, Mcclune MC, Basareroglu C. Dynamic properties of human visual evoked and omitted stimulus potentials. Electroencephalogr Clin Neurophysiol. 1994;91:42–53.

    Article  CAS  PubMed  Google Scholar 

  43. Hernandez OH, Vogel-Sprott M. The omitted stimulus potential is related to the cognitive component of reaction time. Int J Neurosci. 2008;118:173–83.

    Article  PubMed  Google Scholar 

  44. Brainard DH. The psychophysics toolbox. Spat Vis. 1997;10:433–6.

    Article  CAS  PubMed  Google Scholar 

  45. Pelli DG. The video toolbox software for visual psychophysics: transforming numbers into movies. Spat Vis. 1997;10:437–42.

    Article  CAS  PubMed  Google Scholar 

  46. Heinrich SP. A primer on motion visual evoked potentials. Doc Ophthalmol. 2007;114:83–105.

    Article  PubMed  Google Scholar 

  47. Heinrich SP, Bach M. Adaptation characteristics of steady-state motion visual evoked potentials. Clin Neurophysiol. 2003;114:1359–66.

    Article  PubMed  Google Scholar 

  48. Hotelling H. Relations between two sets of variates. Biometrika. 1936;28:321–77.

    Article  Google Scholar 

  49. Lin ZL, Zhang CS, Wu W, Gao XR. Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs. IEEE Trans Biomed Eng. 2007;54:1172–6.

    Article  PubMed  Google Scholar 

  50. Bin GY, Gao XR, Yan Z, Hong B, Gao SK. An online multi-channel SSVEP-based brain-computer interface using a canonical correlation analysis method. J Neural Eng. 2009;6:046002.

    Article  PubMed  Google Scholar 

  51. Zhang Y, Zhou GX, Zhao QB, Onishi A, Jin J, Wang XY, Cichocki A. Multiway canonical correlation analysis for frequency components recognition in SSVEP-based BCIs. Lect Note Comput Sci. 2011;7062:287–95.

    Article  Google Scholar 

  52. Wang DF, Yeung DS, Tsang EC. Weighted mahalanobis distance kernels for support vector machines. IEEE Trans Neural Network. 2007;18:1453–62.

    Article  Google Scholar 

  53. Machado J, Balinot A. Executed movement using EEG signals through a Naive Bayes classifier. Micromachines. 2014;5:1082–105.

    Article  Google Scholar 

  54. Oostenveld R, Fries P, Maris E, Schoffelen JM. FieldTrip: open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Comput Intell Neurosci. 2011;2011:156869.

    Article  PubMed  Google Scholar 

  55. Combaz A, van Hulle MM. Simultaneous detection of P300 and steady-state visually evoked potentials for hybrid brain-computer interface. PLoS One. 2015;10:e0121481.

    Article  PubMed  PubMed Central  Google Scholar 

  56. McCullagh J, Weihing J, Musiek F. Comparisons of P300s from standard oddball and omitted paradigms: implications to exogenous/endogenous contributions. J Am Acad Audiol. 2009;20:187–95.

    Article  PubMed  Google Scholar 

  57. Gray HM, Ambady N, Lowenthal WT, Deldin P. P300 as an index of attention to self-relevant stimuli. J Exp Soc Psychol. 2004;40:216–24.

    Article  Google Scholar 

  58. Linden DEJ. The P300: where in the brain is it produced and what does it tell us? Neuroscientist. 2005;11:563–76.

    Article  CAS  PubMed  Google Scholar 

  59. Kevin MS, John P. Poststimulus EEG spectral analysis and P300: attention, task and probability. Psychophysiology. 1999;36:220–32.

    Article  Google Scholar 

  60. Bin G, Gao X, Wang Y, Li Y, Hong B, Gao S. A high-speed BCI based on code modulation VEP. J Neural Eng. 2011;8:587–9.

    Article  Google Scholar 

  61. Spuler M, Rosenstiel W, Bogdan M. Online adaptation of a c-VEP brain-computer interface (BCI) based on error-related potentials and unsupervised learning. PLoS One. 2012;7:e51077.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgments

The authors would like to thank the subjects for participating in these experiments.

Funding

This study was supported by Hong Kong Scholarship Program (XJ2013045) and China Postdoctoral Science Foundation Project (2014 M552431). No additional external funding was received for this study. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Availability of data and materials

Data will not be shared at this stage. Further studies on these data are ongoing.

Authors’ contributions

YYW designed and performed experiments and analyzed data, and wrote the paper; ML performed experiments and analyzed data. JW designed and performed experiments, and wrote the paper. All authors reviewed and approved the manuscript.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Subjects were studied after giving informed written consent in accordance with a protocol approved by the institutional review board of Xi’an Jiaotong University.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jing Wang.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, Y., Li, M. & Wang, J. Toward a hybrid brain-computer interface based on repetitive visual stimuli with missing events. J NeuroEngineering Rehabil 13, 66 (2016). https://doi.org/10.1186/s12984-016-0179-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12984-016-0179-9

Keywords