- Open Access
Single-trial classification of NIRS signals during emotional induction tasks: towards a corporeal machine interface
Journal of NeuroEngineering and Rehabilitation volume 6, Article number: 39 (2009)
Corporeal machine interfaces (CMIs) are one of a few available options for restoring communication and environmental control to those with severe motor impairments. Cognitive processes detectable solely with functional imaging technologies such as near-infrared spectroscopy (NIRS) can potentially provide interfaces requiring less user training than conventional electroencephalography-based CMIs. We hypothesized that visually-cued emotional induction tasks can elicit forehead hemodynamic activity that can be harnessed for a CMI.
Data were collected from ten able-bodied participants as they performed trials of positively and negatively-emotional induction tasks. A genetic algorithm was employed to select the optimal signal features, classifier, task valence (positive or negative emotional value of the stimulus), recording site, and signal analysis interval length for each participant. We compared the performance of Linear Discriminant Analysis and Support Vector Machine classifiers. The latency of the NIRS hemodynamic response was estimated as the time required for classification accuracy to stabilize.
Baseline and activation sequences were classified offline with accuracies upwards of 75.0%. Feature selection identified common time-domain discriminatory features across participants. Classification performance varied with the length of the input signal, and optimal signal length was found to be feature-dependent. Statistically significant increases in classification accuracy from baseline rates were observed as early as 2.5 s from initial stimulus presentation.
NIRS signals during affective states were shown to be distinguishable from baseline states with classification accuracies significantly above chance levels. Further research with NIRS for corporeal machine interfaces is warranted.
Access technologies currently available for locked-in individuals are largely limited to corporeal machine interfaces (CMIs), particularly brain-computer interfaces (BCIs) based on electroencephalography (EEG) . EEG has been popular in BCI research owing to its high temporal resolution and non-invasiveness. However, EEG has drawbacks including, but not limited to, its steep learning curve , and susceptibility to electrical interference from environmental and physiological sources . Consequently, research efforts have been made towards investigating alternative modalities for brain-computer interfacing. Studies have identified a correlation between cerebral hemodynamic changes - in the form of localized increases in blood flow and oxygen consumption - and electric brain activity . Weiskopf et al. reported on the first BCI based on the blood oxygen level-dependent (BOLD) response measured by functional magnetic resonance imaging (fMRI) . With real-time fMRI feedback, individuals can learn to voluntarily elicit activation in a variety of cortical and subcortical areas [6–8]. Clinical application of a fMRI-BCI is currently impractical due to prohibitive costs and technological limitations . An alternative approach is to measure cerebral and corporeal hemodynamics with near-infrared spectroscopy (NIRS). NIRS is suitable for measuring functional activation in cortical regions 1-3 cm beneath the scalp. The dominant chromophores in the NIR range are oxygenated (HbO) and deoxygenated hemoglobin (Hb), both of which are biologically relevant markers for brain function. Furthermore, water and biological tissue are weak absorbers of light at NIR wavelengths (700-1000 nm) . These factors combine to create an "optical window" through which changes in tissue oxygenation can be monitored. A NIRS instrument consists of light sources by which a tissue volume of interest is irradiated, and detectors that receive light after its interaction with tissue. As a general rule of thumb, light penetration depth is approximately one-half of the distance between a source and a detector . Regardless of penetration distance however, extracerebral blood flow in the superficial tissue typically contributes significantly to NIRS measurements .
NIR light undergoes absorption as it penetrates biological tissue; measurements from NIRS instruments yield a response associated with brain activity attributed to this interaction effect. The slow hemodynamic response manifests itself as a small increase in Hb after the onset of neural activity, subsequently followed by a large but delayed increase in HbO peaking at approximately 10 s [13, 14] after activation and a corresponding decrease in Hb . Changes in the concentrations of oxygenated (Δ[HbO]) and deoxygenated hemoglobin (Δ[Hb]) can be calculated from changes in detected light intensity using the modified Beer-Lambert Law .
Unlike other functional imaging methods, NIRS does not restrict range of motion and has been used to monitor cortical activation in real-world settings [16–18]. NIRS is immune to electrical interference from environmental sources as well as ocular and muscle artifacts . Furthermore, NIRS measurement systems are commercially available at a comparable cost to EEG systems.
Studies on NIRS-BCIs to date have focused on classifying mean amplitude changes in the hemodynamic response induced by mental tasks with well-established psychophysiological bases. Using a 20-channel commercial NIRS measurement system, Sitaram et al.  performed offline classification of left-handed/right-handed motor imagery data using amplitude changes in [O2Hb] and [HHb] as the class discriminatory features. A maximum accuracy of 89% was achieved using a Hidden Markov Model (HMM). Coyle et al.  performed evaluations of a single-channel NIRS system. Able-bodied individuals controlled a binary switch by modulating changes in [O2Hb] over the motor cortex and achieved 50-85% accuracy in online trials. Naito et al.  investigated the use of high-level cognitive tasks for BCI. Measurements were recorded over the prefrontal cortex with a single-channel, single-wavelength NIRS system. Seventeen locked-in individuals were requested to perform different mental tasks corresponding to 'yes' and 'no' in response to a series of questions. An average offline classification accuracy of 80% was achieved in 40% of the locked-in participants using a non-linear discriminant classifier.
The ultimate goal of a corporeal machine interface is to translate functional intent into a corresponding action. A large body of evidence supports the view that the prefrontal cortex (PFC) plays a central role in cognitive control, the ability to translate thought into action to accomplish a given objective . In particular, functional NIRS (fNIRS) studies have found that changes in affective state generated by emotional induction tasks can elicit activation in the PFC [24–26]. Valenced images have been shown to stimulate changes in prefrontal hemodynamics detectable with NIRS . If emotional induction tasks can consistently generate distinct patterns in the NIRS hemodynamic response, they may be useful in an NIRS corporeal machine interface as a preference detector. In particular, one might be able to use NIRS with nonverbal individuals to distinguish between naturally occurring positive and negative emotional responses to sequentially presented visual stimuli.
Our primary objective was to ascertain the feasibility of using visually-cued emotional induction tasks as a corporeal machine interface mechanism. Several aspects of signal analysis and classification were addressed in realizing this objective, namely 1) artifact removal; 2) feature selection; and 3) classifier selection. The effects of various parameters on classification performance were explored by performing feature selection searches over different task valences, recording sites, and signal analysis window lengths. To our knowledge, this is the first time that feature selection has been used to optimize NIRS signal classification rates. To examine whether or not NIRS data can be represented as linearly separable feature subsets, we compared the offline performance of Linear Discriminant Analysis (LDA) and Support Vector Machines (SVM). Lastly, classification performance was employed as a measure to quantify the latency of the prefrontal hemodynamic response to emotional induction tasks. Note that we use the term corporeal interface to acknowledge that NIRS measurements typically encompass both cortical and superficial tissue blood flow contributions.
Ten individuals (5 females, mean age 28.4 ± 6.4 years) participated in the study. Participants had normal or corrected-to-normal vision, and no known indication of the following: 1) degenerative disorders; 2) cardiovascular disorders; 3) metabolic disorders; 4) trauma-induced brain injury; 5) respiratory conditions; 6) drug and alcohol-related conditions; and 7) psychiatric disorders. The aforementioned disorders are known to cause impaired mental function, which may compromise the integrity of collected data. The study was approved by Bloorview Kids Rehab and the University of Toronto Research Ethics Boards. Written consent was obtained from all participants.
NIRS measurements were collected with an ISS Imagent (Champaign, IL) functional brain imaging system. Frequency-modulated light at two wavelengths (690 nm and 830 nm) was delivered to the scalp via two-fibre optic bundles ("source pairs") and collected via different fibre-optic bundles ("detectors"). Sources and detectors were held in place with a soft helmet designed to measure over the prefrontal cortex behind the forehead. Its frame, fabricated from a 0.16 cm thick low-density polyethylene, consisted of an adjustable circumference band with a flexible probe overlaying the forehead. Fibres were affixed to the helmet through holes punched in the probe; holes were situated 1.5 cm apart, creating a uniformly spaced grid.
Each side of the prefrontal cortex was interrogated with four pairs of sources and a detector arranged as depicted in Figure 1 for a total of 16 source-detector channels. The arrangement was placed over each participant's frontal lobe with the most anterior row of sources positioned along the PF1-PF2 line (International 10/20 Electrode system ). One recording site was formed between each source pair and its adjacent detector. A multiplexer controlled the sequencing of sources such that no two sources were on simultaneously. The time needed to cycle once through all 16 sources was 32 ms, corresponding to a sampling rate of 31.25 Hz.
Source-detector separation distances were fixed at 2.1 cm after preliminary testing on a subset of participants. We quantified the similarity between NIRS signals recorded over 2.1 cm and 3.0 cm, a commonly employed separation distance for fNIRS studies. Signal pairs recorded over the two distances exhibited high correlation values, and it was visually verified that attenuated, but measurable, changes in light attenuation were discernible in signals recorded over 2.1 cm.
Respiration was simultaneously recorded using a piezoelectric respiratory effort belt secured around the participant's chest. Data from this auxiliary transducer were sampled at 60 Hz.
Participants performed trials of an emotional induction task. In a trial, the participant was instructed to rehearse an emotion that he/she associates with the contents of each image for the duration of its presentation. Data collection took place in a dimly lit room. The participant sat in a chair placed approximately 1 m from a LCD monitor and was asked to relax and restrict head movement. A trial consisted of a baseline sequence, a task sequence, and a rest sequence (Fig. 2). Each trial began with a 30 s baseline sequence, during which the participant was instructed to relax and focus his/her gaze on a fixation dot presented at the centre of the screen. The participant then performed the task as prompted on the screen for 10 s. The trial then concluded with a 20 s rest sequence to allow for any activation-induced hemodynamic response to subside. During this post-task rest period, the participant was again instructed to focus on the fixation dot on the screen. Trials were self-paced so that the participant could take short breaks as required.
The participant performed the above emotion induction task in response to 2 stimuli: a pair of valenced images from the International Affective Picture system (IAPS) . Prior to data collection, the participant attended a screening session where he/she performed 5 instances of the emotional induction task for each picture from a stimulus pool of 10 IAPS images. The pool was comprised of 5 images rated for high arousal and positive valence (valence = 7.52 ± 1.53, arousal = 6.37 ± 2.33) and 5 images rated for high arousal and negative valence (valence = 2.94 ± 1.71, arousal = 6.52 ± 2.13). The selected images were IAPS items 8501, 8499, 8080, 8190, 8341, 6313, 1525, 8485, 9622, and 1930. After converting raw light intensity data to changes in attenuation (optical density), each image was ranked based on its relative ability to consistently generate changes in optical density across multiple recording sites. From this preliminary analysis, a positive/negative-valence pairing was selected for the classification problem. At the beginning of the session, the participant viewed a self-paced slideshow of images to be presented and was instructed to familiarize himself/herself with each image's contents. The participant completed 6 practice trials to acquaint himself or herself with the task. He/she then performed 30 trials of the emotional induction task for each image of the positive/negative-valence pair in 10 6-trial blocks. Images were presented in randomized order. To alleviate fatigue, halfway through the session a 10-minute break was imposed where the participant was asked to vacate the testing area.
Concentration changes in oxygenated and deoxygenated hemoglobin, denoted respectively as Δ[HbO] and Δ[Hb], were calculated at each of the 8 recording sites from changes in detected light attenuation using the modified Beer-Lambert Law before undergoing artifact removal. The modified Beer-Lambert law states that changes in optical density (ΔOD) can be calculated from a measured change in light attenuation before and after a test condition:
where I B and I A represent light intensity measured under mean baseline and activation conditions, respectively, for the problem of interest. ΔOD is proportional to the extinction coefficient for molar concentrations of the light-absorbing compound (ϵ), the concentration of the compound (c), and optical path length. The optical path length is expressed as a product of source-detector distance r and a multiplier known as the differential pathlength factor (DPF), which is a function of the extinction coefficient of the scattering medium .
Total changes in light attenuation are expressed as a linear sum of contributions from each absorbing compound. Since the primary absorbers of NIR light in cerebral tissue are HbO and Hb, (1) can be expanded as:
where ODλequals optical density at wavelength λ, and are the extinction coefficients for HbO and Hb at λ, and DPFλis the differential pathlength factor for the adult human head at λ. It follows that Δ[HbO] and Δ[Hb] can be determined by calculating changes in optical density at two wavelengths, λ1 and λ2. Solving the system of equations obtains Δ[HbO] and Δ[Hb]:
We used literature values for DPF  and ϵ at the relevant wavelengths  to calculate Δ[HbO] and Δ[Hb]. At a sampling rate of 31.25 Hz, 1875 delta concentration values were obtained for each of HbO and Hb during one 60 s trial of the emotional induction task.
Adaptive noise cancellation has been shown to be effective in removing artifacts from EEG and fMRI brain recordings [31, 32]. Some research groups have employed the technique to remove physiological artifacts from NIRS recordings [33, 34]. We used a least-mean squares (LMS) adaptive filter to remove respiratory artifacts from the hemodynamic signals. Each respiratory signal was first resampled at 31.25 Hz and synchronized to its corresponding hemodynamic signal via a b-spline curve registration procedure . We implemented landmark-based registration based on the alignment of local maxima and minima found in each pair of signals. To facilitate landmark estimation in the hemodynamic signal, signal components over the frequency range of interest were isolated; as such, Δ[HbO] and Δ[Hb] signals were filtered using a 0.4-1 Hz bandpass filter prior to registration. The respiratory signal was then registered to the filtered hemodynamic signal. An adaptive filter with 200 taps was used, and the step size was set to 0.001. Both values were empirically determined. It was noted that at a 31.25 Hz sampling rate 200 taps corresponds to 6.4 s (approximately 2 breaths), which is sufficiently long for modelling the characteristics of the respiratory signal.
Systemic low-frequency oscillations in the hemodynamic signal believed to arise from regional cerebral blood flow  are centered around 0.1 Hz . We filtered out these vasomotion effects using a 3rd order Butterworth filter with a 0.05-0.15 Hz passband. Arterial pulsatility due to systole and diastole are visibly manifested as a series of periodic spikes superimposed over the slowly evolving hemodynamic response. A 30-point moving average filter, which corresponds to data spanning over approximately 1 s, was applied to reduce cardiac effects prior to feature extraction.
Feature selection and classification
Δ[HbO] and Δ[Hb] signals were segmented into baseline and activation intervals to form two sets of 60 (30 baseline, 30 activation) trials for each stimulus. The transition point between the baseline and activation intervals was set as the time of initial stimulus presentation. Six time-domain and seven time-frequency domain features for classification were calculated for Δ[HbO] and Δ[Hb] signals for each trial over each recording site:
Mean: average signal value.
Variance: measure of signal spread.
ZC: Zero Crossings; number of instances where the signal crossed the zero line.
RMS: Root Mean Squared; measure of average signal magnitude.
Skewness: measure of the asymmetry of signal values around its mean relative to a normal distribution.
Kurtosis: measure of the degree of peakedness of a distribution of signal values relative to a normal distribution.
E a : percentage of total signal energy contributed by the approximation signal from a 6-level wavelet decomposition (Daubechies 4) of the time-domain signal.
E dX : percentage total signal energy contributed by each detail signal from a 6-level wavelet decomposition (Daubechies 4) of the time-domain signal. Six percentages were extracted, one for each level of decomposition (X = 1,...,6). Given the length of the signal input, the nominal maximum number of levels for a wavelet decomposition using a Daubechies 4 wavelet is six.
208 candidate features (13 features × 2 signals × 8 sites) were thus calculated for each participant. Research groups to date have primarily focused on classifying NIRS data using mean changes in hemoglobin concentration as a discriminatory feature [20, 21]. In the present study, a large number of candidate features were introduced to the classification problem in an attempt to better characterize the space of possible features (i.e. search space), which contains a number of irrelevant or redundant features for classification. Feature subsets were selected for the classification task. Given the number of trials collected (60), only a two-dimensional feature space was justified. Feature selection was conducted for each participant using all combinations of the following performance parameters for each of the two classifiers of interest:
Task Valence (Positive/Negative): We hypothesized that classification performance correlates positively with subjective evaluation of task difficulty. If a participant finds it easier to perform one of the emotional induction tasks over the other - that is, associate emotions more strongly with one of the visual cues in the pairing - the data from the task may yield higher classification rates.
Recording Sites (Right Prefrontal/Left Prefrontal): We hypothesized that task valence correlates with optimal recording site according to the valence hypothesis, which posits that positive emotions are left-lateralized and that negative emotions are right-lateralized .
Analysis interval (15 s/20 s): We hypothesized that the optimal analysis interval is feature-dependent. We selected time intervals over which signal differences between baseline and activation states were expected to be observed given that the hemodynamic response peaks about 10 s from the start of the task [13, 14]. Therefore, we compared classifier performance using features calculated over analysis time intervals of 15 s and 20 s.
All combinations of classifiers, task valences, recording sites, and analysis interval lengths generated 16 possible feature selection problems.
When appropriately configured, random search algorithms such as genetic algorithms (GAs) allow for the evaluation of a search space more efficiently than most other heuristic search methods  and perform well on noisy search spaces containing local minima . Feature selection was thus performed using a standard GA with a rank-based parent selection strategy, a scattered crossover operator, and a uniform mutation operator (Genetic Algorithm and Direct Search Toolbox, MATLAB).
For each of the 16 problems, 20 runs of the GA were performed with the following parameter settings: 1) population size = 100; 2) number of generations = 30; 3) probability of crossover = 0.6; and 4) probability of mutation = 0.01. Parameter values were selected on the basis of results from several preliminary runs, and align with typical values used in literature . We selected the feature set most frequently converged upon by the GA across the 20 runs. In the event of a tie, the feature set with the higher mean fitness value was selected. The fitness value of each candidate feature subset was defined by its 5-fold cross-validation classification accuracy. A Gaussian radial basis function kernel with unity scaling factor and penalty term was selected for the SVM classifier (Bioinformatics Toolbox, MATLAB).
Ten (10) runs of 5-fold cross-validation were then performed using the optimal feature set selected for each of the 16 problems. Fifty (50) accuracy measures (classification rates) were obtained after 10 runs of 5-fold cross-validation, from which a mean classification rate was calculated. We report the maximum classification rate obtained for each participant, along with corresponding feature set and performance parameter settings.
Quantifying response latency
Classification accuracy was used to quantify when changes from a baseline state can be detected. Using the optimal feature set for each participant, mean classification rates were calculated via 10 runs of 5-fold cross-validation, over a range of analysis interval lengths. The baseline rate was arbitrarily defined as the mean classification accuracy calculated with an analysis interval of size ΔT = 1.0 s. The size of the interval was increased in 0.1 s increments from the transition point to a maximum of ΔT = 20.0 s. The minimum analysis interval length was set based on the number of points required for a 1-level wavelet decomposition using a Daubechies 4 wavelet.
Next, we checked for statistically significant differences between the set of classification accuracies calculated at ΔT = 1.0 s and each set of classification accuracies calculated at ΔT = (1.0 + t) s, where t ranged from 0.1 to 19.0. These results were used to determine a range of analysis interval lengths over which statistically significant activation was detected (Fig. 3):
Mean classification accuracy was plotted as a function of analysis interval size. The accuracies were loess smoothed using a span equal to 20% of the number of data points. Hypothesis test outcome H was also plotted as a function of analysis interval size. H(ΔT) = 1 indicates that a statistically significant difference from baseline accuracy (p < 0.05, corrected resampled t-test) was detected at analysis interval ΔT.
The vector of smoothed accuracies was searched for its maximum value (i.e. maximum classification rate), and its corresponding analysis interval length (ΔT max ) was noted.
To quantify the range of analysis interval lengths with statistically significant activation, two iterative searches were performed forwards and backwards from ΔT max . The mean classification rate at ΔT = v s (0.1 ≤ v ≤ 20) was deemed significantly different from the baseline rate if H = 1 for > 50% of the original (unsmoothed) data points in the range ΔT = v ± 0.5 s. A search was terminated when the aforementioned condition was violated and the termination point marked as a boundary of the range of analysis interval lengths with significant activation.
The feature set and combination of performance parameters that yielded the highest mean classification accuracy for each participant were identified. Table 1 summarizes the results for GA-based feature selection. Features were selected across a range of recording sites, which is not entirely unexpected given NIRS' limited spatial sensitivity. Though [Hb] is thought to be a more reliable indicator of functional activation , the GA selected features derived from Δ[HbO] and Δ[Hb] signals with equal frequency. This implies that among other physiological phenomenon, Δ[HbO] captures valuable information directly correlated with experimentally derived activations and should not be discarded.
Regardless of the classifier of interest, time-domain features, i.e. either one of skewness or mean of Δ[HbO] and Δ[Hb], were consistently selected by the GA as part of the optimal feature pair across and within participants. The aforementioned time-domain features were frequently selected for each participant across the 16 feature selection problems. The GA occasionally selected time-frequency features, and even then, only alongside a time domain feature; it thus appears that time frequency features merely provided information that supplemented the discriminatory time domain features. Time-domain features alone may be sufficient for online implementation of a NIRS corporeal machine interface.
No performance parameters had a significant effect on inter-subject classification accuracy. Average accuracies did not differ between LDA and SVM classifiers (p ≥ 0.05, corrected resampled t-test ). Interestingly, optimal classification accuracy was achieved for 8 of the 10 participants with an LDA-trained classifier, which is advantageous for its computational speed and ease of implementation.
Results indicate that the optimal analysis time-scale varies with the choice of signal features. A 20 s analysis interval was selected for all participants classified using a 2-feature vector containing at least one feature representing signal mean. Discriminatory information may be present in the NIRS hemodynamic signal for a prolonged period after its peak latency since the hemodynamic response needs more than 10 s to return to baseline [44, 45]. In contrast, a 15 s analysis interval was selected for 3 of 4 participants classified using signal skewness as a primary feature.
Maximum percent correct classification (PCC max ) rates across participants ranged from 75.0%-96.7%. Several trends become apparent after participant results were ranked by accuracy (Fig. 4). The four highest classification accuracies were produced using mean changes in [HbO] and [Hb] as discriminatory features. Additionally, six of the top seven performers achieved optimal accuracy in response to positively-valenced stimuli. This suggests that the time course of hemodynamic activity generated by emotional induction tasks may be influenced by valence.
A comparison across participants provided insight into why classification rates may vary. Figure 5 illustrates the trial-averaged hemodynamic response at site L4 for Participants 1 through 3. The GA selected a common feature (MeanHbOL 4) and identical parameters (classifier, recording sites, analysis interval length) for all three individuals. Participants 1 and 3 shared identical features and parameters with the exception of stimulus valence, and achieved the lowest and highest classification accuracies, respectively.
Participant 3 (PCC max = 96.67%) generated a consistent response using both valenced stimuli. A decrease in Δ[HbO] was observed for the duration of the emotional induction task (t = 30 - 40 s), which corroborates with previous study findings on sustained attention . We see a small increase in Δ[Hb] shortly after stimulus presentation consistent with the temporal profile of the NIRS hemodynamic response . These trends were also present in Participant 2's data (PCC max = 89.67%), although there is a longer latency before Δ[HbO] ceases to decrease. In the case of Participant 1 ((PCC max = 75.00%), hemodynamic activity was only visible in the signals generated by the negatively-valenced task. The trial-averaged Δ[HbO] and Δ[Hb] signals also contained larger fluctuations that obfuscated longer time-scale trends. Combining the findings described above, we propose that classification rates are limited by: 1) one's ability to consistently perform the emotional induction task; and 2) the hemodynamic response's rate of change.
From visual inspection of trial-averaged hemodynamic signals, it is apparent that response latency varies among individuals. Figure 6 summarizes optimal analysis interval lengths across participants. Each horizontal bar represents the analysis interval range for which significant activation was detected for a participant.
We begin by defining values of interest: 1) ΔT start , the smallest value of ΔT for which significant activation is detected; 2) ΔT max , the value of ΔT corresponding to PCC max over all analysis interval lengths tested; and 3) ΔT end , the largest value of ΔT for which significant activation is detected. ΔT start and ΔT end define the activation window.
The average time for onset of activation was 12.4 s across participants for whom significant activation was detected. Significant activation was not detected for Participants 6 and 9 and hence their data are not included in this average. It was earlier noted that the optimal feature pair selected for each participant included one of skewness or mean, which we define as a "primary discriminatory feature". Activation windows can be characterized by the primary discriminatory feature employed for classification:
Mean (n = 6) Classification rates improved with increased ΔT. ΔT max for all individuals was 20.0 s, the largest interval size considered in our analysis. These observations agree with results from the feature selection procedure. Participants with higher classification rates had shorter onset times prior to significant activation. Values of ΔT start varied but generally exhibited an inverse relationship with PCC max , ranging from 2.5 s (Participant 7, PCC max = 95.50%) to 19.7 s (Participant 10, PCC max = 78.00%).
Skewness (n = 4) Classification rates also improved with ΔT but peaked before ΔT reached 20.0 s. With the exception of one individual - for whom significant activation was not detected - ΔT max ranged from 14.7 s to 15.7 s. This suggests an analysis interval of ΔT = 15 s is nearly optimal for a feature set that includes skewness. For each of these participants, we identified a short range of analysis interval lengths surrounding ΔT max where significant activation was detected. Activation window sizes ranged from 0.0 s to 4.7 s. Differences between PCC max and baseline rates did not reach significance.
We have established that distinct patterns of hemodynamic activity generated by a visually-cued emotional induction task can be detected using NIRS and classified offline with accuracies significantly exceeding chance levels. Classification rates were comparable with values reported in previous NIRS-BCI studies. Six of the ten participants reached mean classification rates that exceeded the 70% threshold (p < 0.05) suggested by the scientific community as sufficient for communication and device control . It is conceivable that this number may have been higher if more trials were collected; however, data set size was inherently limited by the repetitive nature of the protocol and the mental demand of the task on the participant.
The onset time for a detectable hemodynamic response varied across individuals. Regardless of the types of features used for the classification task, a significant increase in mean classification accuracy was detected for the majority of participants 10 - 15 s after presentation of the visual stimulus. These latencies are in line with values previously reported in NIRS literature [13, 14].
Neurological and psychological factors
Participants generally found the emotional induction task straightforward to perform, and based on the experiences drawn from their involvement in the study, felt that such a paradigm can potentially be implemented in a user-friendly online corporeal machine interface.
Nevertheless, there are several factors that likely impacted data consistency within and across participants. Despite implementing preventative measures in the protocol to mitigate fatigue, four participants cited various aspects of the study as physically tiring. Incorporation of on-line feedback into the experiment may help maintain the participant's concentration and improve performance by providing a clear goal to the task. While one can argue that the benefits of neurofeedback are negligible over a single session, neurofeedback training is essential for operant conditioning of the EEG  and fMRI-BOLD responses [7, 8]. A participant may also begin the emotional induction task at a different time for each trial, further contributing to data inconsistencies. Possible causes include anticipatory effects  and loss of focus due to fatigue .
Some participants found the task easier to perform over time, whereas others found it increasingly difficult to concentrate as he or she repeatedly viewed the same pair of images. This may be attributed to the unique mental strategy each individual cultivated for performing the emotional induction task. Individual strategies ranged from using the image as a visual cue to focus on a more general emotion, to focusing on a salient component in the image. The PFC is involved in maintaining attentional demand , and variations in intensity and latency of hemodynamic activity across individuals might be caused by the different levels of attentional demand required for different strategies. Another possible explanation is that each participant's response to a stimulus was motivated by a different variation of endogenous salience, thereby eliciting different patterns of PFC activity. The image either functioned as a "primary inducer" conveying some intrinsic value, or acted as a secondary inducer that triggered the recall of a related memory or event . The latter, commonly referred to as self-referential processing , is accompanied by a more intense emotional response provided the stimulus contains personal relevance. Notably, the medial prefrontal cortex (mPFC) has been implicated in self-referential processing ; however, because participants were not provided with specific instructions on how to perform the emotional induction task, we cannot draw conclusive inferences about mPFC activity and self-referential processing.
The fact that results from feature selection did not suggest a correlation between stimulus valence and lateralization of brain activity may be due to optode placement. Optodes were located more medially than in several neuroimaging studies on emotional processing that have reported hemispheric specialization in the lateral PFC [53, 54]. In a metaanalysis of emotional activation studies, it was found that the mPFC is systematically activated by emotional stimuli regardless of valence . This suggests that the mPFC plays a general, rather than specific, role in emotional processing primarily mediated by arousal. It corroborates with our observation that a participant generally achieved higher classification rates using a stimulus he/she subjectively perceived as being more emotionally arousing. Five out of six participants who stated a preference for one image in the positive-negative valence picture pair achieved optimal classification accuracy using his or her preferred stimulus. To ascertain the effects of self-relevance in future studies, it would be beneficial to incorporate self-assessment of valence/arousal by each participant for each image.
Because the NIRS method measures venous, arterial and tissue oxygenation, it is more sensitive to localized concentration changes in skin microvasculature than underlying tissue volumes . Combined with the choice of a short source-to-detector separation distance, one may argue that our findings are based solely on oxygenation saturation of the extracranial layer, and are not indicative of functional activation in the cortex. We disagree that this is a limitation of the protocol. In a subset of study participants, we confirmed that hemoglobin concentration changes detected over a 2.1 cm spacing are highly correlated with adjacent measurements acquired over a 3.0 cm spacing, which is commonly used in fNIRS studies. It could also be argued that given the objectives of our study, the physiological origin of the detected hemodynamic response is secondary in importance to the ability to consistently generate a response.
Furthermore, scalp and skull thicknesses vary around the head of an individual . This contributes to variations in signal strength over different recording sites, and is a possible reason why we did not observe any trends in the site locations selected by the GA. The thickness of the extracranial layer dictates the minimum source to detector separation required to probe the cerebral cortex, and ideally, would be optimized for each individual. These dimensions are unknown unless an MRI scan is procured.
Although NIRS offers advantages over conventional EEG interfaces, it introduces instrumentation challenges unique to the technology. Hemodynamic signals are resistant to motion artifacts provided that optodes can be mounted firmly to the skin. However, it is a non-trivial task to secure optical fibres to the head, and design solutions must achieve a balance between stability of the optical fibres, versatility to accommodate a range of head sizes, and comfort. New methods are continuously being developed and a number of solutions have been implemented to date . Secondly, melanin is a known source of attenuation for optical throughput over the NIR range . While absorption and coupling issues caused by hair can be circumvented by measuring over hair-free regions such as the forehead, signal strength and penetration depth remain affected by skin colour.
The long latency of the hemodynamic response severely limits the information transfer rate of a NIRS corporeal machine interface. However, in addition to the hemodynamic response, frequency-domain NIRS measurements may yield a second "fast optical response" directly correlated with neuronal firing. The fast optical response is believed to be caused by changes in light scattering properties of neuronal membranes synonymous with activated cerebral tissue  and is elicited milliseconds after tissue stimulation . Not all researchers are convinced that the fast optical response can be detected non-invasively owing to the fact that the signal is dominated by other physiological artifacts [36, 59], and simulation results suggest that the magnitude of the fast optical response is below the noise level of presently available NIRS systems . If commercial systems that reliably capture the fast optical response become available, NIRS corporeal machine interfaces that respond as quickly as conventional EEG interfaces can be developed.
A priori knowledge of the latency of the hemodynamic response, which has been shown to vary across individuals, may be used to address the above shortcoming. For instance, if the optimal parameters and analysis interval length for signal classification were known for a user, the knowledge can be utilized to customize a corporeal machine interface, thus maximizing his or her abilities and improving response times. Since we could only collect a limited number of trials per participant within an experimental session, we did not have sufficient sample sizes to create completely disjoint data subsets for feature selection and classifier development. While our results may be therefore be optimistic, i.e., akin to "training accuracies", they are nonetheless on par with those reported for NIRS-BCIs using different mental tasks. Practically, a long data collection session (> 2 hours) only yields a modestly-sized data set per participant, given the non-trivial time periods for the cyclic generation and dissipation of hemodynamic responses. Thus collecting large data sets, while necessary, will remain a practical challenge for NIRS-based corporeal machine interfaces in future studies.
In our analyses, we attempted to suppress non-cortical contributions by low pass iltering. This is an inherent limitation as we do not have direct knowledge of the contaminant frequencies. Therefore, the signals we have classified are inevitably comprised of a combination of cortical and systemic blood flow. Other studies have suggested the simultaneous acquisition of deep and shallow signals using an optode arrangement consisting of multiple source-detector separations [61, 62]. In this way, systemic effects recorded in the shallow signal can be directly attenuated in the deep (cortical) signal.
The reliability of the proposed paradigm should be verified with simultaneous acquisition of fMRI and NIRS data, which would allow for accurate localization of externally recorded signals with respect to underlying anatomy. Qualitative amplitude correspondence of NIRS signals to the fMRI-BOLD response can provide insight into which types of emotional induction tasks are best suited for corporeal machine interfaces and their underlying psychophysiological bases.
As an extension to user customization in corporeal machine interfaces, it would also be desirable to investigate the effects of varying the time window for visual stimulus presentation. Like the analysis interval length for signal classification, this parameter could conceivably be optimized such that hemodynamic activity is generated reliably with less effort. Additional types of stimuli for the emotion induction paradigm should be investigated. Somatosensory and auditory stimuli are suitable alternatives for those with visual deficits, as well as multimedia stimuli such as film or music.
This study ascertained the feasibility of NIRS as a platform for a corporeal machine interface. We demonstrated that an emotional induction task in neurologically healthy individuals can elicit measurable hemodynamic responses in the prefrontal cortex. Classification accuracies up to 96.7% were obtained after feature subset selection while varying several performance parameters of interest. Results from the feature selection procedure indicate that mean and skewness parameters are the best discriminatory measures between resting and activation states induced by our task of interest. Relationships were also identified between a number of parameters, namely, feature subset and analysis interval length, and stimulus valence and classification accuracy. Lastly, classification accuracy was used to quantify the latency of the hemodynamic response within participants, with significant increases in accuracy from baseline occurring as early as 2.5 s from initial presentation of the stimulus.
Tai K, Blain S, Chau T: A review of emerging access technologies for individuals with severe motor impairments. Assist Technol 2008, 20: 204-219.
Neumann N, Kubler A: Training locked-in patients: a challenge for the user of brain-computer interfaces. IEEE Trans Neural Syst Rehabil Eng 2003,11(2):169-172. 10.1109/TNSRE.2003.814431
Sannita WG: Individual variability, end-point effects and possible biases in electrophysiological research. Clin Neurophysiol 2006,117(12):2569-2583. 10.1016/j.clinph.2006.04.026
Logothetis NK, Pauls J, Augath M, Trinath T, Oeltermann A: Neurophysiological investigation of the basis of the fMRI signal. Nature 2001,412(6843):150-157. 10.1038/35084005
Weiskopf N, Veit R, Erb M, Mathiak K, Grodd W, Goebel R, Birbaumer N: Physiological self-regulation of regional brain activity using real-time functional magnetic resonance imaging (fMRI): methodology and exemplary data. Neuroimage 2003,19(3):577-586. 10.1016/S1053-8119(03)00145-9
Yoo S, Jolesz F: Functional MRI for neurofeedback: feasibility study on a hand motor task. Neuroreport 2002,13(11):1377. 10.1097/00001756-200208070-00005
deCharms RC, Maeda F, Glover GH, Ludlow D, Pauly JM, Soneji D, Gabrieli JDE, Mackey SC: Control over brain activation and pain learned by using real-time functional MRI. Proc Natl Acad Sci USA 2005,102(51):18626-18631. 10.1073/pnas.0505210102
Caria A, Veit R, Sitaram R, Lotze M, Weiskopf N, Grodd W, Birbaumer N: Regulation of anterior insular cortex activity using real-time fMRI. Neuroimage 2007,35(3):1238-1246. 10.1016/j.neuroimage.2007.01.018
Birbaumer N: Breaking the silence: brain-computer interfaces (BCI) for communication and motor control. Psychophysiology 2006,43(6):517-532. 10.1111/j.1469-8986.2006.00456.x
Villringer A, Chance B: Non-invasive optical spectroscopy and imaging of human brain function. Trends Neurosci 1997,20(10):435-442. 10.1016/S0166-2236(97)01132-6
Delpy DT, Cope M, Zee P, Arridge S, Wray S, Wyatt J: Estimation of optical pathlength through tissue from direct time of flight measurement. Phys Med Biol 1988,33(12):1433-1442. 10.1088/0031-9155/33/12/008
Okada E, Delpy D: Near-infrared ligth propagation in an adult head model. II. Effect of superficial tissue thickness on the sensitivity of the near-infrared spectroscopy signal. Applied Optics 2003,42(16):2915-2922. 10.1364/AO.42.002915
Huppert TJ, Hoge RD, Diamond SG, Franceschini MA, Boas DA: A temporal comparison of BOLD, ASL, and NIRS hemodynamic responses to motor stimuli in adult humans. Neuroimage 2006, 29: 368-382. 10.1016/j.neuroimage.2005.08.065
Jasdzewski G, Strangman G, Wagner J, Kwong KK, Poldrack RA, Boas DA: Differences in the hemodynamic response to event-related motor and visual paradigms as measured by near-infrared spectroscopy. Neuroimage 2003, 20: 479-488. 10.1016/S1053-8119(03)00311-2
Malonek D, Grinvald A: Interactions between electrical activity and cortical microcirculation revealed by imaging spectroscopy: implications for functional brain mapping. Science 1996,272(5261):551-554. 10.1126/science.272.5261.551
Hashimoto K, Tategami S, Okamoto T, Seta H, Abo M, Ohashi M: Examination by Near-Infrared Spectroscopy for Evaluation of Piano Performance as a Frontal Lobe Activation Task. Eur Neurol 2006, 55: 16-21. 10.1159/000091138
Matsuda G, Hiraki K: Sustained decrease in oxygenated hemoglobin during video games in the dorsal prefrontal cortex: A NIRS study of children. Neuroimage 2006, 29: 706-711. 10.1016/j.neuroimage.2005.08.019
Nagamitsu S, Nagano M, Yamashita Y, Takashima S, Matsuishi T: Prefrontal cerebral blood volume patterns while playing video games - A near-infrared spectroscopy study. Brain Dev 2006, 28: 315-321. 10.1016/j.braindev.2005.11.008
Sanei S, Chambers J: EEG signal processing. John Wiley & Sons, Chichester; 2007.
Sitaram R, Zhang H, Guan C, Thulasidas M, Hoshi Y, Ishikawa A, Shimizu K, Birbaumer N: Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain-computer interface. Neuroimage 2007,34(4):1416-1427. 10.1016/j.neuroimage.2006.11.005
Coyle SM, Ward TE, Markham CM: Brain-computer interface using a simplified functional near-infrared spectroscopy system. J Neural Eng 2007,4(3):219-226. 10.1088/1741-2560/4/3/007
Naito M, Michioka Y, Ozawa K, Ito Y, Kiguchi M, Kanazawa T: A Communication Means for Totally Locked-in ALS Patients Based on Changes in Cerebral Blood Volume Measured with Near-Infrared Light. IEICE T Inf Syst 2007,90(7):1028-1037. 10.1093/ietisy/e90-d.7.1028
Miller E, Cohen J: An integrative theory of prefrontal cortex function. Annu Rev Neurosci 2001, 24: 167-202. 10.1146/annurev.neuro.24.1.167
Herrmann M, Ehlis A, Fallgatter A: Prefrontal activation through task requirements of emotional induction measured with NIRS. Biol Psychol 2003,64(3):255-63. 10.1016/S0301-0511(03)00095-4
León-Carrión J, Damas J, Izzetoglu K, Pourrezai K, Martín-Rodríguez J, Martin J, Domínguez-Morales M: Differential time course and intensity of PFC activation for men and women in response to emotional stimuli: A functional near-infrared spectroscopy (fNIRS) study. Neurosci Lett 2006,403(1-2):90-95. 10.1016/j.neulet.2006.04.050
Yang H, Zhou Z, Liu Y, Ruan Z, Gong H, Luo Q, Lu Z: Gender difference in hemodynamic responses of prefrontal area to emotional stress by near-infrared spectroscopy. Behav Brain Res 2007, 178: 172-176. 10.1016/j.bbr.2006.11.039
Jasper JJ: The 10/20 international electrode system. EEG and Clin Neurophysiol 1958, 10: 371-375.
Lang P, Bradley M, Cuthbert B: International affective picture system (IAPS): Digitized photographs, instruction manual and affective ratings. Tech. rep., Technical Report A-6 2005.
Cope M: The application of near infrared spectroscopy to non invasive monitoring of cerebral oxygenation in the newborn infant. PhD thesis. London: University College; 1991.
Duncan A, Meek J, Clemence M, Elwell C, Tyszczuk L, Cope M, Delpy D: Optical pathlength measurements on adult head, calf and forearm and the head of the newborn infant using phase resolved optical spectroscopy. Phys Med Biol 1995,40(2):295-304. 10.1088/0031-9155/40/2/007
Bonmassar G, Purdon P, Jääskeläinen I, Chiappa K, Solo V, Brown E, Belliveau J: Motion and Ballistocardiogram Artifact Removal for Interleaved Recording of EEG and EPs during MRI. Neuroimage 2002,16(4):1127-1141. 10.1006/nimg.2002.1125
He P, Wilson G, Russell C: Removal of ocular artifacts from electro-encephalogram by adaptive filtering. Med Biol Eng Comput 2004,42(3):407-412. 10.1007/BF02344717
Morren G, Wolf M, Lemmerling P, Wolf U, Choi J, Gratton E, De Lathauwer L, Van Huffel S: Detection of fast neuronal signals in the motor cortex from functional near infrared spectroscopy measurements using independent component analysis. Med Biol Eng Comput 2004, 42: 92-99. 10.1007/BF02351016
Zhang Q, Brown E, Strangman G: Adaptive filtering for global interference cancellation and real-time recovery of evoked brain activity: a Monte Carlo simulation study. J Biomed Opt 2007, 12: 044014. 10.1117/1.2754714
Ramsay J, Li X: Curve Registration. J Royal Stat Soc B 1998,60(2):351-363. 10.1111/1467-9868.00129
Mayhew J, Askew S, Zheng Y, Porrill J, Westby G, Redgrave P, Rector D, Harper R: Cerebral vasomotion: a 0.1-Hz oscillation in reflected light imaging of neural activity. Neuroimage 1996,4(3 Pt 1):183-193. 10.1006/nimg.1996.0069
Obrig H, Neufang M, Wenzel R, Kohl M, Steinbrink J, Einhäupl K, Villringer A: Spontaneous Low Frequency Oscillations of Cerebral Hemodynamics and Metabolism in Human Adults. Neuroimage 2000,12(6):623-639. 10.1006/nimg.2000.0657
Hellige J: Hemispheric Asymmetry. Cambridge, MA: Harvard University Press; 1993.
Grefenstette J, Baker J: How genetic algorithms work: a critical look at implicit parallelism. Proceedings of the Third International Conference on Genetic Algorithms 1989, 20-27.
Fitzpatrick J, Grefenstette J: Genetic algorithms in noisy environments. Mach Learn 1988,3(2):101-120.
De Jong K, Spears W: An Analysis of the Interacting Roles of Population Size and Crossover in Genetic Algorithms. Proceedings of the 1st Workshop on Parallel Problem Solving from Nature 1990, 38-47.
Kwong K, Belliveau J, Chesler D, Goldberg I, Weisskoff R, Poncelet B, Kennedy D, Hoppel B, Cohen M, Turner R, et al.: Dynamic magnetic resonance imaging of human brain activity during primary sensory stimulation. Proc Natl Acad Sci USA 1992,89(12):5675. 10.1073/pnas.89.12.5675
Nadeau C, Bengio Y: Inference for the Generalization Error. Mach Learn 2003,52(3):239-281. 10.1023/A:1024068626366
Boynton G, Demb J, Glover G, Heeger D: Neuronal basis of contrast discrimination. Vision Res 1999,39(2):257-269. 10.1016/S0042-6989(98)00113-8
Wobst P, Wenzel R, Kohl M, Obrig H, Villringer A: Linear Aspects of Changes in Deoxygenated Hemoglobin Concentration and Cytochrome Oxidase Oxidation during Brain Activation. Neuroimage 2001,13(3):520-530. 10.1006/nimg.2000.0706
Kübler A, Mushahwar V, Hochberg L, Donoghue J: BCI meeting 2005-workshop on clinical issues and applications. IEEE Trans Neural Syst Rehabil Eng 2006,14(2):131-134. 10.1109/TNSRE.2006.875585
Birbaumer N, Ghanayim N, Hinterberger T, Iversen I, Kotchoubey B, Kübler A, Perelmouter J, Taub E, Flor H: A spelling device for the paralysed. Nature 1999,398(6725):297-298. 10.1038/18581
Liang H, Wang H: Top-down anticipatory control in prefrontal cortex. Theor Biosci 2003, 122: 70-86.
Simpson J Jr, Drevets W, Snyder A, Gusnard D, Raichle M: Emotion-induced changes in human medial prefrontal cortex: II. During anticipatory anxiety. Proc Natl Acad Sci USA 2001,98(2):688. 10.1073/pnas.98.2.688
MacDonald A, Cohen J, Stenger V, Carter C: Dissociating the Role of the Dorsolateral Prefrontal and Anterior Cingulate Cortex in Cognitive Control. Science 2000,288(5472):1835. 10.1126/science.288.5472.1835
Bechara A, H D, Damasio A: Role of the Amygdala in Decision-Making. Ann N Y Acad Sci 2003, 985: 356.
Northoff G, Heinzel A, de Greck M, Bermpohl F, Dobrowolny H, Panksepp J: Self-referential processing in our brain -- A meta-analysis of imaging studies on the self. Neuroimage 2006, 31: 440-457. 10.1016/j.neuroimage.2005.12.002
Canli T, Desmond J, Zhao Z, Glover G, Gabrieli J: Hemispheric asymmetry for emotional stimuli detected with fMRI. Neuroreport 1998,9(14):3233. 10.1097/00001756-199810050-00019
Gray J, Braver T, Raichle M: Integration of emotion and cognition in the lateral prefrontal cortex. Proc Natl Acad Sci USA 2002,99(6):4115. 10.1073/pnas.062381899
Phan K, Wager T, Taylor S, Liberzon I: Functional Neuroanatomy of Emotion: A Meta-Analysis of Emotion Activation Studies in PET and fMRI. Neuroimage 2002,16(2):331-348. 10.1006/nimg.2002.1087
Liu H, Chance B, Hielscher A, Jacques S, Tittel F: Influence of blood vessels on the measurement of hemoglobin oxygenation as determined by time-resolved reflectance spectroscopy. Med Phys 1995, 22: 1209. 10.1118/1.597520
Lynnerup N, Astrup J, Sejrsen B: Thickness of the human cranial diploe in relation to age, sex and general body build. Head Face Med 2005, 1: 13. 10.1186/1746-160X-1-13
Strangman G, Culver J, Thompson J, Boas D: Quantitative Comparison of Simultaneous BOLD fMRI and NIRS Recordings during Functional Brain Activation. Neuroimage 2002,17(2):719-731. 10.1016/S1053-8119(02)91227-9
Gratton G, Corballis P: Removing the heart from the brain: compensation for the pulse artifact in the photon migration signal. Psychophysiology 1995,32(3):292-9. 10.1111/j.1469-8986.1995.tb02958.x
Steinbrink J, Kempf F, Villringer A, Obrig H: The fast optical signal - Robust or elusive when non-invasively measured in the human adult? Neuroimage 2005,26(4):996-1008. 10.1016/j.neuroimage.2005.03.006
Toronov V, Webb A, Choi J, L Safonova MW, Wolf U, Gratton E: Study of local cerebral hemodynamics by frequency-domain near-infrared spectroscopy and correlation with simultaneously acquired functional magnetic resonance imaging. Optics Express 2001,9(8):417-427. 10.1364/OE.9.000417
Luu S, Chau T: Decoding subjective preference from single-trial near-infrared spectroscopy signals. Journal of Neural Engineering 2009, 6: 1-8. 10.1088/1741-2560/6/1/016003
This work was supported in part by Bloorview Children's Hospital Foundation, the Natural Sciences and Engineering Research Council of Canada, and the Canada Research Chairs Program.
The authors declare that they have no competing interests.
KT conceptualized the study, carried out the data collection procedure, conducted data analyses, and drafted the manuscript. TC contributed to study conception, supervised the study and revised the manuscript. All authors read and approved the final manuscript.
Authors’ original submitted files for images
About this article
Cite this article
Tai, K., Chau, T. Single-trial classification of NIRS signals during emotional induction tasks: towards a corporeal machine interface. J NeuroEngineering Rehabil 6, 39 (2009). https://doi.org/10.1186/1743-0003-6-39
- Feature Selection
- Classification Accuracy
- Hemodynamic Response
- Feature Subset
- International Affective Picture System