Human-machine interfaces based on EMG and EEG applied to robotic systems
© Ferreira et al. 2008
Received: 01 February 2007
Accepted: 26 March 2008
Published: 26 March 2008
Two different Human-Machine Interfaces (HMIs) were developed, both based on electro-biological signals. One is based on the EMG signal and the other is based on the EEG signal. Two major features of such interfaces are their relatively simple data acquisition and processing systems, which need just a few hardware and software resources, so that they are, computationally and financially speaking, low cost solutions. Both interfaces were applied to robotic systems, and their performances are analyzed here. The EMG-based HMI was tested in a mobile robot, while the EEG-based HMI was tested in a mobile robot and a robotic manipulator as well.
Experiments using the EMG-based HMI were carried out by eight individuals, who were asked to accomplish ten eye blinks with each eye, in order to test the eye blink detection algorithm. An average rightness rate of about 95% reached by individuals with the ability to blink both eyes allowed to conclude that the system could be used to command devices. Experiments with EEG consisted of inviting 25 people (some of them had suffered cases of meningitis and epilepsy) to test the system. All of them managed to deal with the HMI in only one training session. Most of them learnt how to use such HMI in less than 15 minutes. The minimum and maximum training times observed were 3 and 50 minutes, respectively.
Such works are the initial parts of a system to help people with neuromotor diseases, including those with severe dysfunctions. The next steps are to convert a commercial wheelchair in an autonomous mobile vehicle; to implement the HMI onboard the autonomous wheelchair thus obtained to assist people with motor diseases, and to explore the potentiality of EEG signals, making the EEG-based HMI more robust and faster, aiming at using it to help individuals with severe motor dysfunctions.
Electro-biological signals have become the focus of several research institutes, probably stimulated by the recent findings in the areas of cardiology, muscle physiology and neuroscience, by the availability of more efficient and cheaper computational resources, and by the increasing knowledge and comprehension about motor dysfunctions [1, 2].
Electrical signals coming from different parts of the human body can be used as command signals for controlling mechanical systems. However, it is necessary that the individual in charge of controlling such devices be able to intentionally generate such signals. It is also necessary that the interface adopted (the Human-Machine Interface – HMI) can "understand" and process such signals, setting the command that better fits the wish of the individual. Then, an HMI can be used to improve the capacity of movement of individuals with motor dysfunctions, using, for example, a robotic wheelchair to carry them.
Many electro-biological signals can be used in connection with HMIs. Some of the more commonly adopted signals are the Electro-Myographic (EMG) signal, the Electro-Oculographic (EOG) signal and the Electro-Encephalographic (EEG) signal. This work presents results related to the use of EMG and EEG signals. The use of EOG signal is still incipient in the studies we have developed so far.
Although EEG signals were initially used just in Neurology and Psychiatry, mainly to diagnose brain diseases as epilepsy, sleep disorders and some types of cerebral tumors, many research groups are now using them as a communication channel between a person's brain and electronic machines, in order to develop systems to improve his life condition. The main point of this idea is the Human-Machine Interface (HMI), also called a Brain-Computer Interface (BCI), a system capable to acquire the EEG signal, to extract features there embedded, to "understand" the intention manifested by the user and to control electronic devices such as a PC, a robot or a wheelchair.
In addition, if the objective is to develop a portable and embedded BCI, low cost, small size, small weight and portability are very important advantages of systems based on the EEG signal when compared to other ways to register brain activity . Other advantages of using EEG signals are: they have good temporal resolution and allows extracting features enough to control electronic devices (since appropriate signal processing methods are used).
This work presents a sequence of development for an HMI that takes into account all the previous considerations, and in which the degree of difficulty in both signal acquisition and processing is gradually increased. In such a sequence, in the first stage of the implementation the HMI developed is based on the signal caused by eye blinks (an EMG signal). Such a system was used to control a mobile robot, which was able to navigate in a semi-structured environment. Next, a module capable to acquire and process EEG signals was also implemented, which currently explores the ERS/ERD complex of the EEG signal acquired by two electrodes placed in the occipital region of the head of an individual with motor dysfunction (such a signal is related to visual activity). Such modules have been used to control a mobile robot and a robotic manipulator, respectively. Experimental results using such modules are presented in the paper, as well as some discussion about the future of the research our group is developing is presented.
Brief review on commanding a mobile robot using EMG signals
EMG (ElectroMyoGram) signals are generated by the contraction of the human-body muscles. They are currently being used to command robotic devices like manipulators (robotic arms and hands) and mobile robots (robotic wheelchairs). The goal is to develop systems capable to help people with different motor disabilities.
The systems shown in  and  allow controlling robot manipulators through some muscular signals. In , specifically, the left and right Flexor Carpi Radialis muscle (a muscle near elbow) are used, with the third sensor placed on the Brachioradialis muscle (a muscle on the forehead), to generate a series of activations to open/close a gripper, and to move it to pre-defined positions, thus allowing people with severe motor disability to execute activities of daily life.
In , EMG signals are acquired from biceps brachii, the muscle that is the main responsible for the flexion of the elbow of an individual, to teleoperate a robot arm. Although the dynamic model of the robot arm is taken into account, the experimental results there presented have just shown the robustness of the system when regarding smooth elbow movements. A similar work is presented in . However, in this last one, the experiments conducted show the system accuracy and robustness for both slow and fast catching motions. In addition, experiments with targets being placed in different directions and distances are also conducted. An EMG-based command is also used in a dexterous robot hand in . The system reproduces the finger motions when the user moves his/her fingers, and can be teleoperated as in . The experimental success rate for six different types of finger motions reached more than 77%.
Some systems use EMG-based signals for commanding robotic wheelchairs. A robot wheelchair is useful for people with motor disabilities in both lower and upper extremities, due to paralysis or amputation. In  three solutions are presented to set the wheelchair in motion: an HMI based on EMG signals, face directional gesture, and voice. The EMG signals are acquired from the elevator scapulae muscle, and can be generated by voluntary elevation movements of both left and right shoulder. The experimental results shown in  allowed concluding that the system can be used by people with motor disabilities, although just indoor experiments have been performed. Another conclusion presented in  is that it is necessary to build an environment map to perform long-time outdoor navigation.
In  and  systems very similar to those proposed in  are presented, also using commands based only on EMG signals. The great advantages of the system proposed in , however, are its low cost and its small size, which are due to the use of a non-commercial EMG amplifier. In addition, in  it is used a combination of the movements of the muscles of the shoulder and the neck to command the wheelchair. In the several works which address the command of robots through systems based on EMG signals, many types of muscles are used as command signal generators. In general, the upper extremity muscles, e. g., the muscles for wrist and elbow flexion, are the most commonly used. When the individual does not have such muscles, however, it is common to use the shoulders and/or neck motion muscles. Sometimes, when the individual can not move any part of his/her body, but he/she can blink his/her eyes, the EMG signals can still be useful for commanding devices. In such cases, as addressed here, the EMG signal is generated by blinking the eyes.
Brief review on commanding a robot using EEG signals
The electrical potential caused by the neuronal activity, recorded from the scalp (a non-invasive way) or directly from the brain cortex (ECoG), can be used to control robots and other electronic devices. In the sequence, some meaningful works dealing with such subject are commented, in order to provide a brief overview about brain-actuated devices.
Example of ECoG recording can be found in . The electrical activity acquired on the brain cortex surface is not attenuated as the signal captured on the scalp (after crossing the cranium), thus presenting a better quality. The objective is to map the data corresponding to the multi-channel neural spikes of a monkey to the 3D positions of its arm positions. The predicted position of the hand of the monkey is used to control a robot arm.
A brain-actuated control of a mobile robot is reported in . Two individuals were able to control a small Khepera mobile robot navigating through a house-like environment after a few days of training. EEG potentials were recorded through eight electrodes placed on standard fronto-centro-pariental positions, in a non-invasive way. Spatial filtering, Welch periodogram algorithm and a statistical classifier were used to recognize mental tasks, such as "relax", imagination of "left" and "right" hand (or arm) movements, "cube rotation", "subtraction", and "word association", which were used by a finite state automata for controlling the robot. An asynchronous BCI was adopted, which avoids the waiting for external cues, unlike a synchronous one. A meaningful rate of correct recognition (above 60%), associated to an error rate below 5%, was obtained with such a BCI, which resulted in a brain-actuated control of the robot demanding no more than 35% of the time spent for manually controlling the robot, for both individuals. A similar work is reported in , in which a virtual keyboard and a mobile robot are controlled by using an asynchronous BCI, which was tested by 15 individuals.
Most recent studies have shown that dissatisfaction of individuals can be used to correct machine errors. When an individual sends a command to a device and gets a non-expected response, the awareness of erroneous responses, even when the error is not made by the individual himself, can be recognized in the brain signal captured. This is done through error-related potentials (ErrP) and is used to improve the performance of the BCI .
Several works reporting the use of the signal caused by brain activity to command devices have been published. However, the Human-Machine Interfaces or Brain-Computer Interfaces used are still too much expensive. In some cases, they are even more expensive than the robot, the wheelchair or other device being commanded. Regarding this topic, the HMIs proposed in this work are attempts to get a good compromise between effectiveness for the application and cost.
Experiments based on muscular and cerebral activities are here accomplished in order to verify that a human operator is capable to command robots through Human-Machine Interfaces. Two HMIs, based on different electro-biological signals were developed, namely an EMG-based HMI and an EEG-based HMI. The first one allows a person to command devices through the signal generated by blinking his own eyes . The other one allows decoding brain commands as well as controlling devices like robots . In this section a brief introduction to such systems is presented.
An EMG-based human-machine interface
After being correctly dressed, the cap should be connected to the signal filtering and amplification subsystem. The amplification board embeds a power source that is designed to reduce any spurious interference at the same frequency of the electric appliances or interference coming from other external electronic equipments, such as switching mode power supplies, on the acquisition system. Then, the signal filtering and amplification subsystem is connected to the A/D conversion subsystem. Four analog channels are available in such A/D conversion subsystem, which allow expanding the signal acquisition capacity through cascade connections, thus increasing the number of channels being processed. After establishing such connections, the digital data delivered by the A/D converter is sent to a desktop computer, through a DB9 serial cable.
Then, the system is now operating: the user's electro-biological signal is acquired by electrodes that send it to the signal filtering and amplification subsystem. Afterwards, this signal is sent to another board to be converted to digital data. Finally, such signal is transmitted to a desktop computer, where it is processed to generate (or not) a specific command for controlling a mobile robot. The user of the HMI closes the control loop, providing the necessary biological feedback.
The interface for the user-machine communication is programmed in the desktop computer, as well as the signal processing software that sends the control commands to the mobile robot. These commands are transmitted to the robot through an Ethernet Radio.
The experiments here reported were carried out using a Pioneer 2-DX nonholonomic wheeled mobile robot. This robot has a microcontroller for low level instructions, and an embedded PC (Intel Pentium MMX 266 MHz, 128 MB RAM) for high level tasks like sensing and/or navigation.
EEG-based human-machine interface
Additional attention should be given to artifacts. Eye blink, cardiac rhythms, noise coming from the 50–60 Hz power line and body movement are examples of artifacts. They can mask the studied signal and should be avoided and removed. The frequency band explored here is from 8 to 13 Hz, and with a bandpass filter it is possible to remove artifacts due to eye blinks, which usually occurs between 0.1 and 5 Hz, as well as the noise of 50–60 Hz coming from the power line [18, 19].
The BCI adopted here to extract information on the occurrence of the ERS/ERD events is relatively easy to use. As in the EMG-based case, shaving the operator's head or other special preparation is not necessary. However, a gel is used to improve the contact between the electrodes and the skin. The electrodes are placed in the positions O 1 and O 2, like illustrated in Figure 4, with the reference connected to an ear lobe (according to the 10–20 international system of electrodes positioning).
Such a BCI was tested by a group of 25 individuals (from 20 to 50 years old), some of which had suffered cases of meningitis or epilepsy. Three stages of experimentation were accomplished: in the first one, the operator uses an event detector that recognizes the states of high and low energy of the acquired signal; in the second one, the operator is invited to command the robot in a simulation environment, and, in the last one the operator applies what he learnt in the two previous stages to command a real robot .
An operator is considered capable of having full control of the BCI if he succeeds in the first and second stages, what means if he showed to be able to command the robot in a simulation environment using the BCI.
Two experiments were carried out to validate the BCI and the control scheme as a whole. In the first one, the operator used the BCI to guide a mobile robot in an indoor structured environment, thus emulating a wheelchair taking the operator to the rooms of a house or office, for example. In the second one, the operator uses the BCI to command a manipulator, emulating a prothesis or an orthosis, including the teleoperation via a TCP/IP channel.
First experiment: commanding the mobile robot
Second experiment: commanding the manipulator
It is important to remember that in both cases a calibration process is necessary before starting the experiments. This procedure consists of acquiring about 10 seconds of EEG data to analyze the ERD level. Based on this information, the threshold used to detect an ERS is set to 5 up to 10 times the level corresponding to an ERD. This is very important because these levels change constantly in time and from an individual to another.
Results and discussion
Both HMIs have been used to command robotic devices by individuals previously trained to operate them. The EMG-based HMI was used to command a mobile robot, while the EEG-based HMI was used to command a mobile robot and a robotic manipulator as well. In this section, the results of each test accomplished are reported and discussed.
Success rates blinking the right and left eyes
The main result obtained is a rate of positive identification of the eye blinks about 95.71% of the cases of volunteers with the ability to blink both eyes, which allowed concluding the viability of using the system to command devices.
One out of the eight volunteers that presented a good performance in the experiment with the eye blinks-based system was asked to determine a destination point on the electronic board. After the volunteer selected a destination point through eye blinks, the control software started to guide the robot to such point, following the path determined by a path planning algorithm , which is based on the Dijkstra's Algorithm, that determines a secure path (more distant of walls and obstacles).
Although for the applications here addressed the EEG-based BCI so far discussed have run and performed very well, more natural mental states, such as thinking about moving a right hand in order to move the robot to right, for example, should be more interesting. More mental states provides more flexibility when connecting them, or a combination of them, to actions to be performed by mechanical devices. These topics are currently being addressed by our research group.
Nevertheless, an analysis of the experiments so far accomplished, namely guiding a mobile robot and controlling the positioning of the end effector of a manipulator, shows that the BCI so far adopted has proven to be effective to command robotic systems, including remotely.
Two different HMIs were here developed to allow an operator to control a robot without using his hands. He used only the EMG signal generated by blinking his eyes, or the EEG signal generated by intercalating states of concentration and relaxation of the visual cortex of his brain (visual stimuli). In both cases, the HMIs have proven to be of simple implementation and of low cost, besides exhibiting good performance. The EMG signal was chosen as electro-biological signal due to the fact that it is a well behaved signal easily acquired and processed, in comparison to other electro-biological signals, as the EEG signal, for instance. The results demonstrate that such a HMI is easy to handle by users who can blink their eyes according their wishes. This HMI was tested in controlling a mobile robot: an experiment in which a user should select a final destination to which the robot is supposed to go to, through a suitable sequence of eye blinks, which should be reached by the robot. In all tests, the mobile robot effectively reached the destination selected by the user.
The EEG-based HMI can be seen as an evolution of the EMG-based HMI due to the increase in the degree of difficulty of both the acquisition and processing subsystems. It has been used the so-called ERS/ERD complex, which can be identified in a relatively easy way, which provided, in two cases presented (a mobile robot guidance example and a robotic manipulator control example), a simple and low cost solution. In both cases presented, the desired commands were effectively executed by the robotic devices.
The work so far reported are the beginning of the development of a system intended to assist people suffering of neuromotor diseases, including people with severe dysfunctions. The next steps are to convert a commercial wheelchair in an autonomous mobile vehicle; to implement the HMI on board such autonomous wheelchair to assist people with motor diseases; to explore more characteristics of the EEG signal, in order to make the Brain-Computer Interface (BCI) more robust and faster, thus allowing its secure use by people with severe motor dysfunctions.
The authors thank CAPES (Brazil) and SPU (Argentina) for their financial support to the partnership between Federal University of Espirito Santo, Vitoria, Brazil, and National University of San Juan, San Juan, Argentina, through the binational program CAPG-BA. As part of this financial support, Andre Ferreira got a scholarship to stay six months in San Juan, Argentina, where part of this work was developed. The authors also thank FAPES/Brazil (Process 30897440/2005) for its financial support to this research.
- Cheeín FA, Postigo J: A Fast Finite State Machine Design for a Brain Computer Interface. XI Reunión de Trabajo en Processamiento de la Información y Control, Argentina 2005.
- Millán J, Renkens F, Mouriño J, Gerstner W: Non-Invasive Brain-Actuated Control of a Mobile Robot. Proceedings of the 18th International Joint Conference on Artificial Intelligence, Acapulco, Mexico 2003.
- Kubler A, Kotchoubey B, Kaiser J, Wolpaw JR, Birbaumer N: Brain-computer communication: unlocking the locked in. Psychol Bull 2001,127(3):358–375.View ArticlePubMed
- Mouriño J: EEG-based Analysis for the Design of Adaptive Brain Interfaces. PhD thesis Universitat Politècnica de Catalunya, Barcelona, Spain 2003.
- Pfurtscheller G, da Silva FHL: Event-related EEG/MEG synchronization and desynchronization: basic principles. Clinical Neurophysiology 1999,110(11):1842–1857.View ArticlePubMed
- Rani P, Sarkar M: EMG-based high level human-robot interaction system for people with disability. Proceedings of 2005 IEEE International Workshop on Robot and Human Interactive Communication (ROMAN 2005), Nashville, Tennessee 2005, 280–285.
- Wang J, Wang R, Li F, Jiang M, Jin D: EMG Signal Classification for Myoelectric Teleoperating a Dexterous Robot Hand. Proceedings of the 27th Annual International Conference of the Engineering in Medicine and Biology Society (IEEE-EMBS 2005), Shanghai, China 2005, 6:5931–5933.
- Artemiadis P, Kyriakopoulos K: Teleoperation of a robot manipulator using EMG signals and a position tracker. Proceedings of 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2005), Alberta, Canada 2005, 1003–1008.
- Artemiadis P, Kyriakopoulos K: EMG-based teleoperation of a robot arm in planar catching movements using ARMAX model and trajectory monitoring techniques. Proceedings 2006 IEEE International Conference on Robotics and Automation (ICRA 2006), Orlando, Florida 2006, 3244–3249.
- Moon I, Lee M, Ryu J, Mun M: Intelligent Robotic Wheelchair with EMG-, Gesture-, and Voice-based Interfaces. Proceedings of 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003), Las Vegas, Nevada 2003, 4:3453–3458.View Article
- Han JS, Zenn Bien Z, Kim DJ, Lee HE, Kim JS: Human-machine interface for wheelchair control with EMG and its evaluation. Proceedings of the 25th Annual International Conference of the Engineering in Medicine and Biology Society (IEEE-EMBS 2003), Cancun, Mexico 2003, 2:1602–1605.
- Moon I, Lee M, Chu J, Mun M: Wearable EMG-based HCI for Electric-Powered Wheelchair Users with Motor Disabilities. Proceedings of 2005 IEEE International Conference on Robotics and Automation (ICRA 2005), Barcelona, Spain 2005, 2649–2654.
- Darmanjian S, Kim SP, Nechyba M, Morrison S, Principe J, Wessberg J, Nicolelis M: Bimodal brain-machine interface for motor control of robotic prosthetic. Proceedings of 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003), Las Vegas, Nevada 2003, 4:3612–3617.View Article
- Millán J, Mouriño J: Asynchronous BCI and Local Neural Classifiers: An Overview of the Adaptive Brain Interface Project. IEEE Transactions on Neural Systems and Rehabilitation Engineering 2003,11(2):159–161.View Article
- Ferrez PW, Millán J: You Are Wrong! – Automatic Detection of Interaction Errors from Brain Waves. Proceedings of the 19th International Joint Conference on Artificial Intelligence, 5, Martigny, Switzerland 2005.
- Frizera-Neto A, Celeste WC, Martins VR, Bastos-Filho TF, Sarcinelli-Filho M: Human-Machine Interface Based on Electro-Biological Signals for Mobile Vehicles. Proceedings of the International Symposium on Industrial Electronics (ISIE 2006), Montreal, Canada 2006, 2954–2959.
- Ferreira A, Bastos-Filho TF, Sarcinelli-Filho M, Cheein FA, Postigo JF, Carelli R: Teleoperation of an Industrial Manipulator Through a TCP/IP Channel Using EEG Signals. Proceedings of the International Symposium on Industrial Electronics (ISIE 2006), Montreal, Canada 2006, 3066–3071.
- Haas SM, Frei MG, Osorio I, Pasik-Duncan B, Radel J: EEG Ocular Artifact Removal Through ARMAX Model System Identification Using Extended Least Squares. Communications in Information and Systems 2003, 3:19–40.
- Rohalova M, Sykacek P, Koska M, Dorffner G: Detection of the EEG Artifacts by the Means of the (Extended) Kalman Filter. Measurement Science Review 2001, 1:59–62.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.