Towards the development of a wearable feedback system for monitoring the activities of the upper-extremities
© Xiao and Menon; licensee BioMed Central Ltd. 2014
Received: 28 November 2012
Accepted: 3 January 2014
Published: 8 January 2014
Body motion data registered by wearable sensors can provide objective feedback to patients on the effectiveness of the rehabilitation interventions they undergo. Such a feedback may motivate patients to keep increasing the amount of exercise they perform, thus facilitating their recovery during physical rehabilitation therapy. In this work, we propose a novel wearable and affordable system which can predict different postures of the upper-extremities by classifying force myographic (FMG) signals of the forearm in real-time.
An easy to use force sensor resistor (FSR) strap to extract the upper-extremities FMG signals was prototyped. The FSR strap was designed to be placed on the proximal portion of the forearm and capture the activities of the main muscle groups with eight force input channels. The non-kernel based extreme learning machine (ELM) classifier with sigmoid based function was implemented for real-time classification due to its fast learning characteristics. A test protocol was designed to classify in real-time six upper-extremities postures that are needed to successfully complete a drinking task, which is a functional exercise often used in constraint-induced movement therapy. Six healthy volunteers participated in the test. Each participant repeated the drinking task three times. FMG data and classification results were recorded for analysis.
The obtained results confirmed that the FMG data captured from the FSR strap produced distinct patterns for the selected upper-extremities postures of the drinking task. With the use of the non-kernel based ELM, the postures associated to the drinking task were predicted in real-time with an average overall accuracy of 92.33% and standard deviation of 3.19%.
This study showed that the proposed wearable FSR strap was able to detect eight FMG signals from the forearm. In addition, the implemented ELM algorithm was able to correctly classify in real-time six postures associated to the drinking task. The obtained results therefore point out that the proposed system has potential for providing instant feedback during functional rehabilitation exercises.
KeywordsForce myography Classification Extreme learning machine Constraint-induced movement therapy Functional exercise
Body motion and physiological data registered by wearable sensors have been used for diagnostics, as well as monitoring the rehabilitation progress of people recovering from an injury or living with a chronic disease such as stroke . These data can provide objective feedback on patient’s health status and the progress of rehabilitation, which allow therapists to optimize the rehab routine . Feedback to the patient should be provided in real-time, as there is evidence that instant feedback can further motivate the user to reach the targeted goal or even to keep increasing the amount of exercise . An example of monitoring device that provides instant feedback to the user is the pedometer, which counts the number of steps and effectively motivates people to increase their walking activities towards better health . However, compared to the lower-extremities activity monitoring, there are few affordable and easy to use devices that can provide instant feedback of the activities of the upper-extremities to motivate the patients. Many studies have shown the increase of upper-extremities activity can lead to better outcomes after neurological conditions including stroke , head injury , incomplete spinal cord injury  and cerebral palsy . Thus, there are great needs for having such a device for providing instant feedback of targeted rehab exercise that involves upper-extremities movement.
Current commercial wrist accelerometers, such as the Actical  and the Actigraph , can provide objective measures of arm use based on multidirectional acceleration data of the upper-extremities. However, these systems provide no real-time feedback to the user, nor are able to capture any information about hand use, which is one of the most important upper-extremities functions in our daily life. Besides the use of accelerometers, there are other methods available for capturing both the hand and arm movement. One example is the use of a data-glove, such as the Cyberglove , which incorporates both inertial measurement unit (IMU) sensors and flexible bend sensors for motion capturing. However, data-gloves are generally designed for virtual reality applications that require the use of a host CPU. Moreover, data-gloves limit the tactile sensation of the user’s fingers, thus limiting the effectiveness of rehabilitation protocols involving the somatosensory system.
In addition to commercially available devices, current active research focuses on processing bio-signals through the use of surface electromyography (sEMG) to predict the upper-extremities movements that involve elbow, wrist or/and hand [12–14]. Even though this method frees the hand and allows full tactile sensation, it requires expensive and sizable equipment, as well as high-level signal processing for feature extraction. This approach is therefore not very suitable for inexpensively detecting movements in outdoor activities or in the home environment.
Other than using accelerometer, data acquisition glove or sEMG for monitoring the upper extremity movement, there is a relatively unexplored method named force myography (FMG). FMG is referred to a technique which use force sensor to capture the expansion/contraction of the large surface muscle . The use of FMG to distinguish limb movements was preliminarily explored by O.Amft et al.  who used two force resistive sensors (FRS) on the forearm, and were able to visually distinguish four types of arm gestures on a data plot. The use of FMG was also investigated for monitoring cycling activity by placing FSRs on the upper leg . The research performed by G. Ogris , X. Wang  and Li et al.  showed the possibility to predict different arm and finger gestures by using multiple FSRs pressed against the arm. While their methodologies did not allow having a wearable system for real-time feedback, these works proved the feasibility of using FMG for monitoring upper-extremities gestures.
In this paper, we propose a novel system to detect different upper-extremities postures in real-time through the use of a lightweight and wearable forearm FSR strap. The strap has multiple FSR sensors, whose signals are classified in real-time to distinguish different upper-extremities postures. The FRS strap was conceived to be easy to use by a layperson. Location of the single forearm muscle groups is therefore not required every time the sensor strap is worn. The FRS strap was also designed to be a standalone device, which does not require any external equipment, such as a powerful computer or auxiliary sensors, for its calibration. The strap can therefore be used in unstructured environments, such as the patient’s home.
Among the different existing classifiers, we utilized the Extreme Learning Machine (ELM) for processing signals of our FSR strap system. The ELM was first proposed by G.Huang et.al  in 2004, and has been refined since. In recent publications, ELM has been shown to have equal or superior performance compared to the popular Support Vector Machine (SVM)  and Artificial Neural Network (ANN)  for supervised multiclass classification, but with simpler architecture and faster learning speed [24, 25]. Simple architecture and fast learning speed are crucial for our system. It should in fact be noted that in order to have an affordable and lightweight device to monitor the upper-extremities activity, a low power and low profile microcontroller would to be used. Due to the potentially low computational power available, the simplicity of the classifier’s learning algorithm is a crucial aspect. In additional, every time the FSR strap is worn, the force resistive sensors might be positioned in a different location respect to the muscle groups. The classifier has therefore to be retrained every time the strap is worn – high learning speed of the classifier is therefore a desired feature. The ELM was selected to work with the FSR strap for real-time upper-extremities posture classification for its simplicity and fast learning speed.
To evaluate the performance of the proposed system, we developed a test protocol that resembles the sequence of needed steps required to drink from a cup. The drinking task has widely been used in multiple kinds of rehabilitation interventions, including the constraint-induced movement therapy . We discretized the drinking task in a sequence of movement steps, in order to identify at which point the volunteer failed the task. By correctly classifying each step, the FRS strap can potentially provide feedback to the patient to enhance her/his motivation or to the therapist to assess the patient’s improvements. For example, in constraint-induced movement therapy, the patient is required to repeat an exercise a number of times. The quality of the movement gradually worsens with the number of repetitions because of fatigue. By assessing each movement step, the system can identify at which point the volunteer fails to correctly perform the task. Thus, by classifying the intermediate steps, the system is able to provide feedback to help the patient to maintain the quality of the exercise, as well as to provide more detailed information to the therapist for analysing the progress of the rehabilitation.
The proposed work is innovative from different perspectives. Differently from the work proposed in the literature, we used a portable and minimalistic FSR array to capture FMG patterns of the forearm, which enables us to distinguish complex upper limb posture that involves multiple joint movements. The proposed system is also very simple to be worn and used; for instance, the muscle location is not needed to be identified before placing the FSR strap. This work presents real-time FMG classification, which, to the best of the authors’ knowledge, has not been analysed or presented in previous works. The use of real-time FMG classification through the FSR strap is proposed in the interesting task of classifying arm postures during the well-known drinking task.
FSR strap and its placement
Data acquisition setup
Pattern recognition with non-kernel based extreme learning machine (ELM)
where h(x) is the hidden-layer output corresponding to the input samples from the 8 FSR (x ϵ R8), and β is the output weight vector between the hidden layer and the output layer. For multiclass classification, the predicted class label is the index number of the output node that has the highest value.
where h(a i , b i , x) is a nonlinear piecewise continuous function with i ranging from 1 to L. The parameter a i and b i of h(a i , b i , x) can be randomly generated according to any continuous distribution. Once generated, they can be reused as long as the number of input features and number of hidden nodes do not change. The choice of h(a i , b i , x) is large, for example it can be a sigmoid, Gaussian, sine, hard-limit, triangular or the radial based functions.
where N is the number of training samples.
With the hidden-layer output function decided, the number of hidden node (L) and the regularization parameter (C) were then selected empirically. The common practice for selecting the application dependent parameters, such as L and C, is to use cross validation technique when each time the classifier is trained; however, this approach requires large amount of computation, so it is not suitable for our portable system. Fortunately, the performance of the non-kernel based ELM is not very sensitive to the choice of the parameters L and C. By increasing the value of L, the classification accuracy increases until it reaches plateau; after that, little improvement can be gained. By choosing a large value for L, high accuracy is guaranteed. However, the larger the L is, the more memory is required. The selection process for the parameter C is different; a large value does not guarantee to have good performance. However, a suitable value for C can be chosen within optimal range. The optimal range of C was empirically found to be between 25 and 211 for this application. Due to the fact that the suitable value of L and C do not have to be unique, there is no need to use cross validation technique, which allows the classifier to be quickly trained. The selected values of L and C were respectively 200 and 27, and they were used throughout the experiment for all the participants.
During the training phase, the participant was asked to recreate the six postures shown in Figure 4, and maintain each posture for 7 seconds. During this period, the operator instructed the custom made LabVIEW application to record 5 seconds of data. The entire sequence was repeated 3 times during this phase.
During the testing phase, the FSR strap data went through the same filtering process and then was scaled down using the normalization parameters obtained during the training phase (see Step B2 in Figure 6). The scaled sample was classified in real-time (see Step B3 in Figure 6). The classification result along with the processed FSR strap sample and the action command (visual command provided to the volunteers) were recorded for analysis (see Step B4 in Figure 6). Note that even though the prediction was performed in real-time, there was no feedback for the participant; only the operator could see the instant (less than 200 ms delay) classification results.
Proximal forearm circumference (cm)
Training dataset analysis
Figure 7 also shows that the FMG pattern of the same participant varied slightly among the three repetitions that were performed. Besides variations due to the different amounts of force applied by the participant during the different repetitions, FMG pattern changes were also caused by the small displacements of the FSR strap on the forearm during the movements of the upper extremity. Figure 7 shows that the training of the classifier has taken these small variations into account.
Real-time classification result analysis
Time delay for action response
Section 1 (s)
Section 2 (s)
Section 3 (s)
Real-time test result
Real-time classification accuracy in%
Class label of most misclassified output data
Accuracy of the most misclassified output data in%
3 & 5
Offline test result with randomly generated base for ELM
Real-time recorded classification result (%)
Classification result with random base 1 (%)
Classification result with random base 2 (%)
Classification result with random base 3 (%)
Classification result with random base 4 (%)
Classification result with random base 5 (%)
Analysis for data misclassification
Limitations and future work
The scope of the current work is limited to discrete classification of different postures of the volunteers’ upper-extremities. While this approach can be used to provide valuable information to patients and therapists, future work will address classification of continuous movements using FMG signals. The main challenge for implementing this feature is the need of automatically identifying the signature of different types of multi-joints movements involved in a functional task.
Research was performed towards the development of an affordable and easy to use wearable system that processes force myographic (FMG) signals to provide instantaneous feedback (less than 200 ms) about activities of the upper extremities. A novel force sensing resistor strap for the forearm was developed to capture FMG patterns associated to upper-extremities movements. We utilized extreme learning machine (ELM) to extract the FMG patterns. Specifically, the ELM classifier was implemented in LabVIEW to classify in real-time six different upper-extremities postures associated to the six different steps required to drink from cup. Six healthy volunteers followed a test protocol that was designed to resemble the complete sequence of the drinking task. The average real-time testing accuracy was 92.33% with a standard deviation of 3.19%. This result shows that the proposed device and the use of FMG are potentially suitable to provide accurate feedback to the users about functional movements of their upper extremities.
This work was supported by Natural Sciences and Engineering Research Council of Canada (NSERC) and the Canadian Institutes of Health Research (CIHR).
- Patel S, Park H, Bonato P, Chan L, Rodgers M: A review of wearable sensors and systems with application in rehabilitation. J Neuroeng Rehabil 2012, 9: 21. 10.1186/1743-0003-9-21View ArticlePubMedPubMed CentralGoogle Scholar
- Dobkin BH, Dorsch A: The promise of mHealth: daily activity monitoring and outcome assessments by wearable sensors. Neurorehabil Neural Repair 2011, 25: 788-798. 10.1177/1545968311425908View ArticlePubMedPubMed CentralGoogle Scholar
- Bravata DM, Smith-Spangler C, Sundaram V, Gienger AL, Lin N, Lewis R, Stave CD, Olkin I, Sirard JR: Using pedometers to increase physical activity and improve health: a systematic review. JAMA 2007, 298: 2296-2304. 10.1001/jama.298.19.2296View ArticlePubMedGoogle Scholar
- Croteau KA: A preliminary study on the impact of a pedometer-based intervention on daily steps. Am J Health Promot 2004, 18: 217-220. 10.4278/0890-1171-18.3.217View ArticlePubMedGoogle Scholar
- Cooke EV, Mares K, Clark A, Tallis RC, Pomeroy VM: The effects of increased dose of exercise-based therapies to enhance motor recovery after stroke: a systematic review and meta-analysis. BMC Med 2010, 8: 60. 10.1186/1741-7015-8-60View ArticlePubMedPubMed CentralGoogle Scholar
- Wolf S, Lecraw D, Barton L, Jann B: Forced use of hemiplegic upper extremities to reverse the effect of learned nonuse among chronic stroke and head-injured patients. Exp Neurol 1989, 104: 125-132. 10.1016/S0014-4886(89)80005-6View ArticlePubMedGoogle Scholar
- Backus D: Exploring the potential for neural recovery after incomplete tetraplegia through nonsurgical interventions. PM R 2010, 2: S279-S285. 10.1016/j.pmrj.2010.10.004View ArticlePubMedGoogle Scholar
- Gordon AM: To constrain or not to constrain, and other stories of intensive upper extremity training for children with unilateral cerebral palsy. Dev Med Child Neurol 2011,53(Suppl 4):56-61.View ArticlePubMedGoogle Scholar
- Actical wrist accelerometer http://www.bio-lynx.com/actical.html
- Actigraph wrist accelerometer http://www.theactigraph.com/
- Cyberglove http://www.cyberglovesystems.com/
- Dipietro L, Sabatini AM, Dario P: Artificial neural network model of the mapping between electromyographic activation and trajectory patterns in free-arm movements. Med Biol Eng Comput 2003, 41: 124-32. 10.1007/BF02344879View ArticlePubMedGoogle Scholar
- Khokhar ZO, Xiao ZG, Menon C: Surface EMG pattern recognition for real-time control of a wrist exoskeleton. Biomed Eng Online 2010, 9: 41. 10.1186/1475-925X-9-41View ArticlePubMedPubMed CentralGoogle Scholar
- Tavakolan M, Xiao ZG, Menon C: A preliminary investigation assessing the viability of classifying hand postures in seniors. Biomed Eng Online 2011, 10: 79. 10.1186/1475-925X-10-79View ArticlePubMedPubMed CentralGoogle Scholar
- Wininger M, Kim N-H, Craelius W: Pressure signature of forearm as predictor of grip force. J Rehabil Res Dev 2008, 45: 883-92. 10.1682/JRRD.2007.11.0187View ArticlePubMedGoogle Scholar
- Amft O, Troster G, Lukowicz P, Schuster C: Sensing Muscle Activities with Body-Worn Sensors. In Int Work Wearable Implant Body Sens Networks. Cambridge, MA: IEEE; 2006:138-141.View ArticleGoogle Scholar
- Kreil M, Ogris G, Lukowicz P: Muscle activity evaluation using force sensitive resistors. In 2008 5th Int Summer Sch Symp Med Devices Biosens. Hong Kong: IEEE; 2008:107-110.View ArticleGoogle Scholar
- Ogris G, Kreil M, Lukowicz P: Using FSR based muscule activity monitoring to recognize manipulative arm gestures. In 2007 11th IEEE Int Symp Wearable Comput. Boston, MA: IEEE; 2007:1-4.View ArticleGoogle Scholar
- Wang X, Zhao J, Yang D, Li N, Sun C, Liu H: Biomechatronic approach to a multi-fingered hand prosthesis. In 2010 3rd IEEE RAS EMBS Int Conf Biomed Robot Biomechatronics. Tokyo: IEEE; 2010:209-214.View ArticleGoogle Scholar
- Li N, Yang D, Jiang L, Liu H, Cai H: Combined use of FSR sensor array and SVM classifier for finger motion recognition based on pressure distribution map. J Bionic Eng 2012, 9: 39-47. 10.1016/S1672-6529(11)60095-4View ArticleGoogle Scholar
- Huang G-B: Extreme learning machine: a new learning scheme of feedforward neural networks. In 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541). IEEE 2004, 2: 985-990.Google Scholar
- Cortes C, Vapnik V: Support-vector networks. Mach Learn 1995, 20: 273-297.Google Scholar
- Lippmann R: An introduction to computing with neural nets. IEEE ASSP Magazine 1987, 4: 4-22.View ArticleGoogle Scholar
- Huang G-B, Ding X, Zhou H: Optimization method based extreme learning machine for classification. Neurocomputing 2010, 74: 155-163. 10.1016/j.neucom.2010.02.019View ArticleGoogle Scholar
- Huang G-B, Zhou H, Ding X, Zhang R: Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern B Cybern 2012, 42: 513-29.View ArticlePubMedGoogle Scholar
- Bonifer N, Anderson KM: Application of constraint-induced movement therapy for an individual with severe chronic upper-extremity hemiplegia. Phys Ther 2003, 83: 384-98.PubMedGoogle Scholar
- FSR® 402 http://www.interlinkelectronics.com/FSR402.php
- Liarokapis MV, Artemiadis PK, Kyriakopoulos KJ, Manolakos ES: A learning scheme for reach to grasp movements: on EMG-based interfaces using task specific motion decoding models. IEEE J Biomed Heal Informatics 2013, 17: 915-921.View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.