- Open Access
Using visual feedback distortion to alter coordinated pinching patterns for robotic rehabilitation
© Matsuoka et al; licensee BioMed Central Ltd. 2007
- Received: 18 April 2006
- Accepted: 30 May 2007
- Published: 30 May 2007
It is common for individuals with chronic disabilities to continue using the compensatory movement coordination due to entrenched habits, increased perception of task difficulty, or personality variables such as low self-efficacy or a fear of failure. Following our previous work using feedback distortion in a virtual rehabilitation environment to increase strength and range of motion, we address the use of visual feedback distortion environment to alter movement coordination patterns.
Fifty-one able-bodied subjects participated in the study. During the experiment, each subject learned to move their index finger and thumb in a particular target pattern while receiving visual feedback. Visual distortion was implemented as a magnification of the error between the thumb and/or index finger position and the desired position. The error reduction profile and the effect of distortion were analyzed by comparing the mean total absolute error and a normalized error that measured performance improvement for each subject as a proportion of the baseline error.
The results of the study showed that (1) different coordination pattern could be trained with visual feedback and have the new pattern transferred to trials without visual feedback, (2) distorting individual finger at a time allowed different error reduction profile from the controls, and (3) overall learning was not sped up by distorting individual fingers.
It is important that robotic rehabilitation incorporates multi-limb or finger coordination tasks that are important for activities of daily life in the near future. This study marks the first investigation on multi-finger coordination tasks under visual feedback manipulation.
- Index Finger
- Visual Feedback
- Finger Movement
- Coordination Pattern
- Target Line
Stroke and other neurological disorders affect more and more people as the general population ages. Early rehabilitation increases the chance that patients retain the ability to function in activities of daily life (ADL). To assist this functional gain, patients are often taught compensatory movements that are functionally effective for the level of disability they have at the time of rehabilitation. However, even if these patients regain additional movements over time, it is common for them to continue using the compensatory movements taught due to entrenched habits, increased perception of task difficulty, or personality variables such as low self-efficacy or a fear of failure[3, 4]. If there is a technique to allow these patients to move away from the entrenched compensatory movements so that other potentially more effective movements can be explored and practiced, it may lead to increased function in ADL. We hypothesize that this type of exploration may be possible using virtual rehabilitation environments that can distort visual feedback of patients' movements away from their current habit without their awareness.
To date virtual and robotic rehabilitation environments have focused on increasing strength and range of motion of a single limb [1–6]. Patton et al. (2001) used the perturbation force profile to strengthen muscles and extend the range of motion. Our group has shown that visual feedback distortion in a virtual environment enables both able-bodied and traumatic brain injury (TBI) subjects to produce more force and move further than their perceived movements[4, 7]. The distortion is remapping between the actual movements and the virtual visual feedback the subjects receive about their movements. The Just Noticeable Difference (JND) defines the lowest amount of discrepancy between the actual movements and the virtual visual feedback. As long as the distortion is less than the JND, the subjects are unable to detect the distortion[4, 8]. Subsequently, we showed that young, elderly, and disabled subjects were able to increase force and distance moved, without perceiving the difference and without an increased amount of perceived effort[4, 9]. When this feedback distortion environment was used for rehabilitation, we witnessed a long-term increase of the maximum force production and range of motion of stroke and chronic TBI subjects[4, 6].
It is important that robotic rehabilitation incorporates multi-limb or finger coordination tasks that are important for ADL in the near future (note: in this paper, the thumb is also called a finger for simplicity). None of the previous research in robotic or virtual rehabilitation to our knowledge, however, addresses coordination tasks. The purpose of the visual feedback distortion for individual finger training was to change the force or distance goal corresponding to each level, thus challenging the maximum force production or range of motion. In the multi-finger case, distorting visual feedback of all fingers equally does not result in the overall change in coordination. The present work used a different distortion paradigm that we call "error enhancement" on individual fingers. According to this paradigm, subjects were asked to work at a demanding task with an objective performance criterion. Departures from ideal performance were displayed as errors, which were distorted to appear larger. Some evidence for the utility of this approach for one limb comes from Wei et al, who showed that magnifying visual error by physically displacing the arm's trajectory resulted in smoother and straighter trajectories.
In this paper, we investigated whether we could train able-bodied individuals to achieve a prescribed pinching movement with distort visual feedback of individual fingers consisting of the error relative to the desired movement. Specifically, we aimed to answer (1) whether different coordination pattern could be trained with visual feedback and have the learned pattern transfer to trials without visual feedback, (2) whether distorting individual finger at a time would affect the amount of error reduction for both fingers, and (3) whether overall learning would be sped up by distorting individual fingers.
Once the pinching movement had been isolated from the rest of the hand and arm movements, the experimenter moved the subject's index finger and thumb back and forth starting from the widest pinch (100% calibrated pinch span) and ending with their fingers touching each other (0% calibrated pinch span) for calibration. The experimenter moved the fingers to assure that both fingers span enough distance during this calibration. For the thumb-only conditions, the experimenter moved the thumb alone from 100% to 0% of the pinching span. While subject's fingers were moved back and forth, fingers' mean pinch span limits and trajectory were calculated.
Visual feedback setup
A target line crossing both bars moved up and down to indicate the desired finger movement trajectory. The line started at the bottom of the bars, and as soon as the subject crossed the line with the shaded portion of either bar, the line began to move. On every trial, the line moved for 8 seconds according to the equation where Δt is the time in seconds since the beginning of the trial and L is the normalized position of the line along the bars (Figure 2(b)). This specific movement profile was chosen as a challenging but learnable coordination pattern so that we can observe a learning trend over tens of trials.
Trials without distortion
Trials without distortion provided the true normalized finger movements as the visual feedback. The trial absolute error (the mean absolute difference between the normalized position of each finger and the target line) was displayed to the subject after each trial.
Trials with distortion
Trials with distortion had a magnification of the error between the finger position and the position of the target line by 20%. This distorted visual feedback was provided in real time and the numerical error given to the subject at the end of each trial was also increased by 20%.
Trials with no-feedback
After the first 20 trials, one no-feedback trial was incorporated randomly every 20 trials. On the no-feedback trials, the normalized position of each finger was not shown; the screen appeared the same, except that no part of the visual feedback bars was shaded. These trials were included to assess whether subjects could reproduce the target movement without feedback about finger position.
Three experimental conditions. Finger that received visual feedback distortion is listed for five conditions tested.
Number of Subjects
ITB (index-thumb-both) condition
The ITB condition (along with TIB and C below) was designed to address (1) whether different coordination pattern could be trained with visual feedback and have the learned pattern transfer to trials without visual feedback, (2) whether distorting individual finger at a time would affect the amount of error reduction for both fingers, and (3) whether overall learning would be sped up by distorting individual fingers.
This condition consisted of 200 trials with the first 80 trials designated for establishing baseline without distortion. The subject then encountered a section of 40 trials (trials 81–120) in which the visual feedback for the index finger was distorted, followed by a section of 40 trials (trials 121–160) in which the visual feedback for the thumb was distorted. The experiment concluded with a section of 40 trials (trials 161–200) in which the visual feedback for both the index finger and the thumb was distorted.
TIB (thumb-index-both) condition
The procedure for the TIB condition was similar to the ITB procedure, except that the section of trials with distorted thumb feedback occurred before the section of trials with distorted index finger feedback. This condition was included to capture the effect of the thumb being distorted as the first distorted finger (in ITB, we can only observe the effect of thumb distortion after the thumb performance was influenced by the index finger distortion), and to compare whether there is any overall learning difference by distorting the thumb first in stead of the index finger.
C (control) condition
In the C condition, all 200 trials contained no distortion. This condition acted as the controls for ITB and TIB conditions to show whether the individual finger distortion changed the shape of error reduction profile and whether the overall learning could be sped up.
NTN (thumb-only condition mirroring ITB) condition
In ITB, TIB, and C conditions, subjects learned to move two fingers at the same time. To understand whether and/or how much of the change in the shape and speed of error reduction came from the fact that subjects were learning two finger motions at the same time, we conducted a similar experiment as the ITB condition without the index finger movement.
This condition consisted of 160 trials with the first 80 trials designated for establishing baseline without distortion. To mimic the ITB condition, the subject then encountered additional 40 trials (trials 81–120) without distortion (for the ITB, the index finger was distorted during trials 81–120). Then the following 40 trials (trials 121–160) had distortion in the visual feedback for the thumb.
TNN (thumb-only condition mirroring TIB) condition
The procedure for the TNN condition was similar to the NTN procedure, except it mirrored the TIB condition. Therefore, the subject encountered thumb distortion during trials 81–120, and no distortion during trials 121–160.
Post-experiment questionnaires were provided to assess whether the subjects detected distortion, whether they used specific movement strategies, and their perception of task difficulty.
For data analysis, the experiments are divided into blocks of 20 trials. First 20 trials are considered to be practice and Block 1 corresponds to trials 21 to 40 and so forth until Block 9 which corresponds to trials 181 to 200. There were four sections corresponding to trials 41–80 (Section 1), trials 81–120 (Section 2), trials 121–160 (Section 3), and trials 161–200 (Section 4). These sections were typically under different distortion types (except for the control condition).
We define a few different measures to analyze the effect of the different distortion modes. Trial absolute error is the mean absolute difference between the normalized position of each finger and the goal finger location. Mean absolute error is the sum of the trial absolute error for each block or section for each subject. Normalized error is the mean absolute error divided by the mean absolute error for the first block/section to remove the inter-subject variability in baseline performance. Mean absolute difference is computed over trials and over subjects between the normalized position of the index finger and the normalized position of the thumb. All four measures are unit-less. Mean lag (in seconds) is a mean computed over trials, fingers and subjects of the difference in time that maximized the correlation between the finger position and target line position for each trial and finger. Essentially, this quantity measures the time period by which the subject's response trailed the target movement.
A contrast analysis was conducted when the distortion switched from one finger to another to test whether the crossover trend between the increasing error in one finger and the decreasing error in another finger was significant. Repeated-measures ANOVAs were conducted for index finger and the thumb. Test condition was a between subjects factor, and section was a within-subjects factor.
Different coordination patterns were learned
Distortion effect on error reduction
When the thumb was distorted in section 3 for the ITB condition, the error reduction for the thumb was significantly more than for the C condition (p = 0.005). Meanwhile, the error reduction for the index finger error was significantly less than for the C condition (p = 0.003). The contrast analysis showed the contrast of 0.17 for the ITB condition and -0.063 (negative contrast means the index finger dropped more than the thumb) for the C condition, and the crossover trend was significant for both conditions (p < 0.001 for ITB and p = 0.003 for C).
Repeated-measures ANOVA for the index finger showed no significant main effect of conditions (p = 0.21), significant main effect of section (p < 0.001), and significant interaction of condition and section (p < 0.001). Repeated-measures ANOVA for the thumb showed no significant main effect of conditions (p = 0.83), significant main effect of section (p < 0.001) and significant interaction of condition and section (p < 0.001). The main effect of section was due to learning done after trial 80, since trials before that weren't included in the analysis.
Data from NTN and TNN conditions were analyzed with ANOVA for between-subjects factor of condition and within-subjects factor of section. No significant main effect of subject condition was observed (p = 0.58). No significant main effect of section was observed (p = 0.23) indicating no learning past trial 80 since this was a simpler task than the two finger coordination tasks (such as ITB, TIB and C). And finally there was no significant interaction between the TNN and NTN conditions (p = 0.99) showing the order of distortion didn't change the results.
Distortion did not change the final performance
To determine whether distortion affected terminal performance, we compared the mean normalized error in section 4 for each distortion condition to the C condition. None of these comparisons was significant (ITB: p = 0.07 for index, 0.69 for thumb; TIB: p = 0.14 for index finger, 0.99 for thumb).
Post-experiment questionnaires revealed that none of the subjects detected the distortion. All but four subjects stated that they tried to move the finger and the thumb in a coordinated way. And all subjects stated that the task required a significant amount of concentration and felt that the task was extremely difficult.
Altering the coordination pattern among multiple fingers is considered to be a challenging problem. Latash et al conducted an experiment where subjects ramped up the sum of the static forces produced by all their fingers shown on the computer screen from zero to a designated force. When subjects were informed of the coefficients used for the fingers (i.e. some finger forces were multiplied by 0.5 or 2 before being summed), subjects immediately changed their force production pattern to compensate for those changes. When subjects were not informed of the coefficients, no adaptation to the distorted feedback was observed and they used the same coordination pattern as when all the coefficients were 1. This result shows the difficulty in changing the coordination pattern among multiple fingers. It is, however, interesting to note that our results showed the change in the coordination pattern between the index finger and the thumb carried over to the no-feedback trials.
For all subjects in our experiment, the habitual pinching pattern was not symmetric between the index finger and the thumb. Despite their different habitual coordination patterns, we were able to train subjects to use the same new coordination pattern with visual feedback guidance. This learning was confirmed to have taken place even without the presence of the visual feedback during the interleaved trials with no feedback. In this task, subjects learned a particular pattern of movement as they tried to minimize visual error. Because emphasis was placed on the visual error, it is no surprise that a subject's error was greater when executing the learned movement without visual feedback of position. However, the mean total absolute error in the no-feedback case still decreased as the experiment progressed. More transfer to the no-feedback case might have been observed if more no-feedback trials had been included. For effective learning and transfer of a motor task to occur, subjects may need to learn to use internal cues rather than relying on extrinsic feedback[13, 14].
Although the TIB condition more closely mimicked the C condition, they too showed a steep decline in index finger error and a counter-learning trend in thumb error when distortion shifted from the thumb to the index finger (section 2 to section 3). The lack of significant differences between TIB and C conditions can be explained by the fact that the trial absolute error for the thumb was significantly larger than that of the index finger for the C condition at the beginning of the experiment. Both C and TIB subjects saw the thumb error as larger in section 2, and both conditions focused on minimizing that error. The thumb error was not significantly different from the index finger error for the C condition at the end of the experiment. Thus, as the experiment proceeded, subjects in the C condition may have worked on error reduction for both fingers more evenly. This may parallel the focus of TIB subjects to the index finger error and then to both fingers.
It is interesting to note that there was a trend for the non-distorted finger's error to increase. It is possible that the subjects were unable to maintain the performance of the non-distorted finger while they were concentrating on reducing the error of the distorted finger. This view is consistent with the idea of minimizing variance in task space while the motor variability appears in the uncontrolled manifold. This implies that the focus was originally divided between two fingers to achieve two tasks at the same time. When the distortion emphasized the error of one of the fingers, the task requirement for that finger went up, and the amount of task-level focus that could be provided for the non-distorted finger decreased. This is consistent with the fact that all of the subjects indicated in the questionnaire that the task was extremely difficult and that it required a high level of concentration at all times. If the task is made easier, it may be possible to devote more focus on the distorted finger and reduce the error, while not compromising the non-distorted finger task. In addition, if the distortion changes the task goal (as in our previous studies and therapeutic paradigms) rather than enhancing on the error, the result may be different. This is a key issue that must be addressed, in order to investigate and retrain movement coordination patterns.
The TIB and ITB conditions did not perform better than the C condition when both finger movements were distorted together. There was no difference in performance between the index finger and the thumb because they were treated identically. This fact shows that simply exaggerating the error for both fingers together is not effective. Also, in the TNN and NTN conditions, distortion had no effect on the normalized error of subjects. These results are similar to those of Patton, which reported that error augmentation using a multiplicative gain did not improve terminal performance in a reaching task. Error augmentation through a constant offset was found to be more effective, but that type of distortion would not be relevant for the task we considered.
It is a common fear that the learned effect in a distorted environment would "wash out" to the baseline immediately after the training took place. However, when working with disabled individuals, these effects have been shown to not wash out (unlike for able-bodied individuals)[3, 6]. Learned effects do not wash out for disabled individuals because they may have learned to activate different sets of muscles during the distorted feedback training. When the distorted feedback is removed, they are left with the new coordination patterns that they practiced repetitively during therapy, and which work to accomplish tasks in daily life.
Our ultimate goal is to manipulate visual feedback in a virtual robotic rehabilitation environment to steer patients away from entrenched coordination habit that may not be best for them (e.g. either because they have the muscular strength to improve on task performance, or because this habitual movement is causing other physical problems such as tendonitis). We showed that (1) training under visual feedback allowed new coordination pattern to transfer to no-feedback trials, (2) feedback distortion changed the amount of error reduced for each finger separately, and (3) distorting individual fingers separately (or together) did not affect the overall speed of learning in movement error reduction. By interleaving no visual feedback trials more often, a dependence on the visual display may be avoided and may allow better transfer of the new coordination strategy to daily activities. This study was not conducted to show that impaired or unimpaired people would be able to achieve "better" performance when the visual feedback manipulation was provided. Rather, we initiated an important investigation on multi-limb coordination tasks under visual feedback manipulation.
This project was partially funded by NIDRR grant H133A020502 and NSF PECASE award 0238204.
- Reinkensmeyer DJ, Kahn LE, Averbuch M, McKenna-Cole A, Schmit BD, Rymer WZ: Understanding and treating arm movement impairment after chronic brain injury: progress with the ARM guide. J Rehabil Res Dev. 2000, 37 (6): 653-662.PubMedGoogle Scholar
- Taub E, Crago JE: Constraint-induced movement therapy: a new approach to treatment in physical rehabilitation. Rehabilitation Psychology. 1998, 43: 152-170. 10.1037/0090-55188.8.131.52.View ArticleGoogle Scholar
- Patton JL, Mussa-Ivaldi FA, Rymer WZ: Altering movement patterns in healthy and brain-injured subjects via custom designed robotic forces. 2001, 2: 1356-1359.Google Scholar
- Brewer BR, Fagan M, Klatzky RL, Matsuoka Y: Perceptual limits for a robotic rehabilitation environment using visual feedback distortion. IEEE Trans Neural Syst Rehabil Eng. 2005, 13 (1): 1-11. 10.1109/TNSRE.2005.843443.View ArticlePubMedGoogle Scholar
- Lum PS, Burgar CG, Shor PC, Majmundar M, Van der Loos M: Robot-assisted movement training compared with conventional therapy techniques for the rehabilitation of upper-limb motor function after stroke. Arch Phys Med Rehabil. 2002, 83 (7): 952-959. 10.1053/apmr.2001.33101.View ArticlePubMedGoogle Scholar
- Brewer BR, Klatzky RL, Matsuoka Y: Initial Therapeutic Results on Visual Feedback Manipulation in Robotic Rehabilitation. 2006, submitted-Google Scholar
- Brewer BR, Klatzky RL, Matsuoka Y: Visual Feedback Distortion in a Robotic Rehabilitation Environment. 2006Google Scholar
- Matsuoka Y, Allin SJ, Klatzky RL: The tolerance for visual feedback distortions in a virtual environment. Physiol Behav. 2002, 77 (4-5): 651-655. 10.1016/S0031-9384(02)00914-9.View ArticlePubMedGoogle Scholar
- Fagan M, Matsuoka Y, Klatzky RL: Feedback Distortion for Rehabilitation: Gauging Perceived Physical Effort. 2003, 159-165.Google Scholar
- Wei Y, Bajaj P, Scheidt R, Patton JL: A real-time haptic/graphic demonstration of how error augmentation can enhance learning. 2005, 4406-4411.Google Scholar
- SensAble Technologies I: . www.sensable.comGoogle Scholar
- Latash ML, Gelfand IM, Li ZM, Zatsiorsky VM: Changes in the force-sharing pattern induced by modifications of visual feedback during force production by a set of fingers. Exp Brain Res. 1998, 123 (3): 255-262. 10.1007/s002210050567.View ArticlePubMedGoogle Scholar
- Proteau L, Isabelle G: On the role of visual afferent information for the control of aiming movements toward targets of different sizes. J Mot Behav. 2002, 34 (4): 367-384.View ArticlePubMedGoogle Scholar
- Flinn NA, Radomski MV: Learning. Occupational therapy for physical dysfunction. Edited by: Trombly CA, Radomski MV. 2002, Philadelphia , Lippincott, Williams and Wilkins, 5th EditionGoogle Scholar
- Todorov E, Jordan MI: Optimal feedback control as a theory of motor coordination. Nat Neurosci. 2002, 5 (11): 1226-1235. 10.1038/nn963.View ArticlePubMedGoogle Scholar
- Patton JL, Phillips-Stoykov ME, Stojakovich A, Mussa-Ivaldi FA: Evaluation of robotic training forces that wither enhance or reduce error in chronic hemiparetic stroke survivors. Exp Brain Res.Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.