- Open Access
Activities of daily living with bionic arm improved by combination training and latching filter in prosthesis control comparison
Journal of NeuroEngineering and Rehabilitation volume 18, Article number: 45 (2021)
Advanced prostheses can restore function and improve quality of life for individuals with amputations. Unfortunately, most commercial control strategies do not fully utilize the rich control information from residual nerves and musculature. Continuous decoders can provide more intuitive prosthesis control using multi-channel neural or electromyographic recordings. Three components influence continuous decoder performance: the data used to train the algorithm, the algorithm, and smoothing filters on the algorithm’s output. Individual groups often focus on a single decoder, so very few studies compare different decoders using otherwise similar experimental conditions.
We completed a two-phase, head-to-head comparison of 12 continuous decoders using activities of daily living. In phase one, we compared two training types and a smoothing filter with three algorithms (modified Kalman filter, multi-layer perceptron, and convolutional neural network) in a clothespin relocation task. We compared training types that included only individual digit and wrist movements vs. combination movements (e.g., simultaneous grasp and wrist flexion). We also compared raw vs. nonlinearly smoothed algorithm outputs. In phase two, we compared the three algorithms in fragile egg, zipping, pouring, and folding tasks using the combination training and smoothing found beneficial in phase one. In both phases, we collected objective, performance-based (e.g., success rate), and subjective, user-focused (e.g., preference) measures.
Phase one showed that combination training improved prosthesis control accuracy and speed, and that the nonlinear smoothing improved accuracy but generally reduced speed. Phase one importantly showed simultaneous movements were used in the task, and that the modified Kalman filter and multi-layer perceptron predicted more simultaneous movements than the convolutional neural network. In phase two, user-focused metrics favored the convolutional neural network and modified Kalman filter, whereas performance-based metrics were generally similar among all algorithms.
These results confirm that state-of-the-art algorithms, whether linear or nonlinear in nature, functionally benefit from training on more complex data and from output smoothing. These studies will be used to select a decoder for a long-term take-home trial with implanted neuromyoelectric devices. Overall, clinical considerations may favor the mKF as it is similar in performance, faster to train, and computationally less expensive than neural networks.
Modern prosthetic hands can now recreate the complex movements of the human hand [1,2,3,4,5,6,7,8]. Unfortunately, unintuitive control makes prostheses difficult to use  and is a major factor in prosthesis dissatisfaction and abandonment . Neural and electromyographic (EMG) signals from residual muscles provide a rich source of movement information that could be used to provide intuitive prosthesis control. With rare exceptions [11, 12], commercial prosthetic control uses only two-electrode EMG setups that provide sequential control of up to two degrees-of-freedom [1, 3, 5, 6, 13, 14], far inferior to the abilities of modern prosthetic hands.
Using more neuromuscular inputs can provide more intuitive prosthesis control through richer, more diverse data that can be classified into different movements based on residual neuromuscular activation patterns. Several groups have demonstrated intuitive prosthesis control with decoders that classify distinct movements [15,16,17,18,19,20,21,22]. Generally, these decoders classify EMG patterns to several pre-determined grip patterns (e.g., close hand, open hand, pinch). More advanced classification strategies incorporate proportionality into the classes (e.g., enable partial hand closure)  or allow simultaneously active classes (e.g., simultaneously rotate wrist and close hand) . Classifiers can be limited by allowing the user only a predetermined, fixed number of movement types.
Continuous decoders provide another approach to intuitively control a prosthesis. Although nomenclature varies in the literature, here we refer to a continuous decoder as a decoder that predicts kinematic positions anywhere within some movement window (i.e., an infinite number of positions) and learns from continuous kinematic values, not classes with predefined kinematic values. We choose not to refer to continuous decoders as regression-based because not all continuous decoders use regression [24, 25]. Instead of distinct classes of movements, continuous decoders allow independent and simultaneous movement of individual degrees-of-freedom (e.g., finger or wrist movements), allowing the user to potentially produce any grip pattern. Continuous decoders have been used to provide prosthesis control with linear [26,27,28] and nonlinear [29,30,31] algorithms. Nonlinear algorithms generally take longer to train and are subject to overfitting; however, they may capture the nonlinear nature of multi-dimensional EMG data , especially for complex movement patterns. Recently, hybrid decoders have combined classification and regression control with promising results [33, 34].
Kalman filters, and related variations, provide a linear approach to continuous control that has been used with cortical  and peripheral [26, 36] recordings. Multi-layer perceptrons (MLPs), a relatively simple artificial neural network, have been used for both classification [32, 37, 38] and continuous [29, 39,40,41] control. Convolutional neural networks (CNNs), which, in the prosthetic domain, convolve temporally over their input, have also been used for classification [42, 43] and continuous [30, 31] control. CNNs temporal convolution enables them to learn features from raw EMG data [31, 42], whereas other strategies typically use time-domain features, such as the mean absolute value of segmented EMG.
Beyond the algorithm itself, the data used to train the algorithm affects prosthesis control. Training an algorithm in multiple arm positions can improve prosthesis control whether the position data is used to train the algorithm [32, 44] or not , although incorporating positional data may not benefit linear algorithms . For continuous controllers without a discrete set of classes, using training data with simultaneous movements of more than one degree-of-freedom may expose nonlinearities in EMG data and preferentially benefit nonlinear algorithms with greater learning capacities.
Modifying a continuous decoder’s output can improve prosthesis control by improving stability. Smoothing filters, such as a traditional low-pass filter can smooth the output and reduce unwanted jitter . One study incorporated a camera system to selectively low-pass filter the output when a grasp was detected . Low-pass filters inevitably introduce delay into the control. Nonlinear smoothing filters can reduce small-amplitude jitter without slowing larger movements [49, 50].
Prosthesis control improvements are generally demonstrated sequentially in three domains: offline, online, and real-world. Offline comparisons measure the ability of a decoder to map EMG inputs to kinematic outputs, without the user in the loop. Several offline comparisons have helped improve the state of the art for both continuous  and classification [17, 18, 32, 43, 51,52,53] control. Although they are a useful first step in improving control, offline improvements do not necessarily imply online improvements , largely because they do not incorporate the continuous interaction of the user with the controller. Online improvements incorporate user feedback and have been demonstrated through Fitts’ law or virtual target tasks. Several online comparisons have demonstrated differences among control strategies [28, 30, 31, 37, 42, 54,55,56,57,58]. The relationship of online performance to real-world performance is debated [59,60,61]. Comparing decoders through real-world activities of daily living provides the most realistic approximation of how a decoder would function in everyday life. Real-world comparisons have demonstrated the benefits of continuous [62, 63], classifier [20, 50, 64,65,66,67], and hybrid  control strategies over commercial control; however, only a couple of comparisons have been made among research-grade control strategies [50, 67], both of which were classifiers.
In the present work, we conduct a two-phase real-world comparison of 12 continuous decoders that span the three major components affecting control (training data, algorithms, and smoothing). In the first phase, we compare training an algorithm with individual finger or wrist movements vs. combination movements (e.g., simultaneous flexion of all fingers to make a functional grasp) and compare the effect of using a nonlinear smoothing filter . We study these conditions in a two-degree-of-freedom decode using advanced, previously published, continuous control decoders: a modified Kalman filter (mKF) , an MLP , and a CNN . All variations together yield 12 different decoders for comparison. In the second phase, we performed a more extensive evaluation of five-degree-of-freedom decodes with the mKF, MLP, and CNN using the combination training and smoothing filter found most beneficial in phase one. We focus on both performance-based objective and user-centric subjective measures. To our knowledge, this study provides the first comparison of the same individuals using multiple continuous decoders and a physical prosthesis to complete real-world tasks. A large part of our motivation for this study is to identify the “best” decoder among the advanced decoders we have previously published, in part because the results of this study will guide our clinical translation of advanced prosthesis control in a long-term take-home clinical trial.
Signal acquisition and prosthesis setup
Signal acquisition has been described previously . In brief, surface EMG (sEMG) was collected from a sEMG sleeve  with the 512-channel Grapevine System (Ripple Neuro LLC, Salt Lake City, UT). Thirty-two single-ended channels were acquired at 1 kHz and filtered with a 6th-order high pass Butterworth filter (15 Hz), 2nd-order low-pass Butterworth filter (375 Hz), and 60, 120, and 180 Hz notch filters. After connecting the sEMG sleeve to the acquisition device, channels were manually inspected and removed if shorted channels were detected (generally less than two channels). The differential pairs of all monopolar channels were calculated, and features (single-ended and differential) were created at 30 Hz using the mean absolute value of a 300-ms buffer (i.e., 528 features from an overlapping 300-ms boxcar filter). At 30 Hz, the buffer is updated every 33 ms, resulting in a 266-ms overlap between features. This update rate and buffer length has been used by our group extensively with various decoders [30, 36].
Non-amputee participants donned the DEKA LUKE Arm  with a bypass socket enabling non-amputee use of a prosthetic arm, which has been shown to be representative of use by individuals with amputation . The prosthesis, bypass socket, and sEMG sleeve are shown in Fig. 1a. The bypass socket, which has a freely-rotating wrist attachment, had rubber bands providing wrist rotation resistance (0.34 Nm to 0.85 Nm at rest and maximal rotation, respectively). Baseline EMG activity was subtracted from each feature before collecting training data by subtracting the average EMG activity during a 10-s period where the user supported the weight of the bypass socket and DEKA LUKE Arm in a neutral position approximately waist height. Training data were recorded as participants mimicked the pre-programmed movements of the DEKA LUKE Arm. The kinematics from the pre-programmed movements were used to fit the algorithms described in the next section.
The mKF was implemented as has been demonstrated previously . A Gram–Schmidt orthogonalization algorithm  was used to select 48 features as inputs for the mKF. Using 48 features was empirically found to be an effective number in . Prior to selecting features, we temporally aligned the kinematics and features by maximizing the average correlation between every feature and kinematic. The 48 EMG features and kinematics were used to fit the parameters of a linear Kalman filter . The post-hoc thresholds which were used to modify the Kalman filter online  were optimized offline . The feature selection, Kalman filter fitting, and threshold optimization were completed using all the training data (i.e., no training and validation split). The mKF was trained, and the threshold optimized using MATLAB 2018b on a Windows 10 computer with 64 GB RAM and an Intel Xeon E5-2620v3. The same computer was used for runtime predictions.
The nonlinear MLP was implemented as has been demonstrated previously . The MLP consisted of two hidden layers, each with 128 nodes, and a hyperbolic tangent (tanh) activation layer. The training and testing data were split 80% and 20%, respectively. As with the mKF, the MLP used 48 input features as chosen by the same stepwise Gram-Schmidt orthogonalization algorithm to provide cleaner feature data. Prior to selecting features, we temporally aligned the kinematics and features by maximizing the average correlation between every feature and kinematic. The MLP trained for three dataset aggregation iterations, each with ten epochs (found sufficient in ). The MLP was trained in Python 3.6 using TensorFlow 1.13 on a Windows 10 computer with 64 GB RAM and an Intel Xeon E5 1650v3 (not trained on a GPU). The same computer was used for runtime predictions.
The nonlinear CNN was implemented as has been demonstrated previously . The CNN, comprising eight total layers, used all 528 features to fit the kinematic values. 75% of the data was used to train the algorithm, and the remaining 25% was used as a validation set. The network trained until the loss on the validation set was equal to or larger than the previously smallest loss for five iterations. The CNN was trained using MATLAB 2018b on an NVIDIA Quadro M4000 8 GB graphics card. The system ran on Windows 10 and had 128 GB RAM and an Intel Xeon E5-2620v4 processor. The trained network was transferred to a Windows 10 computer with 64 GB RAM and an Intel Xeon E5-2620v3 processor for runtime predictions.
After training, users were given position control of the prosthesis. In position control, the prosthesis returns to its rest position if the user is not actively making a movement. In velocity control (the counterpart of position control), the prosthesis does not move when the user rests. For tasks requiring multiple active movements, position control forces the user to complete both movements simultaneously, whereas velocity control allows the user to complete these movements sequentially.
A smoothing filter was implemented as has been demonstrated previously . The “latching filter” we used is a computationally inexpensive, recursive, nonlinear filter that smooths small-amplitude jitter but allows quick changes to its output. The latching filter nonlinearly adjusts its level of smoothing based on the difference between the previous, smoothed decoder estimate and the current decoder estimate to determine how much to move in the direction of the current decoder estimate (i.e., how much to smooth the transition from the previous estimate to the new estimate). For small estimate changes, the output is heavily smoothed, resulting in stability for low-amplitude movements. For large estimate changes, the output is nearly unsmoothed, resulting in quick movements. The latching filter makes the reasonable assumption that jitter amplitude is small relative to the movement range and that jitter occurs more frequently than do intended small movements.
We conducted this study in two phases (Fig. 2a). In the first phase, we compared training paradigms and the use of a smoothing filter for all three algorithms, totaling 12 decoders: three algorithms, two training paradigms, and two output types. Due to the number of decoders and the consequential experiment duration, we selected a simple task, the clothespin relocation task, with a low, two degree-of-freedom control strategy (grasp and wrist rotation), which simplified the training and allowed the experiments to be completed in 2–3 h.
In phase two, we increased the decoder output to five degrees-of-freedom to evaluate the three algorithms’ control in a high-degree-of-freedom setting representative of modern myoelectric prostheses. The degrees-of-freedom used were: (1) thumb flexion, (2) index finger flexion, (3) middle, ring, and little finger flexion (which are mechanically coupled on the DEKA LUKE Arm), (4) wrist flexion, and (5) wrist rotation. We used the results of the first phase to inform the training paradigm and smoothing filter use. As we were comparing only three algorithms in the second phase, we were able to perform four tasks (fragile egg, zipper, pouring, and folding) in a 2–3-h experimental session.
Phase one: training type and output smoothing comparison
Ten participants (24 ± 4 years [mean ± SD]; seven right-handed, three left-handed; nine male, one female) with no prior prosthesis experience mimicked the DEKA LUKE Arm movements in three training periods to provide training data for the control algorithms. The two-degree-of-freedom decode training consisted of opening and closing the hand as well as rotation of the wrist. Prior to each training period, study administrators coached the participants through one trial of each movement, to acquaint the participants with mimicking the movements. In the first training period, users completed 12 trials of individual movements: close hand, open hand, pronate, supinate. Each individual movement consisted of a 0.7-s rise time, 3-s hold, and 0.7-s return to rest position (for a total of 4.4 s per movement). The second training period repeated the first period; however, only four trials of each movement were completed. The first and second training periods took 5 and 1.5 min, respectively.
The third period introduced combination movements, where both degrees-of-freedom were simultaneously active for the middle portion of the trial (see Fig. 2b for a visual representation). Users completed four trials each of eight different movements: close hand and pronate, close hand and supinate, open hand and pronate, open hand and supinate, pronate and close hand, supinate and close hand, pronate and open hand, and supinate and open hand. For one combination movement, the first degree-of-freedom moved to its intended position with a 0.7-s rise time, 5.2-s hold, and 0.7-s return to resting position. The second degree-of-freedom moved to its intended position 0.4 s after the first degree-of-freedom reached its intended position with a 0.7-s rise time, 3.0-s hold, and 0.7-s return to its rest position. In total, one combination training trial consisted of 6.6 s. The third training period took 5 min.
To compare individual and combination training paradigms, the training sets were merged into two sets of equal duration before being used to train the decoder (so that any improved performance would not be due to the decoder training on more data). The first and second training periods were combined to produce an individual-movement-only set. The second and third training periods were combined to produce a set with individual and combination movements. After merging the datasets, the mKF, CNN, and MLP were trained on the kinematic and EMG data for continuous control.
Modified clothespin relocation task
In the modified clothespin relocation task (Fig. 1b), participants transferred three large clothespins from a horizontal bar to a vertical bar on the side of the dominant hand, referred to as the “inside-out” task . The task was chosen for several reasons: its simplicity allows the user to quickly learn and master the task, mitigating effects due to learning; its short duration allows several trials to be completed rapidly, which was necessary for comparing 12 decoders; and, it necessarily requires simultaneously active degrees-of-freedom, typical of many daily activities. We have used this task with the DEKA LUKE Arm previously .
Participants started a timer with the prosthetic hand, transferred a single clothespin, and stopped the timer with the prosthetic hand. A trial was considered failed when a clothespin was dropped. Participants were instructed that the success rate (proportion of successfully transferred clothespins) and the transfer time would be recorded. There was no time limit for the task. Participants completed nine trials (three sets of three) with each decoder. The order of the control paradigm was randomized among participants to prevent an order effect. After completing nine trials with a particular decoder, participants completed the NASA Task Load Index (TLX) subjective workload survey .
Participants completed the task for 12 total decoders, including each possible combination of training paradigm (individual and combination), algorithm (mKF, MLP, and CNN), and output type (raw and smoothed).
We compared the effect of training paradigm and smoothing filter for each algorithm with α = 0.05. The data were significantly non-normal for several conditions (Shapiro–Wilk test ), so nonparametric statistics were used. Median completion times for successful trials were calculated for each condition for each participant (N = 10 participants). The number of drops was calculated for each condition for each participant. For the training paradigm comparison, the data were aggregated to compare individual and combination trainings with Wilcoxon’s signed-rank test  for each algorithm; each group contained both smoothed and raw output types (N = 20; two medians for each participant). For the output type comparison, the data were aggregated to compare raw and smoothed outputs with Wilcoxon’s signed-rank test for each algorithm; each group contained both individual and combination trainings (N = 20; two medians for each participant). Because the hypotheses were pre-planned, no correction for multiple comparisons was applied .
We compared the extent that the algorithms predicted simultaneously active degrees-of-freedom. Because this was a post-hoc analysis, we did not have digital markers indicating when a clothespin trial was started. Instead, we examined the proportion of time an algorithm predicted hand movement and wrist rotation simultaneously, divided by the total time the hand or wrist was active. We determined the threshold for “active” movements by the range of movements needed to complete the clothespin task. The wrist needed to rotate a minimum of 35% of its movement window to place the clothespin, and the hand needed to close a minimum of 20% of its movement window to hold the clothespin. We counted only simultaneous movements over 1 s in duration to exclude potentially sporadic predictions that were likely not part of the task completion. We used the median proportion of time each algorithm predicted simultaneous movements for the algorithm comparisons. The medians across participants were distributed normally (Shapiro–Wilk test ), so we conducted a one-way ANOVA  to determine if there were significant differences among the algorithms. If the ANOVA revealed differences, individual comparisons were made using Tukey’s Honestly Significant Difference procedure .
Phase two: algorithm comparison
In the second phase, we extended the degrees-of-freedom from two to five to assess the algorithms’ performance in more complex, biologically realistic conditions. Users controlled the following degrees-of-freedom: flexion/extension of the thumb, flexion/extension of the index finger, flexion/extension of the middle, ring, and little fingers (which are mechanically coupled on the DEKA LUKE Arm), flexion/extension of the wrist, and pronation/supination of the wrist. Study administrators coached the participants through a single trial of each movement type before each training period. The training for phase two consisted of four distinct training periods, each 5 min long, for a total of 20 min.
Nine participants (30 ± 14 years [mean ± SD]; seven right-handed, two left-handed; seven male, two female) with some prior prosthesis experience (2–3 h) mimicked the prosthesis through movements of individual degrees-of-freedom. Six participants from phase one had completed the phase one study; however, not all the previous participants were able return for the phase two study. The remaining three had completed other prosthesis studies with our group. For the first training period, movements consisted of a 0.7-s rise time, 5.2-s hold time, 0.7-s fall time (for a total of 6.6 s per movement). Participants mimicked four trials on each degree-of-freedom. For the second through fourth training periods, combination movements were introduced.
The second period consisted of grasp movements (simultaneous flexion of the thumb, index, and middle, ring, and little degrees-of-freedom) combined with wrist movements (flexion, extension, pronation, and supination). Participants completed the first grasp movement alone, with a 0.7-s rise time, 5.2-s hold time, and 0.7-s fall time (6.6 s total). Participants then completed grasp and wrist combination movements, where both the grasp and wrist movement were simultaneously active in the middle portion of a trial (Fig. 2b). In these combination trials, the grasp movement rose to its intended position in 0.7 s, held for 5.2 s, and returned to its original position in 0.7 s. The wrist movement started 0.4 s after the first movement reached its intended position, moved to its intended position with a 0.7-s rise time, held for 3.0 s, and returned to its original position in 0.7 s. These combination movements were completed for each wrist movement (wrist flexion, extension, pronation, and supination). Then, they were completed in the reverse order, where the wrist movement occurred first, and the grasp movement occurred part-way through the wrist movement. We chose to complete the movements in both orders due to the CNN convolving over time. Participants completed four trials of each sequence, resulting in 36 total trials.
Periods three and four repeated the second period; however, the grasping movement was replaced with opening the hand (extending the thumb, index, and middle, ring, and little digits) and a pinch movement (flexing thumb and index digits), respectively. After merging the datasets from the four training periods, the mKF, CNN, and MLP were trained on the kinematic and EMG data for continuous control.
In the second phase, participants completed four tasks representative of activities of daily living in the following order: moving a fragile egg, using a zipper, pouring, and folding a towel (Fig. 3). A time limit was specified for each task. The order of the three decode algorithms (mKF, CNN, and MLP) was randomized for each participant to avoid order effects, with the same random order being used within each of the four tasks. After completing a task with each algorithm, participants ranked the algorithms in order of preference. Examples of the tasks are included in the supplementary material (Additional file 1: Video S1). Each task is now discussed in greater detail.
Fragile egg task
In the fragile egg task, participants moved a mechanical “fragile egg” [79, 80] horizontally 15 cm from one side of a 6.25-cm vertical barrier to the other within a 45-s time limit, beginning on the side of the dominant hand. This task evaluated the dexterity provided by each decode algorithm in that if too much force was applied, the mechanical egg would “break” and emit an audible click. The participants were instructed to use a two-finger pinch to pick up the mechanical egg. The breaking force for these experiments was set at about 20 N, and the device mass was about 615 g (0.03 N/g).
For each trial, participants started a timer with the prosthetic hand, transferred the mechanical egg, and stopped the timer with the prosthetic hand. A trial was considered failed if the mechanical egg was broken or dropped. Participants were instructed that both the success rate (the proportion of trials in which a transfer was successful) and the transfer time would be recorded. When introduced to the task, participants were given two minutes of practice with each decode algorithm to decrease the learning effects in the recorded trials. Participants attempted 15 trials with each decode algorithm. Immediately before the recorded trials commenced, participants were given one final practice trial with the given decode algorithm. A verbal warning alerted the participants when there were 15 s remaining. After attempting the 15 recorded trials with a particular algorithm, participants completed a NASA-TLX survey.
In the zipper task, participants closed a horizontally mounted zipper within a 30-s time limit, beginning on the side of the dominant hand. Timed-out trials were reported as 30 s. The zipper task tests precision grasping with combination wrist movements. For a trial to be successful, the zipper must have closed by at least 6.5 cm, as indicated with a mark. We modified the base of the zipper so that the handle would not lie less than 45 degrees from being flat because the shape of the prosthetic fingers prevented gripping the zipper when lying flat. For each trial, participants started a timer with the prosthetic hand, closed the zipper, and stopped the timer with the prosthetic hand. Participants were instructed that both the number of releases (times the zipper slipped from their grasp) as well as the time would be recorded. When introduced to the task, participants were given three practice trials with each decode algorithm in order to decrease the learning effects in the recorded trials. Participants attempted five trials with each decode algorithm. Immediately before the recorded trials commenced, participants were given one final practice trial with the given decode algorithm. After attempting the five recorded trials with a particular algorithm, participants completed a NASA-TLX survey.
In the pouring task, participants poured rice from an aluminum can into a cup within a 1-min time limit. The rice-filled can was placed on the side of the dominant hand, and the empty cup was placed near the midline. This task assessed the ability of the decode algorithms to maintain a grasp during wrist rotation. For these experiments, the aluminum can was filled with 200 g of rice.
For each trial, participants started a timer with the prosthetic hand, poured the rice from the can (7.6 cm diameter, 11.3 cm tall) to the cup (8.6 cm diameter, 11.7 cm tall), and stopped the timer with the prosthetic hand. Participants were instructed that the amount of rice (in grams) successfully transferred to the cup as well as the transfer time would be recorded. The participants were not allowed to use their native hand to stabilize the cup, and if the can was dropped the trial was considered unsuccessful. When introduced to the task, participants were given three practice trials with each decode algorithm in order to decrease the learning effects in the recorded trials. Participants attempted five trials with each decode algorithm. Immediately before the recorded trials commenced, participants were given one final practice trial with the given decode algorithm. After attempting the five recorded trials with a particular algorithm, participants completed a NASA-TLX survey.
In the folding task, participants folded a small hand towel (39 cm by 58 cm) twice using a pinch grasp within a 1-min time limit. The first fold was done from short-end to short-end. The participant could then use their non-dominant native hand to rotate the towel 90 degrees for the second fold, which was also from short-end to short-end. For both folds, participants were instructed not to use the edge of the table to facilitate grasping the towel. Additionally, they were not to use their native hand to guide the end of the towel into the prosthesis. Ideally, participants simultaneously grasped the short edge of the towel with their native hand and the prosthesis.
For each trial, participants started a timer with the prosthetic hand, folded the towel twice, and stopped the timer with the prosthetic hand. Participants were instructed that the number of releases (times the towel slipped from their grasp) and the time would be recorded. When introduced to the task, participants were given three practice trials with each decode algorithm to decrease the learning effects in the recorded trials. Participants attempted five trials with each decode algorithm. Immediately before the recorded trials commenced, participants were given one final practice trial with the given decode algorithm. After attempting the five recorded trials with a particular algorithm, participants completed a NASA-TLX survey.
We compared algorithms for each metric within each task with α = 0.05. The data were significantly non-normal for several conditions (Shapiro–Wilk test ), so nonparametric statistics were used. Median completion times for successful trials were calculated for each condition for each participant (N = 9 participants). For the pouring task, the task was considered successful if the total amount transferred was greater than 199.9 g (no spill). For each metric, a Kruskal–Wallis test  was used to determine if there were differences among the decode algorithms. If the Kruskal–Wallis revealed differences, paired comparisons were made using Wilcoxon’s signed-rank test . Because the hypotheses were pre-planned, no correction for multiple comparisons was applied . Post-hoc, we decided to compare aggregate user preferences. We grouped the user preferences for each task and performed a Kruskal–Wallis test across algorithms. As this was an unplanned comparison, we corrected the pairwise comparisons with the Dunn-Sidak correction for multiple comparisons .
Phase one: comparing latching filter and training type
Latching filter comparison
The latching filter generally reduced the number of dropped clothespins but sometimes increased the clothespin transfer time (Fig. 4a–c). For the MLP, dropped clothespins were significantly reduced from 3 to 1 (p < 0.01; Fig. 4a). For the CNN, dropped clothespins were significantly reduced; however, the median values were both 1 (p < 0.05; Fig. 4a). The latching filter significantly decreased transfer times from 11.0 to 7.4 s for the MLP (p < 0.01), but significantly increased transfer times for the CNN and mKF from 6.9 to 7.2 s (p < 0.05) and 7.8 to 10.1 s (p < 0.001), respectively (Fig. 4b). The latching filter significantly reduced subjective workload for the MLP from 54 to 41 (p < 0.05) but had no significant effect on subjective workload for the CNN and mKF (Fig. 4c).
Training type comparison
Training with combination movements generally improved the outcomes for the clothespin relocation task (Fig. 4d–f). Training with combination movements significantly reduced dropped clothespins for the MLP from 2.5 to 1 (p < 0.05; Fig. 4d). For the CNN, dropped clothespins were significantly reduced; however, the median values were both 1 (p < 0.05; Fig. 4d). Training with combinations significantly reduced the median time needed to transfer a clothespin for CNN and MLP from 8.3 to 6.7 s (p < 0.05) and 11.8 to 7.9 s (p < 0.001), respectively (Fig. 4e). Combination training significantly reduced the subjective workload for the CNN, from 44 to 32 (p < 0.01; Fig. 4f). Training type had no significant effect on the subjective workload for the mKF and MLP; however, both were reduced with combination training (Fig. 4f).
Simultaneous movement analysis
Simultaneous movements were significantly above zero for all three decoders (p < 0.001, all). The proportions of simultaneously active degrees-of-freedom were significantly different between algorithms (p < 0.01). The MLP had significantly more simultaneous movement than the CNN (16.6% ± 0.5% and 9.0% ± 0.6%, respectively [mean ± SEM]; p < 0.01). The mKF had significantly more simultaneous movement than the CNN (15.6% ± 0.5% and 9.0% ± 0.6%, respectively [mean ± SEM]; p < 0.05).
Phase two: comparing algorithms
Fragile egg task
Preferences for the fragile egg task were significantly different for the different decode algorithms (p < 0.01; Fig. 5e). The CNN was significantly preferred over the MLP (median ranks 1 and 3, respectively; p < 0.05). Similarly, the mKF strongly trended toward being preferred over the MLP (median ranks 2 and 3, respectively; p = 0.05). Subjective workload also significantly differed among algorithms (p < 0.05; Fig. 5b). The CNN had significantly lower workload than did the MLP (51, 74, respectively; p < 0.01). The mKF had significantly lower workload than did the MLP (51, 74, respectively; p < 0.05). There were no significant differences in success rate across the algorithms; however, the mKF and CNN were generally more successful than the MLP (Fig. 5c). There were no significant differences in completion time across algorithms, although the mKF generally performed faster than the CNN and MLP (Fig. 5a).
Preferences for the zipper task differed significantly among algorithms (p < 0.05; Fig. 5e). The CNN was significantly preferred over the MLP (median ranks 1 and 3, respectively; p < 0.05). There were no significant differences in completion time (Fig. 5a), but the median completion time for the mKF was the fastest. Inadvertent releases and subjective workload for the decodes did not significantly differ among algorithms (Fig. 5d, b, respectively).
The pouring task was generally perceived as less difficult than the fragile egg and zipper tasks (Fig. 5b). There were no statistical differences for any of the recorded metrics (time, success rate, preference, and subjective workload). The MLP was preferred by a few participants, in contrast to the fragile egg and zipper tasks where it was never preferred (Fig. 5e). Participants had high success rates for the pouring task across algorithms and rarely spilled (Fig. 5c); we report only on whether the task was completed without spills or drops. Completion times were similar among algorithms (Fig. 5a).
The folding task was generally perceived as less difficult than the fragile egg and zipper tasks (Fig. 5b) and did not result in any significant differences for the recorded metrics. More participants seem to prefer the CNN for this task (Fig. 5e) even though the trend suggests they inadvertently released the towel more than with the mKF or MLP. The CNN and MLP may also perform faster than the mKF (Fig. 5a).
Aggregate preferences across tasks
Aggregated across tasks, user preferences were significantly different (p < 0.05; Fig. 5e). Users preferred the CNN over the MLP (median ranks 1 and 3, respectively; p < 0.01), and the mKF trended toward being preferred over the MLP when aggregated across tasks (median ranks 2 and 3, respectively; p = 0.07).
Algorithm training and prediction times
On average, the mKF trained in 0.7 s, the MLP trained in 47 s, and the CNN trained in 135 s in phase one. The threshold optimization averaged 155 s in phase one. In phase two, the mKF trained in 2.7 s, the MLP trained in 193 s, and the CNN trained in 262 s. The threshold optimization took 480 s in phase two. For testing, the mKF took 0.6 ms, the MLP took 1.4 ms, and the CNN took 3.0 ms. The latching filter took 0.027 ms to modify the algorithm output. Detailed training and testing results are included in Table 1.
In this two-phase study, we demonstrate the importance of using diverse training sets that include combinatorial movements; the value of smoothing the algorithm output; and the efficacy of linear and nonlinear continuous decode algorithms for restoring functional ability to prosthesis users. To our knowledge, no study has attempted to compare real-world functional improvements of continuous prosthesis control using multiple algorithms, training paradigms, and output smoothing from the same participants.
In phase one, we found that training on combination movements improves functional performance and that a latching filter  can reduce jitter albeit with some cost in speed. In phase two, we found that a linearly-based decode algorithm (the mKF) can perform similarly to nonlinear deep-learning techniques (the CNN and MLP). From a clinical perspective, the mKF may be preferable sometimes due to reduced algorithm training time and computational resources required for real-time control. Phase two demonstrates how user-centric outcomes (user preferences and subjective workload) elucidated differences among algorithms when performance metrics alone could not.
Adding combination movements to the training data benefited the participants two-fold: they completed tasks faster and with greater control. The improved speed and control seem to preferentially benefit nonlinear algorithms (MLP, CNN) by producing more consistent control among participants, whereas the mKF was relatively consistent in both conditions. In terms of speed, the MLP benefited the most from the combination movement training; however, it was also the slowest in the single-movement training paradigm and had the most room for improvement. In terms of subjective workload, results from combination training suggest that nonlinear approaches benefited most from combination training; however, the difference was significant only for the CNN. Perhaps combination trainings introduce nonlinearities that preferentially benefit nonlinear neural networks.
The latching filter generally enhanced control at the cost of speed. We were surprised how little the latching filter benefited the mKF relative to the CNN and MLP. The latching filter uses a hyperparameter that adjusts the level of “latching” (and inherent speed) of kinematic changes. The latching filter uses the hyperparameter to preferentially dampen small changes in the decoder estimates (which are assumed to be inadvertent movements) while allowing gross movements to occur with little or no dampening . Our piloting and informal use of these algorithms in lab suggest nonlinear approaches generally predict kinematic positions similar to the data used for training and thus move to heavily trained positions quickly and unimpeded by the latching filter (e.g., rest, full flexion); in contrast, the Kalman filter predicts kinematics better dispersed throughout the kinematic range, which may be due to a relatively low Kalman gain  that places more value on past estimates than current measurements and decreases responsiveness. Thus, gross movements could be slowed by a low Kalman gain and the latching filter as they are initiated. We expect that the hyperparameter setting favored the nonlinear algorithms, as raw decoder output from the nonlinear algorithms likely moved between extreme positions (e.g., open hand to closed hand) faster and were not slowed as much by the latching filter when gross movement was intended. We expect that optimizing the hyperparameter could improve speed and control for the mKF especially, but also the CNN and MLP; however, such an optimization is beyond the scope of this study. The lack of improvement in dropped clothespins with the mKF might also have occurred because its control was different than the other two decoders, and so the participants did not become as accustomed to its responsiveness.
In phase two, we examined the decode algorithm’s performance using combination training and the latching filter in a high-degree-of-freedom setting (five; increased from two in phase one). With the reduced experimental condition count in this phase, we were able to thoroughly investigate the algorithms’ performance across a broader range of functional tasks. This phase also included a larger, more diverse training dataset, which we anticipated might preferentially benefit nonlinear algorithms.
We chose tasks representative of daily life that probe qualitatively different aspects of prosthesis control. The fragile egg task, generally judged most difficult by participants, requires precise movements and tests the decoder’s proportionality. The zipper task, also judged difficult by participants, requires consistent precision pinching with simultaneous wrist rotation. The fragile egg tasks and zipper tasks, which test precise movements, could be more difficult if the task were occluded by the bypass socket or prosthesis itself. However, a similar occlusion may occur in the real world. We did not have any participants comment on the difficulty of the tasks as being due to occlusion. The pouring task necessitates smooth wrist rotation during stable grasping. The folding task requires two-handed object manipulation and grasp stability with positional changes which often degrade control performance . We found the variety of the tasks to be useful, but the tasks were easier for the participants than pilot experiments suggested and, in some cases, resulted in floor and ceiling performance effects. The tasks would need to be more difficult to better elucidate performance differences among the algorithms in the future.
In part because of these floor and ceiling effects, user preferences provided strong evidence for identifying the “best” algorithm. Whereas many performance metrics varied only slightly, user preferences showed greater differences among algorithms. Aggregated across tasks, preferences indicated the CNN was preferred over the MLP, and the mKF also trended toward being preferred over the MLP. As testing multiple algorithms in the same experimental sessions allowed users to rank their preferred algorithms, we see incorporating user preferences as a strength of the study. Translating research practices to the clinic requires user-centric, data-driven approaches. Although performance differences may be small among algorithms, our findings show that differences in user experience exist across the participants, providing additional evidence toward clinical translation. Subjective workload also provided useful information on user experience, showing that the CNN and mKF were easier to use during the fragile egg task. From the user’s perspective, subjective workload may be the difference between adopting or abandoning a prosthesis; while two prosthetic systems may be capable of similar performance, the most intuitive, easy-to-use prosthesis will certainly be preferred.
We found it surprising that the mKF performed similarly to the nonlinear CNN whereas the MLP underperformed. Our informal use of the mKF suggests that the mKF performed well because it has a smoother, more predictable response to input changes, which can improve functional control . We anticipate the CNN performed well because it used all 528 of the EMG features (versus 48 from the channel selection algorithm for mKF and MLP) and convolves across time, thus training on more complex data and potentially learning from temporal characteristics which could be advantageous over the MLP. In , the MLP uses the first 16 principal components as input features, different from our use of 48 selected features, which could be another factor in its poorer performance. We informally observed that participants struggled to proportionally control the prosthesis for the MLP, and it would often close quickly and break the fragile egg, which may indicate a lack of generalization from the training data.
It is important to note that the purpose of our study was not to provide an exhaustive comparison of the mKF, CNN, and the MLP per se; rather, our goal was to determine whether combination training sets and the use of a latching filter would improve these previously published decoders, and which combinations to recommend for clinical translation. Although one algorithm may perform better under different circumstances (e.g., the number or type of features provided, training kinematics associated with mirror movements, or the amount of data in testing and validation sets) exploring all possible variations would have required even more participant time, which was already limited. Even with the reported experimental variations, most participants were starting to fatigue and were eager to remove the heavy prosthesis by the end of the experimental session—hence there was a practical limit to the number of decoders and variations in our comparison (e.g., only training type and latching filter using previously published algorithms).
The training data used are critical for the performance of any decoder. In our study, we used mimicry training, where the user follows the prosthesis through preprogrammed movements, and the algorithm fits the feature inputs to the kinematics of the prosthesis. Inherent in this training paradigm is the assumption that the user perfectly mimics the kinematics of the hand, which inevitably is an imperfect assumption. Imperfect kinematic and feature data could have been preferentially detrimental to certain decoders (e.g., the MLP may perform better with cleaner labels). Another method for creating kinematics is to use mirror training, where the kinematics of the opposite hand are recorded while the user completes bilaterally mirrored movements. This approach has been shown to provide reliable training data for prosthesis control [83, 84] and could be further studied in real-world environments. Even though we did not use mirror movements, we aligned our kinematics and features during channel selection to improve performance. Additionally, some of the training data used single finger motions not exclusively in use for the tasks. It is possible that the training paradigms could have been improved for the specific tasks completed in this study; however, in real-world settings the tasks are not known a priori and therefore a generalizable training dataset is preferable. Recent studies have reported on some of the most used grasp types [85, 86]. Based on the findings of these studies, a specific, yet generalizable, training set could be developed for real-world usage. Another method of improving features could include optimizing electrode placement. Although beyond the scope of this study, one could use the channel selection algorithm  to find geometric or anatomical patterns in the location of the most useful electrodes. These patterns could guide electrode placement for advanced prostheses used by persons with transradial amputation; however, the usefulness of the patterns would vary depending on the level of the amputation.
Our study did not directly measure or restrict compensatory movements. Future work would benefit from quantifying or restricting compensatory movements. For some tasks (e.g., pouring task), compensatory movements could make stable grip control more important to the user than the ability to simultaneously activate multiple degrees-of-freedom. One benefit of the bypass socket  used in this study is that the rotating wrist attachment generally keeps the prosthesis hanging below the user’s wrist, regardless of shoulder, elbow, or wrist orientation, thus reducing the ability of the user to make compensatory movements and hence indirectly requiring simultaneous movements. The clothespin inside-out task conducted in phase one is particularly difficult to complete with compensatory movements. As a post-hoc analysis of the clothespin task, we found that the mKF and MLP had more simultaneous movements than the CNN, suggesting that they may have required less compensation. Although our study did not compare the approaches above with conventional direct control, one previous study  found that arm movements with an advanced decoder  were more similar to the endogenous hand when compared with conventional direct control in the clothespin relocation task. This finding suggests more advanced decoders can provide the user with more natural movements and reduce compensation.
Reliable, simultaneous movements in position mode could provide the user with a prosthesis experience like that of the endogenous hand, in which simultaneous movements are natural. In velocity mode, a user may choose to complete multi-degree-of-freedom tasks sequentially, moving a single degree-of-freedom at a time. Sequential movements are far less natural and much slower than simultaneous movements if the simultaneous movements can be completed reliably. Our results suggest simultaneous movements were used for task completion; however, simultaneous movements alone do not mean the movements were intended. The mKF and the MLP had significantly more simultaneous movements than the CNN; however, reported user experiences and our informal observation suggest that the MLP may not have made the simultaneous movements reliably. The CNN, with less simultaneous movements, may have provided reliable grip control which benefited the task and was reflected in user experiences, although successful task performance would have required greater compensatory movements from the user.
Although common performance metrics were used, direct comparisons with other studies are difficult due to different experimental conditions (e.g., degrees-of-freedom, physical prosthesis) and methodologies. In , where a two-degree-of-freedom, continuous, linear regression-based control strategy was compared with a conventional control strategy, the participant required about 10 s per clothespin for the horizontal to vertical transfer with the regression-based approach, whereas the conventional strategy required about 14 s. In , the two-degree-of-freedom “proportional simultaneous” control strategy (most similar to what we have termed continuous control) required about 5 s per clothespin transfer (participants completed both the horizontal to vertical and vertical to horizontal transfers), whereas the conventional strategy required about 10 s per clothespin. Neither study [50, 62] controlled for compensatory movements, which may lead to faster completion times, as compared with our use of the bypass socket  which reduces compensatory movements. Our median transfer times per clothespin ranged from 7 s to 11 s to depending on the decoder, using two-degrees-of-freedom. Classifier [20, 66, 67] and hybrid  approaches on the clothespin relocation task have ranged from about 5 s to 30 s per clothespin. In , three participants averaged 24 s per clothespin prior to a take-home trial with their prostheses, most like our study with naïve prosthesis users; after the trial, the participants averaged 8 s. In , a relatively large cohort (n = 17) of persons with below-elbow amputation completed the clothespin relocation task using conventional direct control. Users took about 6 s per clothespin transfer. Because compensation was not measured or restricted in , users may have been able to complete the task quickly with task-effective body compensation.
The fragile object task has been performed with various prostheses and fragile objects. Generally, the ratio between the break force and mass determines the task difficulty; a lower ratio implies greater difficulty. In the present study, non-amputee participants achieved a 63% median success rate. The break force in the present experiments was about 20 N, and the device mass was 615 g, resulting in a 0.03 N/g ratio. In a previous study from our group, an individual with an amputation achieved about 55% success with the mKF on a similar fragile egg task without sensory feedback . In [89, 90], the success rates without sensory feedback were 38% and 45% with a fragile object that broke at 1.23 N and had a mass of 80 g, a ratio of 0.15 N/g, approximately five times the ratio of our study. In , the success rate was about 86% without sensory feedback, also using a different fragile object, which broke at 10.7 N and weighed 8 g, a 1.34 N/g ratio (approximately 45 times the ratio of our study). Transfer times, where reported, were similar among the aforementioned studies and the present study, ranging from 10 s to 13 s per transfer.
One goal of this study was to determine the best decoders and training paradigms for a portable take-home system. This study provides strong evidence from real-world functional tasks for using diverse training sets as input for a decoder and modifying the decode output with an LF. Selecting between linear and nonlinear decoders is less clear. Our findings indicate comparable performance between an mKF and a CNN, where user preferences were slightly increased for the CNN. Clinical deployment, however, may favor the mKF, which trains faster and is less computationally expensive. Furthermore, in a take-home clinical trial where more training data may be collected over time, aggregation of these datasets may begin to favor deep learning approaches  like the CNN or MLP as technologies continue to improve. Overall, our results provide valuable information toward the clinical implementation of advanced control strategies for prosthesis users.
This work compared myoelectric prosthetic control using activities of daily living with 12 continuous decoders in a two-phase study. In the first phase, comparing training paradigms and decoder smoothing, we found that training control algorithms with more complex movement patterns (i.e., simultaneously active digits and/or wrist) improves control accuracy and speed. Our results showed that nonlinear smoothing of the decoder output can improve accuracy although sometimes at a cost in speed. In the second phase, objective performance among the mKF, MLP, and CNN was similar for many tasks, but subjective preferences favored the CNN and mKF. Users’ subjective experiences are a key additional consideration.
This study was completed to select a decoder for a long-term take-home trial with implanted neuromyoelectric devices. The results demonstrate the importance of using rich training paradigms and nonlinear algorithm smoothing to improve continuous prosthetic control. The results also highlight the importance of user-focused subjective metrics in comparing decoders. Where performance differences could not effectively select a “best” decoder, user preferences added strong evidence toward selecting the mKF or CNN. Clinical considerations may favor the mKF as it is faster to train and computationally less expensive than the CNN. Subjective experiences may be even more important in long-term real-world environments and may influence voluntary prosthesis use. Overall, the results herein demonstrate the efficacy of continuous decoders for enabling intuitive prosthetic control of modern bionic prostheses.
Availability of data and materials
The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.
Modified Kalman filter
Convolutional neural network
Task load index
Mobius Bionics LLC. LUKE Arm System.. 2019. https://www.mobiusbionics.com/wp-content/uploads/2019/09/Mobius-Bionics-LUKE-Product-Spec-Sheet.pdf.
Taska Prosthetics. TASKA Hand. 2020. https://www.taskaprosthetics.com/the-taska. Accessed 8 Jul 2020.
Otto Bock HealthCare LP. Bebionic hand User Guide. 2018. https://www.ottobock.ca/media/local-media/prosthetics/upper-limb/files/15667-bebionic-user-guide.pdf.
Otto Bock HealthCare LP. Michelangelo Hand Brochure. 2017. http://www.ottobockus.com/media/local-media/prosthetics/upper-limb/michelangelo/files/michelangelo-brochure.pdf.
Motion Control Division of Fillauer. Utah Arm U3 and U3+ User Guide. 2020. https://fillauer.com/wp-content/uploads/2020/09/1910025-USER-GUIDE-Utah-Arm-U3-and-U3-Plus-Rev-C-08-24-2020.pdf.
Open Bionics LLC. Hero arm—user guide. 2020. https://openbionics.com/hero-arm-user-guide/. Accessed 8 Jul 2020.
Össur Touch Solutions. i-Limb Quantum Titanium. 2020. https://media.ossur.com/image/upload/v1573570123/product-documents/en-us/PN20247/catalogs/PN20247_i-Limb_Quantum_Titanium.pdf.
Össur Touch Solutions. i-Limb Access. 2020. https://media.ossur.com/image/upload/v1573570122/product-documents/en-us/PN20239/catalogs/PN20239_i-Limb_Access.pdf.
Engdahl SM, Christie BP, Kelly B, Davis A, Chestek CA, Gates DH. Surveying the interest of individuals with upper limb loss in novel prosthetic control techniques. J Neuroeng Rehabil. 2015;12(1):1–11. https://doi.org/10.1186/s12984-015-0044-2.
Biddiss E, Chau T. Upper limb prosthesis use and abandonment: a survey of the last 25 years. Prosthet Orthot Int. 2007;31(3):236–57. https://doi.org/10.1080/03093640600994581.
Coapt LLC. Complete Control System Gen2 Handbook. 2020.
Otto Bock HealthCare LP. Myo Plus pattern recognition. 2020.
Asghari Oskoei M, Hu H. Myoelectric control systems—a survey. Biomed Signal Process Control. 2007;2(4):275–94. https://doi.org/10.1016/j.bspc.2007.07.009.
Touchbionics. I-limb quantum. 2018. http://www.touchbionics.com/products/active-prostheses/i-limb-quantum. Accessed 7 Mar 2019.
D’Anna E, et al. A closed-loop hand prosthesis with simultaneous intraneural tactile and position feedback. Sci Robot. 2019. https://doi.org/10.1126/scirobotics.aau8892.
Kuiken TA, et al. Targeted muscle reinnervation for real-time myoelectric control of multifunction artificial arms. JAMA. 2009;301(6):619–28. https://doi.org/10.1001/jama.2009.116.
Waris A, Niazi IK, Jamil M, Englehart K, Jensen W, Kamavuako EN. Multiday evaluation of techniques for EMG-based classification of hand motions. IEEE J Biomed Heal Inform. 2019;23(4):1526–34. https://doi.org/10.1109/JBHI.2018.2864335.
Young AJ, Smith LH, Rouse EJ, Hargrove LJ. Classification of simultaneous movements using surface EMG pattern recognition. IEEE Trans Biomed Eng. 2013;60(5):1250–8. https://doi.org/10.1109/TBME.2012.2232293.
Hargrove LJ, Li G, Englehart KB, Hudgins BS. Principal components analysis preprocessing for improved classification accuracies in pattern-recognition-based myoelectric control. IEEE Trans Biomed Eng. 2009;56(5):1407–14. https://doi.org/10.1109/TBME.2008.2008171.
Hargrove LJ, Miller LA, Turner K, Kuiken TA. Myoelectric pattern recognition outperforms direct control for transhumeral amputees with targeted muscle reinnervation: a randomized clinical trial. Sci Rep. 2017;7(1):1–9. https://doi.org/10.1038/s41598-017-14386-w.
D’Anna E, et al. A somatotopic bidirectional hand prosthesis with transcutaneous electrical nerve stimulation based sensory feedback. Sci Rep. 2017;7(1):1–15. https://doi.org/10.1038/s41598-017-11306-w.
Ortiz-Catalan M, Håkansson B, Brånemark R. An osseointegrated human-machine gateway for long term sensory feedback and control of artificial limbs. Sci Transl Med. 2014a;6(257):1–9.
Scheme E, Englehart K. Training strategies for mitigating the effect of proportional control on classification in pattern recognition-based myoelectric control. J Prosthe Orthot. 2013;25(2):76–83. https://doi.org/10.1097/JPO.0b013e318289950b.
Jiang N, Englehart KB, Parker PA. Extracting simultaneous and proportional neural control information for multiple-dof prostheses from the surface electromyographic signal. IEEE Trans Biomed Eng. 2009;56(4):1070–80. https://doi.org/10.1109/TBME.2008.2007967.
Wu W, Gao Y, Bienenstock E, Donoghue JP, Black MJ. Bayesian population decoding of motor cortical activity using a Kalman filter. Neural Comput. 2006;18(1):80–118. https://doi.org/10.1162/089976606774841585.
Vu PP, et al. A regenerative peripheral nerve interface allows real-time control of an artificial hand in upper limb amputees. Sci Transl Med. 2020;12(533):1–12. https://doi.org/10.1126/scitranslmed.aay2857.
George JA, et al. Biomimetic sensory feedback through peripheral nerve stimulation improves dexterous use of a bionic hand. Sci Robot. 2019;4(32):eaax2352. https://doi.org/10.1126/scirobotics.aax2352.
Ameri A, Kamavuako EN, Scheme EJ, Englehart KB, Parker PA. Support vector regression for improved real-time, simultaneous myoelectric control. IEEE Trans Neural Syst Rehabil Eng. 2014a;22(6):1198–209. https://doi.org/10.1109/TNSRE.2014.2323576.
Dantas H, Warren DJ, Wendelken SM, Davis TS, Clark GA, Mathews VJ. Deep learning movement intent decoders trained with dataset aggregation for prosthetic limb control. IEEE Trans Biomed Eng. 2019;66(11):3192–203. https://doi.org/10.1109/TBME.2019.2901882.
George JA, Brinton MR, Duncan CC, Hutchinson DT, Clark GA. Improved training paradigms and motor-decode algorithms: results from intact individuals and a recent transradial amputee with prior complex regional pain syndrome. In: 40th International Engineering in Medicine and Biology Conference. 2018. Vol 2018. doi: https://doi.org/10.1109/EMBC.2018.8513342.
Ameri A, Akhaee MA, Scheme E, Englehart K. Regression convolutional neural network for improved simultaneous EMG control. J Neural Eng. 2019. https://doi.org/10.1088/1741-2552/ab0e2e.
Adewuyi AA, Hargrove LJ, Kuiken TA. Resolving the effect of wrist position on myoelectric pattern recognition control. J Neuroeng Rehabil. 2017. https://doi.org/10.1186/s12984-017-0246-x.
Hansen TC, Dantas H, George JA, Clark GA, Warren DJ, Mathews VJ. Shared controllers improve control and performance of upper-limb prostheses. 2019. doi: https://doi.org/10.13140/RG.2.2.36537.72805.
Amsuess S, et al. Context-dependent upper limb prosthesis control for natural and robust use. IEEE Trans Neural Syst Rehabil Eng. 2016;24(7):744–53. https://doi.org/10.1109/TNSRE.2015.2454240.
Gilja V, et al. A high-performance neural prosthesis enabled by control algorithm design. Nat Neurosci. 2012;15(12):1752–7. https://doi.org/10.1038/nn.3265.
George JA, Davis TS, Brinton MR, Clark GA. Intuitive neuromyoelectric control of a dexterous bionic arm using a modified Kalman filter. J Neurosci Methods. 2020;330(April 2019):108462. https://doi.org/10.1016/j.jneumeth.2019.108462.
Ortiz-Catalan M, Håkansson B, Brånemark R. Real-time and simultaneous control of artificial limbs based on pattern recognition algorithms. IEEE Trans Neural Syst Rehabil Eng. 2014b;22(4):756–64. https://doi.org/10.1109/TNSRE.2014.2305097.
Hargrove LJ, Englehart K, Hudgins B. A comparison of surface and intramuscular myoelectric signal classification. IEEE Trans Biomed Eng. 2007;54(5):847–53. https://doi.org/10.1109/TBME.2006.889192.
Ameri A, Kamavuako EN, Scheme EJ, Englehart KB, Parker PA. Real-time, simultaneous myoelectric control using visual target-based training paradigm. Biomed Signal Process Control. 2014b;13(1):8–14. https://doi.org/10.1016/j.bspc.2014.03.006.
Kamavuako EN, Englehart KB, Jensen W, Farina D. Simultaneous and proportional force estimation in multiple degrees of freedom from intramuscular EMG. IEEE Trans Biomed Eng. 2012;59(7):1804–7. https://doi.org/10.1109/TBME.2012.2197210.
Hahne JM, et al. Linear and nonlinear regression techniques for simultaneous and proportional myoelectric control. IEEE Trans Neural Syst Rehabil Eng. 2014;22(2):269–79. https://doi.org/10.1109/TNSRE.2014.2305520.
Ameri A, Akhaee MA, Scheme E, Englehart K. Real-time, simultaneous myoelectric control using a convolutional neural network. PLoS ONE. 2018;13(9):1–13. https://doi.org/10.1371/journal.pone.0203835.
Zia ur Rehman M, et al. Multiday EMG-Based classification of hand motions with deep learning techniques. Sensors (Switzerland). 2018;18(8):1–16. https://doi.org/10.3390/s18082497.
Fougner A, Scheme E, Chan ADC, Englehart K, Stavdahl Ø. Resolving the limb position effect in myoelectric pattern recognition. IEEE Trans Neural Syst Rehabil Eng. 2011;19(6):644–51. https://doi.org/10.1109/TNSRE.2011.2163529.
Teh Y, Hargrove LJ. Understanding limb position and external load effects on real-time pattern recognition control in amputees. IEEE Trans Neural Syst Rehabil Eng. 2020;28(7):1–1. https://doi.org/10.1109/tnsre.2020.2991643.
Radmand A, Scheme E, Englehart K. On the suitability of integrating accelerometry data with electromyography signals for resolving the effect of changes in limb position during dynamic limb movement. J Prosthe Orthot. 2014;26(4):185–93. https://doi.org/10.1097/JPO.0000000000000041.
Young D, et al. Closed-loop cortical control of virtual reach and posture using Cartesian and joint velocity commands. J Neural Eng. 2019. https://doi.org/10.1088/1741-2552/aaf606.
Downey JE, et al. Blending of brain-machine interface and vision-guided autonomous robotics improves neuroprosthetic arm performance during grasping. J Neuroeng Rehabil. 2016;13(1):1–12. https://doi.org/10.1186/s12984-016-0134-9.
Nieveen J, Brinton MR, Warren DJ, Mathews VJ. A nonlinear latching filter to remove jitter from movement estimates for prostheses. IEEE Trans Neural Syst Rehabil Eng. 2020. https://doi.org/10.1109/TNSRE.2020.3038706.
Fougner AL, Stavdahl Ø, Kyberd PJ. System training and assessment in simultaneous proportional myoelectric prosthesis control. J Neuroeng Rehabil. 2014;11(1):1–13. https://doi.org/10.1186/1743-0003-11-75.
Ortiz-Catalan M, Rouhani F, Branemark R, Hakansson B. Offline accuracy: a potentially misleading metric in myoelectric pattern recognition for prosthetic control. Proc Annu Int Conf IEEE Eng Med Biol Soc EMBS. 2015;2015:1140–3. https://doi.org/10.1109/EMBC.2015.7318567.
Scheme E, Englehart K. On the robustness of EMG features for pattern recognition based myoelectric control. A multi-dataset comparison. 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society EMBC 2014, pp. 650–653, 2014. doi: https://doi.org/10.1109/EMBC.2014.6943675.
Cracchiolo M, et al. Decoding of grasping tasks from intraneural recordings in trans-radial amputee. J Neural Eng. 2020;17(2):026034. https://doi.org/10.1088/1741-2552/ab8277.
Jiang N, Vujaklija I, Rehbaum H, Graimann B, Farina D. Is accurate mapping of EMG signals on kinematics needed for precise online myoelectric control? IEEE Trans Neural Syst Rehabil Eng. 2014;22(3):549–58. https://doi.org/10.1109/TNSRE.2013.2287383.
Vu PP, et al. Closed-loop continuous hand control via chronic recording of regenerative peripheral nerve interfaces. IEEE Trans Neural Syst Rehabil Eng. 2018;26(2):515–26. https://doi.org/10.1109/TNSRE.2017.2772961.
Jiang N, Lorrain T, Farina D. A state-based, proportional myoelectric control method: online validation and comparison with the clinical state-of-the-art. J Neuroeng Rehabil. 2014;11(1):110. https://doi.org/10.1186/1743-0003-11-110.
Birdwell JA, Hargrove LJ, Weir RFF, Kuiken TA. Extrinsic finger and thumb muscles command a virtual hand to allow individual finger and grasp control. IEEE Trans Biomed Eng. 2014;62(1):218–26. https://doi.org/10.1016/j.physbeh.2017.03.040.
Smith LH, Kuiken TA, Hargrove LJ. Real-time simultaneous and proportional myoelectric control using intramuscular EMG. J Neural Eng. 2014. https://doi.org/10.1088/1741-2560/11/6/066013.
Vujaklija I, et al. Translating research on myoelectric control into clinics-are the performance assessment methods adequate? Front Neurorobot. 2017. https://doi.org/10.3389/fnbot.2017.00007.
Betthauser JL, et al. Limb position tolerant pattern recognition for myoelectric prosthesis control with adaptive sparse representations from extreme learning. IEEE Trans Biomed Eng. 2018;65(4):770–8. https://doi.org/10.1109/TBME.2017.2719400.
Hargrove L, Miller L, Turner K, Kuiken T. Control within a virtual environment is correlated to functional outcomes when using a physical prosthesis. J Neuroeng Rehabil. 2018. https://doi.org/10.1186/s12984-018-0402-y.
Hahne JM, Wilke MA, Koppe M, Farina D, Schilling AF. Longitudinal case study of regression-based hand prosthesis control in daily life. Front Neurosci. 2020;14(June):1–8. https://doi.org/10.3389/fnins.2020.00600.
Hahne JM, Schweisfurth MA, Koppe M, Farina D. Simultaneous control of multiple functions of bionic hand prostheses: performance and robustness in end users. Sci Robot. 2018;3(19):3630. https://doi.org/10.1126/scirobotics.aat3630.
Amsuess S, Goebel P, Graimann B, Farina D. A multi-class proportional myocontrol algorithm for upper limb prosthesis control: validation in real-life scenarios on amputees. IEEE Trans Neural Syst Rehabil Eng. 2015;23(5):827–36. https://doi.org/10.1109/TNSRE.2014.2361478.
Kuiken TA, et al. Targeted reinnervation for enhanced prosthetic arm function in a woman with a proximal amputation: a case study. Lancet. 2007;369(9559):371–80. https://doi.org/10.1016/S0140-6736(07)60193-7.
Kuiken TA, Miller LA, Turner K, Hargrove LJ. A comparison of pattern recognition control and direct control of a multiple degree-of-freedom transradial prosthesis. IEEE J Transl Eng Heal Med. 2016. https://doi.org/10.1109/JTEHM.2016.2616123.
Patel GK, Castellini C, Hahne JM, Farina D, Dosen S. A classification method for myoelectric control of hand prostheses inspired by muscle coordination. IEEE Trans Neural Syst Rehabil Eng. 2018;26(9):1745–55. https://doi.org/10.1109/TNSRE.2018.2861774.
George JA, Neibling A, Paskett MD, Clark GA. Inexpensive surface electromyography sleeve with consistent electrode placement enables dexterous and stable prosthetic control through deep learning. 41st International Engineering Medicine and Biology Conference, vol. 2020, 2020. http://arxiv.org/abs/2003.00070.
Paskett MD, et al. A modular transradial bypass socket for surface myoelectric prosthetic control in non-amputees. IEEE Trans Neural Syst Rehabil Eng. 2019;27(10):2070–6. https://doi.org/10.1109/TNSRE.2019.2941109.
Nieveen J, Warren D, Wendelken S, Davis T, Kluger D, Page D. Channel selection of neural and electromyographic signals for decoding of motor intent. In Myoelectric Controls Conference. 2017, p. 720.
Hussaini A, Kyberd P. Refined clothespin relocation test and assessment of motion. Prosthet Orthot Int. 2017;41(3):294–302. https://doi.org/10.1177/0309364616660250.
Olsen NR, et al. An adaptable prosthetic wrist reduces subjective workload. bioRxiv. 2019. https://doi.org/10.1101/808634.
Hart SG, Staveland LE. Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Adv Psychol. 1988. https://doi.org/10.1016/S0166-4115(08)62386-9.
Peat J, Barton B. Medical statistics: a guide to data analysis and critical appraisal. In: Medical statistics: a guide to data analysis and critical appraisal 1st ed. John Wiley & Sons, Inc. 2008. https://doi.org/10.1002/9780470755945.
Gibbons JD, Chakraborti S. Nonparametric statistical inference. In: Lovric M, editor. International encyclopedia of statistical science. 1st ed. Berlin, Heidelberg: Springer-Verlag; 2011. https://doi.org/10.1007/978-3-642-04898-2_420.
Armstrong RA. When to use the Bonferroni correction. Ophthalmic Physiol Opt. 2014;34(5):502–8. https://doi.org/10.1111/opo.12131.
Hogg RV, Ledolter J. Engineering statistics. New York: MacMillan; 1987.
Hochberg Y, Tamhane AC. Multiple comparison procedures. 1st ed. New Jersey: Wiley; 1987.
Meek SG, Jacobsen SC, Goulding PP. Extended physiologic taction: design and evaluation of a proportional force feedback system. J Rehabil Res Dev. 1989;26(3):53–62.
Engeberg ED, Meek S. Improved grasp force sensitivity for prosthetic hands through force-derivative feedback. IEEE Trans Biomed Eng. 2008;55(2):817–21. https://doi.org/10.1109/TBME.2007.912675.
Kruskal WH, Wallis WA. Use of ranks in one-criterion variance analysis. J Am Stat Assoc. 1952. https://doi.org/10.1080/01621459.1952.10483441.
Šidák Z. Rectangular confidence regions for the means of multivariate normal distributions. J Am Stat Assoc. 1967. https://doi.org/10.1080/01621459.1967.10482935.
Muceli S, Farina D. Simultaneous and proportional estimation of hand kinematics from EMG during mirrored movements at multiple degrees-of-freedom. IEEE Trans Neural Syst Rehabil Eng. 2012;20(3):371–8. https://doi.org/10.1109/TNSRE.2011.2178039.
George JA, Tully TN, Colgan P, Clark G. Bilaterally mirrored movements improve the accuracy and precision of training data for supervised learning of neural or myoelectric prosthetic control. 2020. https://doi.org/10.13140/RG.2.2.34651.59689.
Cini F, Ortenzi V, Corke P, Controzzi M. On the choice of grasp type and location when handing over an object. Sci Robot. 2019;4(27):1–14. https://doi.org/10.1126/scirobotics.aau9757.
Sundaram S, Kellnhofer P, Li Y, Zhu J-Y, Torralba A, Matusik W. Learning the signatures of the human grasp using a scalable tactile glove. Nature. 2019;569(7758):698–702. https://doi.org/10.1038/s41586-019-1234-z.
Aszmann OC, et al. Elective amputation and bionic substitution restore functional hand use after critical soft tissue injuries. Sci Rep. 2016;6(September):1–9. https://doi.org/10.1038/srep34960.
Salminger S, et al. Functional outcome scores with standard myoelectric prostheses in below-elbow amputees. Am J Phys Med Rehabil. 2019;98(2):125–9. https://doi.org/10.1097/PHM.0000000000001031.
Valle G, et al. Biomimetic intraneural sensory feedback enhances sensation naturalness, tactile sensitivity, and manual dexterity in a bidirectional prosthesis. Neuron. 2018;100(1):37-45.e7. https://doi.org/10.1016/j.neuron.2018.08.033.
Petrini FM, et al. Six-month assessment of a hand prosthesis with intraneural tactile feedback. Ann Neurol. 2019;85(1):137–54. https://doi.org/10.1002/ana.25384.
Clemente F, D’Alonzo M, Controzzi M, Edin BB, Cipriani C. Non-invasive, temporally discrete feedback of object contact and release improves grasp control of closed-loop myoelectric transradial prostheses. IEEE Trans Neural Syst Rehabil Eng. 2016;24(12):1314–22. https://doi.org/10.1109/TNSRE.2015.2500586.
We thank the participants in this study for their willingness to invest their time to improve advanced prostheses to benefit amputee populations.
This work was sponsored by the Hand Proprioception and Touch Interfaces (HAPTIX) program administered by the Biological Technologies Office (BTO) of the Defense Advanced Research Projects Agency (DARPA) through the Space and Naval Warfare Systems Center, Contract No. N66001-15-C-4017. Research reported in this publication was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health (NIH) under Award Number UL1TR002538 and TL1TR002540. The content is solely the authors’ responsibility and does not necessarily represent the official views of their employers, DARPA, or the NIH.
Ethics approval and consent to participate
All participants took part in the experiments after giving their informed consent, as approved by the University of Utah IRB (ID 00098851).
Consent for publication
The author(s) declare(s) that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Additional file 1: Video S1. Example videos of phase two tasks for each algorithm.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Paskett, M.D., Brinton, M.R., Hansen, T.C. et al. Activities of daily living with bionic arm improved by combination training and latching filter in prosthesis control comparison. J NeuroEngineering Rehabil 18, 45 (2021). https://doi.org/10.1186/s12984-021-00839-x
- Prosthetic control
- Activities of daily living
- Neural prostheses
- Bionic arm
- DEKA LUKE arm
- Electromyography (EMG)