- Open Access
Robot-aided assessment of lower extremity functions: a review
Journal of NeuroEngineering and Rehabilitation volume 13, Article number: 72 (2016)
The assessment of sensorimotor functions is extremely important to understand the health status of a patient and its change over time. Assessments are necessary to plan and adjust the therapy in order to maximize the chances of individual recovery. Nowadays, however, assessments are seldom used in clinical practice due to administrative constraints or to inadequate validity, reliability and responsiveness. In clinical trials, more sensitive and reliable measurement scales could unmask changes in physiological variables that would not be visible with existing clinical scores.
In the last decades robotic devices have become available for neurorehabilitation training in clinical centers. Besides training, robotic devices can overcome some of the limitations in traditional clinical assessments by providing more objective, sensitive, reliable and time-efficient measurements. However, it is necessary to understand the clinical needs to be able to develop novel robot-aided assessment methods that can be integrated in clinical practice.
This paper aims at providing researchers and developers in the field of robotic neurorehabilitation with a comprehensive review of assessment methods for the lower extremities. Among the ICF domains, we included those related to lower extremities sensorimotor functions and walking; for each chapter we present and discuss existing assessments used in routine clinical practice and contrast those to state-of-the-art instrumented and robot-aided technologies. Based on the shortcomings of current assessments, on the identified clinical needs and on the opportunities offered by robotic devices, we propose future directions for research in rehabilitation robotics. The review and recommendations provided in this paper aim to guide the design of the next generation of robot-aided functional assessments, their validation and their translation to clinical practice.
Standardized sensorimotor assessments after neurological disorders have the potential to aid the understanding of recovery and to support the design of effective therapeutic interventions, with the ultimate goal of maximizing the patient’s chances of rehabilitation. Despite the general consensus on this statement among clinicians, neuroscientists and rehabilitation engineers, sensorimotor assessments are not routinely performed in the clinical practice [1, 2]. Duncan et al. identified four high-level determinants that impact routine assessments in practice: i) Knowledge, Education, and Perceived Value in Outcome Measurement (i.e. information on validity and reliability); ii) Support/Priority for Outcome Measure Use (i.e. organizational and management factors); iii) Practical Considerations (e.g. time, cost); iv) Patient Considerations (e.g. usefulness of the assessment to the patient’s treatment). The limited use of assessments in clinical practice reduces the chances to obtain feedback on the therapeutic intervention and consequently decreases the efficiency of therapy planning and adjustment [1, 3, 4]. Objective proofs are needed to justify healthcare expenses and reimbursement from insurances [1, 3]. In research, the lack of sensitive and reliable outcome measures can hamper the results of clinical trials aimed at determining the efficacy of new treatments, if changes due to the intervention under study fail to be detected . Thus, valid, reliable and sensitive assessments are useful in areas that encompass therapeutic, research and financial domains (Fig. 1).
The last decades have seen an increasing use of robotic devices for neurorehabilitation training in clinical centers [6–8]. Besides training, translational researchers in neurorehabilitation have proposed the use of robotic devices to overcome some of the limitations in traditional clinical assessments. Robotic devices represent an alternative method to provide more objective, sensitive, reliable and time-efficient assessments in clinical practice [6, 9, 10]. Sensors are embedded or can be easily added in robotic devices in order to provide quantitative measures of variables such as, for example, joint angles. Instrumented devices enable the recording of new variables (e.g., smoothness) that were not accessible before. Standardized assessment protocols and repeatable conditions can be achieved with the use of robotic devices. Patient’s motivation, which is a factor that can influence the assessment outcomes , can be promoted by using virtual reality applications to provide constant engagement, along with standardized instructions. Moreover, assessments can be integrated into the training session without requiring additional setup and measurement time. Training variables (e.g., duration, number of repetitions) can also be used to provide an indication of the patient’s performance and allow comparison between sessions.
However, the frequent criticism from clinicians towards these engineering solutions is that the outcome measures provided by robotic devices are too abstract, do not translate to function and lack ecological validity. Moreover, robotic devices often require a long setup time and a certain degree of technical knowledge to be operated . In a typical setting, the therapist has between 30 min and 1 h to deliver the therapeutic intervention. If the assessment protocol takes too much time to be performed, the solution may not be adopted. In some cases, the increase in sensitivity and reliability is discarded in favor of an existing subjective, yet time-efficient, assessment that can be applied in any clinical setting. These may be some of the reasons why robot-based assessments have not yet been integrated in clinical practice at a large scale. Therefore, future developments in rehabilitation robotics should enable the clinician to choose among a set of assessment tools according to the specific needs of the patient. We encourage engineers to develop assessment technologies that are not limited by practical constraints and administrative burdens. We believe that the barriers that prevent the translation of robotic assessments to clinical use must be understood so that they can be overtaken. Hence, to guide the development of future robotic-based assessment tools, it is fundamental that we understand the needs of the key players and adjust our motivation to develop new technological solutions.
This paper is targeted to researchers and technical developers in the field of robotic neurorehabilitation. The goal is to provide a comprehensive review of the state-of-art robot-assisted methods, with focus on the lower limb, and identify gaps in which robotic technologies can solve current issues in the assessment of sensorimotor functions. We present and discuss existing assessment methods for lower limb functions used in routine clinical practice and contrast those to state-of-the-art instrumented and robotic technologies. We also provide guidelines and recommendations for the development and validation of new sensor- and robotic-based assessment methods, taking into account the clinical needs. The review and recommendations provided in this paper aim to guide the design of the next generation of robotic devices.
Walking recovery is among the most desired goals of patients after a neurological injury [12, 13]. In order to maximize the recovery of the walking function, an optimal therapeutic plan should be defined and adjusted according to the patient’s progress. However, the lack of quantitative and sensitive assessments of lower limb functions that can be used during every day clinical practice limits the possibility to record the patient’s progress over time. For this reason, the scope of this review is constituted by measures and assessment methods that target body functions of the lower limbs, with a particular focus on those related to walking. We decided to exclude assessments of functions that, although needed for walking, are influenced by body systems other than lower limbs (e.g. balance). The methods and papers mentioned in this review were selected from an electronic search in PubMed and Google Scholar. Concerning the robotic measures, for each section we searched for the particular topic (e.g. “range of motion”) and the word “robotic” OR “robot”. Only papers relating to the lower extremities were considered. We looked also at the literature relevant to the robotic gait trainers and exoskeletons. The recent review from Zhang et al.  provided a good list of references on ankle devices. We also performed a manual search among the references considered relevant that we found in the selected articles. We aimed at a comprehensive, but not necessarily systematic or exhaustive review.
Assessments of sensorimotor functions can be discussed in the framework of a comprehensive classification for describing health and health-related states developed in 2001 by the World Health Organization. The International Classification of Functioning, Disability and Health (ICF) forms a conceptual basis for the definition, measurement and policy formulations for health and disability . The main aim of the ICF is to provide decision-makers in heath related sectors with a planning and policy tool. Moreover, relevant data can be collected in a consistent and internationally comparable manner. In the ICF, limitations of function and disability are not considered to be etiology-specific but rather are seen as reflecting common manifestations of underlying health conditions . In the same way, the assessments discussed in this review are not disease-specific but are applicable to different kind of populations. The ICF is a useful framework to conceive new robot-based assessment tools and to categorize existing ones. The ICF describes health and health-related states by means of three categories: functioning at the level of body or body part (Body functions and structures), the whole person (Activity), and the whole person in a social context (Participation) . The functions addressed by this review are listed together with their ICF classification in Table 1.
Assessments validation – psychometric properties
In this section, we will present some of the most relevant statistical analyses that are commonly used to evaluate the psychometric properties of an assessment tool. Throughout the paper, we will refer at these definitions to describe the adequacy of the clinical and robotic-based assessment methods. One of the main challenges for the acceptance of new robot-based assessments in clinical practice is their validation. The lack of information on the validity and reliability of an assessment has been identified as one of the barriers to its use [1, 6].
Reliability must be tested first when developing a new assessment method. An instrument cannot be valid if the values it provides from repeated measurements are not consistent . The most common methods to assess the reliability of an instrument in medicine and sport are the Intra-class Correlation Coefficient (ICC) and the Standard Error of Measurement (SEM). The ICC targets the relative reliability (the degree to which individuals maintain their position in a sample over repeated measurements); the SEM measures absolute reliability (the degree to which repeated measurements vary for individuals) . These two methods are, therefore, complementary. ICC values are strongly influenced by the heterogeneity of the subjects (i.e. a high ICC can be obtained even if large differences between trials are present, provided that between-subjects variability is high) . Results of reliability test measured with ICC in a particular population cannot be extended to a study including a different population. The SEM quantifies the precision of individual scores within the subjects , but its direct calculation involves the determination of the standard deviation (SD) of a large number of scores from an individual. In practice this is not possible; therefore, the SEM is estimated (Table 2). SEM is independent of the population from which it was determined and it is not affected by between-subjects variability as is the ICC . Absolute reliability can be also evaluated using the Bland-Altman plots . Here, for each subject, the mean of two measurements is plotted against their difference.
The presence of systematic bias is confirmed when the mean of the differences between the two tests is significantly different from zero. The limits of agreement (LOA) are another measure of absolute reliability: they indicate the range where, for a new individual from the studied population, the difference between any two tests will lie with a 95 % probability . When the test is used to detect changes between sessions within the same individual, these changes can be considered significant only if they fall outside the LOA. Therefore, the broader the LOA, the larger the minimal detectable change (MDC) would be for a given sample size in an experiment.
Validity assessment is usually more complex because generally the “true” value of a measure is not known with absolute certainty. The general approach for validating robot-based assessments so far consisted in applying correlation between instrumented measures and clinical scores in order to find which parameters measured by robots are able to reconstruct established clinical tests (concurrent validity). However, tying the validation of an instrumented method to a score that is subjective and ordinal-based could be questionable. When a gold standard is already established (e.g. isokinetic dynamometer for muscle strength measurement), concurrent validity can be tested against it. Without such standards, validity is tested indirectly as the ability of a tool to measure the underlying theoretical construct (construct validity) .
Responsiveness is the ability of a test to accurately detect change when it has occurred . Reliability highly influences responsiveness because real changes can be masked by the measurement error if the reliability of the test is poor. Measures characterized by a limited number of categories have intrinsically low responsiveness because large changes in status usually are required to the patient in order to change category. Ceiling and floor effects limit responsiveness at the extremes of the score range, since further improvement or deterioration cannot be monitored. The minimal clinically important difference (MCID) is a concept useful to consider the patient’s perspective when dealing with assessments. It involves both a minimal amount of patient reported change and changes important enough to modify patient management .
Overview of clinical assessments and robotic measures of lower limb functions
The following sections provide an overview of assessments methods for different outcome measures. For each outcome measure, its definition and relevance, the ways it is measured in clinic and in research settings are presented. For each of the available instrumented and robotic measures, the advantages over the current clinical assessments as well as points for improvement are also discussed. An overview of the validity, reliability and responsiveness of the clinical assessments discussed in this paper can be found in Table 3. Table 4 provides a list of psychometric properties of available robot-aided assessments. However, the limited amount of studies on validation of the proposed robotic measures prevented the completeness of the table.
Range of motion
Definition of the measure
Range of Motion (ROM) can be defined as the range, measured in degrees, through which a joint can be moved around one of its axes. Active ROM (aROM) is performed by the voluntary movement of the patient, while the assessment of passive ROM (pROM) implies that the therapist (or a robotic devices) rotates the patient’s joint distal segment with respect to the proximal segment  while the patient tries to relax. A minimum level of joint ROM is required to perform activities of daily life in a safe and energy-efficient way [24, 25]. For example, reduced knee ROM in the sagittal plane prevents an adequate foot clearance and leads to compensatory mechanisms . After a neurological injury it is common to observe a decreased ROM and a pathological behavior at the extremes of the ROM. To quantify this pathological behavior the “end feel” is sensed, which is defined as the resistance of the joint in response to a gentle overpressure applied at the end of the ROM . A decreased ROM and pathological end feel can be due to weakness, spasticity, pain, tendon and muscle contractures or ectopic bone formation [27, 28].
Clinical assessment and open issues
The most common instrument used in clinical practice for measuring joint ROM is the universal goniometer. The therapist must place the axis of the instrument over the axis of movement of the joint, aligning the stationary arm with the proximal segment and the moveable arm with the distal segment. pROM is assessed to determine the mobility of a joint regardless of the voluntary ability of the patient and it is usually slightly greater than aROM and much greater in case of muscle weakness. aROM values can be diminished when the movement is performed against gravity, especially in weak patients. When assessing the end-feel, the therapist manually determines the type of this resistance (e.g. “hard”, “soft”, “firm” etc.), which is indicative of different pathologies or conditions that can affect the normal ROM of a joint .
Moderate to substantial intra-rater reliability and validity for ROM measurements can be achieved by means of the universal goniometer (Table 3), but inter-rater reliability is generally lower and highly dependent on the therapist’s experience [23, 29–31]. The inter-rater reliability of pROM and of end-feel measurements is particularly critical because it depends on the torque exerted by the therapist on the patient’s joint [30, 32]. Therefore, it is highly recommended that the assessment is performed by the same therapist following a rigid standardized measurement protocol . Additional sources of errors in the measurements are the incorrect identification of the joint axis, the improper alignment of the goniometer arms with the body segments (also due to the movement of the joint) and the parallax error when reading the scale . Moreover, the measures can be affected by compensatory motions occurring at other joints.
State of the art in rehabilitation robotics
Measures of ROM are obtained through angular position sensors, for which different technologies are available. Within the existing robotic devices available for clinical use, isokinetic dynamometers (see section Muscle force) embed ROM measurement procedures [34, 35]. Driven gait robots for treadmill walking (e.g. Lokomat , LOPES , ALEX , ARTHuR ) and exoskeletons for overground walking (e.g. Vanderbilt , Kinesis , ReWalk , Ekso , H2 , Vlexo ) usually embed potentiometers or encoders in the robotic joints to measure joint angles. Nevertheless, the only method for pROM assessment in a gait trainer available for clinical use is implemented in the Lokomat: the procedure requires the therapist to move the limbs of the patient strapped in the device . For research purposes, several attempts to obtain instrumented measurements of the ankle joint have been made, often embedding ROM and stiffness evaluation (see section Joint impedance ) in the same device. For example, potentiometers were used in two ankle robots to train and assess active and/or passive plantar- and dorsiflexion ROM in stroke patients [45–47] and in a device able to assess ankle rotations in the 3 planes . Another robotic ankle trainer, the Anklebot, embeds encoders to estimate the ankle dorsi-plantarflexion and inversion-eversion angles .
End-feel assessment, at the best of our knowledge, has not been realized yet in a lower limb device. Nevertheless, attempts to develop an instrumented end-feel assessment were made for the shoulder joint [50, 51]. The authors used a force sensor to measure the applied force and a motion tracking system to assess the joint displacement. The rationale behind this approach is that the end-feel can be interpreted as the displacement induced by a force applied at the end of the joint ROM. It is, therefore, a measure of stiffness and as such it can be quantified by applying a known force and measuring the joint displacement at the end of the ROM . However, research in this field is still at an early stage and no information on validity and reliability of the measurements are available.
Future developments in rehabilitation robotics
Rehabilitation and assistive robots usually make use of angular position sensors in their hardware for control purposes and it would, therefore, be natural to conceive robot-aided joint ROM assessments. The development of new technologies in rehabilitation robotics can address many of the issues of current clinical measures of joint motion. aROM measures can be improved by using robots that are able to compensate for gravity while the subject performs active movements, making the assessment independent of the body orientation with respect to gravity. Transparency of robots must be ensured by means of backdrivable actuators or particular control strategies (e.g. admittance control , frequency oscillators ). The mechanical limits of a robotic joint should be designed in order to allow a subject to reach the whole ROM. Otherwise, the measures will saturate to this limit, leading to an underestimation of the patient’s ROM . The stabilization of the patient’s joints other than the joint of interest and the reduction of compensatory movements can be provided by mechanical fixation to the robotic device. Nevertheless, compensatory movements can be very difficult to detect, especially when they occur within the same joint under test; in this case they can only be identified from the careful eye of the examiner . During the measurement of pROM and end-feel, robots can impose a standardized movement in terms of torque and/or speed . This would improve the reliability of the test making it independent of the operator. Moreover, pre-defined sequences of movements can be programmed using robotic devices in order to have a standardized measurement protocol.
Exoskeletons for overground walking could potentially be used for measurements in static and dynamic conditions provided that gravity, friction and inertia are adequately compensated (see section Walking function/Gait pattern ). For example, a versatile passive exoskeletal research platform (Vlexo) developed to study human-robot interactions was designed to have robotic joint ROM higher than the human ROM . Each degree of freedom could be blocked to avoid compensatory movement. Thanks to the high adaptability and instrumentation possibilities, it would potentially become a good tool for measuring simultaneously the ROM of hip (abd-adduction, int-ext rotation, flex-extension) and knee in static and dynamic conditions.
End-feel assessment procedures can be implemented with a similar approach as for the shoulder joint [50, 51], using for example motorized exoskeleton devices [55, 56] or ankle robots [46, 48, 49] equipped with angular position and force sensor.
Concerning the measurement technology, the most used angular sensors in robotics are potentiometers, due to their robustness, accuracy and low price. However, since they must be aligned with the joint’s axis of rotation, the measures could potentially suffer from misalignments when the anatomical joint does not have a single axis of rotation or when the setup is not properly done. To overcome this issue, other sensor technologies that do not require the identification of the joint axis can be used. Flexible goniometers based on strain gauge technology are available on the market (e.g. Biometrics Ltd. – uniaxial or biaxial, ). The end blocks are fixed to the segments that form the joint and the angle of flexion-extension and abduction-adduction can be recorded, provided that the device is attached in a suitable plane. They have very good performances both in static and dynamic conditions [58–60], but they are at present not sufficiently robust for daily clinical usage. In wearable applications, strain sensors  and optic fibers  have been used due to their low encumbrance and low weight, but at the moment their performance is not adequate for accurate measurements. Among the wearable sensor technologies, Inertial Measurement Units (IMUs) are promising instruments, given the good performances shown so far, especially in knee dynamic ROM measurements [63–66]. However, they require calibration and signal processing algorithms that perform sensor fusion and compensate for possible inaccuracies due to electromagnetic interferences.
Further studies are recommended to define the hardware configuration, the sensor technology and the measurement protocol that maximize the validity and reliability of the aROM, pROM and end-feel assessment in a clinical context, with the temporal and economic limitations that this implies. Wearable technologies could give an insight of the ROM that the patient is able to display in a real-life situation.
Definition of the measure
Muscle strength is defined as the amount of force generated by muscle contraction . Muscle weakness, or the inability to generate normal levels of force, has clinically been recognized as one of the limiting factors in the motor rehabilitation of patients following stroke  and it is one of the major clinical manifestation in hereditary neuromuscular disorders and injuries of the spinal cord . The amount of preserved voluntary muscle contraction has been proven to be highly correlated with walking ability in incomplete SCI  and stroke . In the elderly population, lower limb muscle weakness has been associated with an increased risk of falls . In the lower limbs, muscle weakness can be ascribed to disuse atrophy and to the disruption in descending neural pathways leading to inadequate recruitment of motoneuron pools [68, 72]. Assessing muscle strength is important to determine the severity of the injury, to plan the therapy and to monitor the effects of rehabilitation treatments .
Clinical assessment and open issues
In clinical practice, muscle strength is typically assessed using manual muscle testing (MMT) (e.g. Medical Research Council scale ). MMT grades strength according to the ability of a muscle to act against gravity or against a resistance applied by an examiner (0: no muscle contraction, 5: holds test position against maximal resistance) . However, the accuracy and sensitivity of MMT is low and the same grade in MMT corresponds to a large range of absolute strength values . It was reported by  that Beasley found that a variation of less than 25 % in muscle strength for the knee extensor cannot be detected by MMT . MMT is strongly influenced by the experience of the examiner, who must avoid compensatory movements by the subject and ensure a standard positioning. MMT suffers from ceiling effects, because the maximum score (5.0) is assigned before a normal level of muscle strength is truly reached . MMT was found not adequate as a screening tool and insufficient in tracking the progress of a patient undergoing therapy [77, 78]. Subtle increases in muscle strength are only detectable with instrumented methods.
Quantitative measures of muscle strength can be performed during isometric, isoinertial or isokinetic contraction. In an isometric test the subject is asked to perform a maximum voluntary contraction (MVC) against a fixed resistance and the maximum value of the force/torque is retained. In clinical practice, this test is mostly performed with a hand-held dynamometer (HHD) or myometer. The HHD is a portable force sensing device that can be placed between the hand of the examiner and the body segment to test, similar to how an examiner would perform a MMT . The examiner must be able to apply a resistance equal or greater than the patient’s force. Like the MMT, the myometry is, therefore, depending on the amount of resistant force the practitioner is able to apply to the segment of interest and on his ability to stabilize proximal joints [79, 80]. Nevertheless, with respect to MMT, myometry has higher sensitivity and it is less prone to ceiling effects . Reliability and validity of HHD measures can be further increased by fixating the device with a belt [81, 82], so that the resistance applied against the movement is not dependent on the examiner’s force. Load cells mounted on supportive frames can also be used for this purpose . Isoinertial tests consist in lifting a constant load throughout the joint ROM and the outcome is the maximum load that can be lifted once (1-RM) . Isoinertial tests are usually executed using sport devices, like the leg extension machine, modified in order to record the joint angle . During an isokinetic contraction the joint angular velocity is kept constant by a machine, the isokinetic dynamometer. The subject is asked to forcefully contract the muscles during the whole ROM while the peak torque is calculated. This test can only be performed with a robotic device and it will be discussed in the next section. Isokinetic tests could be useful to unmask speed-dependent strength impairments . Although the isokinetic dynamometer is considered the gold standard for muscle strength measurements, price, encumbrance and setup time limit its use in a clinical setting. Therefore, it was proposed to use preferably isometric or isoinertial tests in clinical practice due to their reduced cost and easiness of use [84, 85]. The three test modalities have indeed similar good construct validity (relation with physical function) and substantial test-retest reliability  and high correlations have been found between isometric and isokinetic torque measures, although isometric tests lead normally to higher values of muscle strength [84, 87]. It is important that users are aware that these three conditions provide different estimates of muscle strength. Nevertheless, it was demonstrated that using the HHD according to standard procedures and fixation, excellent inter and intra-tester reliability and a good correlation with the isokinetic dynamometer can be achieved [73, 79, 81, 88]. Therefore, given the cost and long measurement time (around 25 min) required by the isokinetic dynamometer, it was suggested to favor the use of HHD in clinical practice [79, 88].
State of the art in rehabilitation robotics
The most known device for muscle strength measures is the isokinetic dynamometer. This machine allows the measurement of joint torques in controlled conditions: isometric at selected joint angles or isokinetic at selected angular velocities [79, 89]. A servo-controlled lever arm provides resistance to the subject’s joint when it reaches a defined angular velocity (≥0 deg/s). Different mechanical configurations allow testing of hip flexion-extension and ab-adduction, knee flexion-extension, ankle plantar-dorsiflexion and eversion-inversion. The patient’s trunk and the segments proximal to the joint tested must be stabilized with straps and the axis of the dynamometer must be carefully aligned with the axis of the joint to test to avoid measurement inaccuracy . In isokinetic tests the subject is asked to push “as hard and as fast as possible” while the device provides resistance to the movement of the limb so that it cannot accelerate beyond the machine’s preset angular speed . A continuous passive motion (CPM) has been proposed for severely impaired subjects, where the robot moves the limb and the dynamometer lever arm at a preset velocity while recording forces applied to the lever arm . Reliability and validity of the isokinetic dynamometer are substantial but the high cost and the long setup time limit its use in everyday clinical practice.
In rehabilitation robotics, muscle strength has been measured integrating force sensors into the structure of exoskeletal devices for quantifying physical human-robot interaction and estimating the force exerted by the patient. Directly measuring the interaction force at the attachment points requires a load cell, placed at the connection between the cuff/orthosis and the exoskeleton link, such as in a modified version of the Lokomat [91, 92]. Otherwise, the estimation of interaction torques can be achieved through a force sensor in series with the actuators, like in the Lokomat  and in the ALEX , or through linear potentiometers for measuring the length of the springs used in the actuators of the LOPES I . The torques produced at each joint are calculated online from the joint position and the linear force values. The Lokomat, in particular, allows the execution of hip and knee isometric strength tests in the sagittal plane: the patient is positioned with 30° hip flexion and 45° knee flexion and asked to flex or extend the joints against the resistance provided by the orthosis. A moderate to substantial inter- and intra-rater reliability of this method was found with patients with and without neuro-muscular disorders .
The ankle joint is usually measured separately from the hip and knee joints with dedicated devices used in a sitting position [14, 94]. An ankle robot constituted by a footplate fixed through a six-axis force sensor to a servomotor shaft that controls its angular position and speed was used for measuring isometric muscle strength: the subject’s ankle was locked at the 0° ankle dorsiflexion, and maximal voluntary contraction was taken [46, 47]. Isometric torques of the ankle joint in different kinematic configurations were obtained from a device able to measure ankle torques around the three articular axes (plantar-/dorsiflexion, int-/external rotation and pronation/supination). The 6-DOF structure allows linear and angular displacement of the ankle with respect to the shank. Each DOF is blockable in different configurations and torques and angles can be measured .
Future developments in rehabilitation robotics
Despite the poor psychometric properties of the MMT, methods alternative to this test that can be easily integrated in a clinical setting are lacking. Robotic devices can address many of the problematics previously identified. The responsiveness of muscle strength tests is important for detecting small changes during the progression of rehabilitation. Therapy goals can be set based on the minimum force required for performing activities of daily living, like walking or sit-to-stand [95, 96]. It is important that a test is able to detect changes at least equal to the MCID. However, MCID of muscle strength changes in patients with neurological disorders have not yet been established. Ceiling effects must be avoided in order to have a measurement scale that can be used also with mildly affected patients. Robotic devices have the potential to provide more sensitive assessments thanks to the sensors embedded in their structure. Standard and repeatable testing conditions can be achieved by implementing a system for fixating the patient to the device and preventing undesired movements and by programming a standard sequence of movements that should avoid fatigue effects . Moreover, assessment procedures can be integrated in a therapy session performed with a rehabilitation device without requiring additional setup time.
The isokinetic dynamometer is a first attempt to provide a state-of-the-art robotic assessment method . A large body of research on this device have unraveled the possible shortcomings and studied different applications and measurement protocols. In particular, factors such as gravity compensation, damping of the system, human-machine interface and alignment of the human and robot axes of rotation have been considered in many publications [85, 89, 99]. This knowledge can be applied to the development of future robot-aided muscle strength assessments, despite the fact that the differences in hardware prevent the complete reproducibility of the results. Testing subjects with severe weakness requires particular attention because subtle levels of muscle strengths can be masked by the use of device that is too heavy for the patient or the use of a position that does not eliminate the effect of gravity . Lastly, the motivation of the patient plays an important role  and it would be worthy to investigate how this human factor affects the outcome measures and consequently to standardize the protocol and the instructions.
Definition of the measure
Proprioception can be defined as the ability of an individual to determine joint and body movement (kinaesthesia) as well as position (statesthesia) of the body, or body segments, in space [100, 101]. It is based on sensory signals provided to the brain from muscle, joint, and skin receptors , with muscle spindles playing the major role . Proprioceptive feedback has been demonstrated to be a key component of motor control and functional joint stability . A diminished proprioceptive acuity at the ankle joint is associated with a lower unipedal stance time, which is a measure relevant for evaluating frontal plane postural control . Loss of proprioception has been reported both in neurological (e.g. stroke, SCI, peripheral neuropathy) and in orthopedic patients (e.g. knee osteoarthritis) and it has been associated with an increased risk of falls in the elderly .
Clinical assessment and open issues
Assessment of lower limb proprioception in clinical practice is based mainly on two rather simple tests: the movement detection at the big toe and the Romberg sign . In the first the examiner moves the patient’s toe upward or downward and asks to detect the direction and the amplitude of the movement. In the Romberg test, the subject is asked to close his eyes while standing with his feet close together. A non-specific proprioceptive deficit would usually result in the loss of balance. While useful as a quick method to detect the presence of proprioceptive abnormalities, these tests are not sensitive enough to detect mild impairments or to track changes over time. Moreover, the test at the big toe depends strongly on the pressure applied by the examiner and the amplitude of the movement imposed . Furthermore, only the distal segments of the upper and lower limb are tested and no assessments of the proximal joints are performed. A more specific test, even if less used in clinical practice and mainly in upper limb examination, is the joint position reproduction or matching (JPR) . In this test the patient is blindfolded and the examiner moves his/her limb to a target position. The patient is then asked to match this position either with the contralateral limb or with the same limb after it has been brought back to the starting position. This test is normally performed without any instrument and the visually observed mismatch in position is retained as a rough measure of proprioceptive precision [102, 107]. Goniometers can also be used to measure the joint angle before and after the matching but their reliability and measurement error have been shown to vary widely . Items related to proprioception are included also in the sensory-related section of the Fugl-Meyer Score for stroke patients. Here small alterations in the position of hip, knee, ankle and great toe are evaluated . However, the stimulus provided by the examiner is inherently subjective and sensitivity is limited to 3 levels (absent, impaired or normal proprioception).
State of the art in rehabilitation robotics
Instrumented tests for proprioception in lower limbs have been developed using motorized devices or isokinetic dynamometers. An overview of these experimental devices and methods can be found in [100, 110].
The classic JPR test discussed above can be easily instrumented. A machine moves the subject’s limb to the target position. The subject is then asked to match this position, either by actively moving the limb or by pressing a button when the limb passively moved by the machine reaches the target position. However, it has to be taken into account that active and passive motion of the limbs are not equal in terms of sensory feedback . JPR methods are not suitable for people with cognitive impairments since they are highly dependent on memory . Moreover, they have been found to have slight to moderate reliability . A JPR test for assessing hip and knee joint proprioception has been implemented in the robotic gait orthosis Lokomat and tested in healthy subjects and 23 incomplete SCI subjects . The subject’s leg was positioned at a target hip and knee angle and then moved away to a distractor position. The subject was then asked to place the limb at the remembered target position using a joystick to control the robot. The absolute error between target and remembered position was retained as outcome measure. The test-retest reliability in SCI was found to be fair at hip joint and substantial at the knee joint but the Bland-Altman plots showed broad LOA that indicate a low sensitivity in SCI individuals. Heteroscedasticity was also reported. Nevertheless, the score correlated well with the clinical assessment of proprioception and a significant difference between SCI patients and healthy subjects was found.
A second approach for measuring proprioception is the threshold to detection of passive motion (TTDPM). In this test the body segment under test is moved by a machine in a predefined direction. The subject is asked to press a button as soon as he/she detects a movement. Movements are presented at different velocities since the proprioceptive threshold decreases with increasing speed [100, 111]. A motorized apparatus for testing hip, knee, ankle and toe detection threshold was developed by Refshauge et al. and the influence of speed and joint position on the test outcomes was studied [111, 112]. A modified isokinetic dynamometer and a chair with motorized arms have been used for assessing passive flexion/extension and varus/valgus movements of the knee in healthy subjects and osteoarthritic patients (OA) [104, 113]. From the initial posture, the servomotor rotated the knee at a constant low velocity of below or equal to 1°/s). The threshold position of detection of the movement was retained, with smaller threshold values indicating greater proprioceptive acuity. Reliability was found to be excellent both within and between raters, both for OA and healthy subjects. In both studies the subjects wore headphones and an eye mask. The TTDPM was tested also using the Lokomat : hip and knee separately were moved according to a randomized order of speeds (0.5–4°/s), directions and catch trials (no movement). Angle and reaction time were used to calculate a movement detection score. The score presented substantial reliability and a high correlation with a clinical score of proprioception, showing better sensitivity (it is possible to measure reaction times ≥ 50 ms) and no ceiling effects. Faster speeds were able to elicit a response in severely impaired subjects that could not detect movements at 0.5 °/s. The TTDPM test leads generally to more precise and less variable measures of proprioception acuity than the JPR test. Interestingly, the two tests have shown no concurrent validity .
Future developments in rehabilitation robotics
These studies demonstrate that instrumented and robotic assessments of proprioception are feasible and present several advantages over clinical assessments of proprioception. Measures of proprioception in clinical practice are rather coarse and lack granularity. Standardization is nearly absent and the outcome of clinical tests is often a binary answer.
Lower limb robotic devices provide the possibility to maintain a high consistency in the protocol (speed, points of contact, timing) between trials. The responsiveness of the robot-based measure was demonstrated also by the ability to detect a wide range of angle errors in subjects that were judged unimpaired by the clinical assessment [106, 114]. Moreover, the influence of motor impairment on the control of lower limbs can be eliminated because the leg can be passively moved by the robot. Lastly, robotic devices can provide useful information on joints that are not normally addressed in clinical practice, where the most common examination involves only the big toe . It is likely that specific information on other joints might provide an insight on different components of sensory function useful to track changes in recovery after injury . On the other side, the straps of exoskeletal devices may provide additional cutaneous feedback to the subject, thus influencing the measurements . When designing a new robotic device or protocol for proprioception assessment it is important to consider that the test methods (JPR or TTDPM) do not provide the same information . Different versions of the protocol exist also within the same test and again their choice can highly influence the results . The speed of a TTDPM test highly influences the outcome measures [100, 114] and must be accurately controlled by the robotic device. Active and passive movements are likely to activate different proprioceptive mechanisms .
Robot-based assessments of proprioception require longer time of administration with respect to clinical assessments, but they are able to provide reliable and sensitive information on proprioceptive acuity that allows a more detailed examination useful for diagnosis or accurate tracking of the recovery of the patient.
Abnormal joint torque coupling and synergies
Definition of the measure
Due to cortical damage, stroke survivors and cerebral palsy (CP) children can lose the ability to move their joints independently, which result in abnormal coupled, pathophysiological movement patterns, also called synergies. The loss of independent control of joint moments is caused by involuntary co-activation of muscles over multiple joints .
Brunnstrom  defined two often occurring pathophysiological synergies in the lower extremities:
Extension synergy consisting of internal rotation, adduction and extension of the hip, extension of the knee, and plantar flexion and inversion of the ankle
Flexion synergy consisting of external rotation, abduction, and flexion of the hip, flexion of the knee and dorsal flexion and eversion of the ankle
Clinical assessment and open issues
Loss of independent joint control limits the performance on activities of daily living. Therefore, in both clinical and in research settings abnormal joint torque coupling is often being assessed and this is mostly done using the Fugl-Meyer Assessment of Physical Performance . This scale has been shown to be a reliable, sensitive and valid method for the assessment of motor impairment after stroke [117–119]. However, it can be argued that for the quantification of abnormal joint torque coupling this scale lacks sensitivity due to the use of a 3-point scale (0 = cannot perform,1 = performs partially, 2 = performs fully) for the assessment of each component of torque coupling.
State of the art in rehabilitation robotics
Robotic and robot-related measures could possibly provide more accurate information. Over the last decade several studies have investigated abnormal joint torque coupling using robotic and robot-related measures [68, 92, 120–126]. The majority of these studies quantified the synergies in static situations during isometric contractions and used a similar approach. Subjects were strapped into a (robotic) device (most often the Lokomat) that constrains every movement of the concerned leg and the pelvis. The device was equipped with force sensors to measure all the interaction forces/torques that the subject exerts with this leg on the device, for instance the cuffs of the Lokomat were instrumented with 6-DOF load cells [92, 120, 121]. Participants produced isometric torques in a particular direction (primary), while torques in all other the directions (secondary) were also measured. Abnormal torque coupling was quantified as the difference in secondary torque production between healthy individuals and stroke survivors. Studies differed in the amount of joints and planes that were investigated and the position in which the coupling was assessed. Thelen et al.  assessed the coupling while subjects were positioned in an adjustable chair with ankle fixed to six degree- of-freedom load cell, whereas others assessed the coupling while subject where standing in the toe-off and/or mid-swing position with the test leg unloaded [68, 120–122].
Thelen et al. showed that individuals with cerebral palsy produced a knee extension moment during hip extension and vice versa whereas healthy subjects produced a knee flexion moment during hip extension and a hip flexion moment during knee extension. Quantification of abnormal joint couplings using a (robotic) device has provided evidence for different couplings. Neckel et al.  found that stroke survivors only showed an abnormal coupling between hip abduction and flexion and had similar couplings as found in healthy subjects for the other degrees of freedom. Cruz and Dhaher  observed that stroke survivor coupled knee extension with hip adduction. Tan et al.  found strong coupling between ankle frontal plane torque and hip sagittal plane torques and vice versa that were not present in the healthy control subjects (ankle plantar flexion with hip adduction, ankle eversion with hip extension and ankle inversion with hip flexion). Recently, Sanchez et al.  also found evidence for the earlier found coupling between hip extension and adduction, and ankle plantar flexion and hip adduction. So, evidence starts to accumulate that stroke survivors have abnormal coupling between hip adduction, hip extension and plantar flexion.
To our knowledge only one study has attempted to identify abnormal joint torque coupling during walking . In this study participants were moved along a predetermined locomotor trajectory using the Lokomat while interaction and ground reaction forces were measured. However, the difficulty with this setup is that it is hard to disentangle the torques required for walking and maintaining balance and those resulting from the abnormal joint torque coupling. Therefore, although assessed in a quasi-dynamic situation, the results may not be generalizable to voluntary walking.
The reliability (test-retest, inter-rater, intra-rater) has not yet been assessed for these abnormal couplings, nor has its responsiveness been determined. The criterion validity has not explicitly been investigated, however Cruz and colleagues  demonstrated using step wise regression that the coupling between knee extension and hip adduction was the best predictor of gait speed amongst other strength and coupling variables. None of the aforementioned studies did correlate their coupling measures with a clinical scale like the Fugl-Meyer to assess the construct validity.
Future developments in rehabilitation robotics
To summarize, robotic measures may be able to quantify abnormal joint torque coupling more precisely compared to clinical measures such as the Fugl-Meyer Assessment of Physical Performance. However, the reliability, responsiveness and validity of these measures need to be further investigated. Additionally, robotic assessment is still performed under static or quasi-dynamic conditions, which might not quantify well how these couplings limit walking. For assessing abnormal couplings in the upper extremities, the assessments have moved from a static approach  to a dynamic approach where the couplings are assessed during reaching movements using robotic devices . We foresee that a similar shift will happen for the lower extremities. Integration of the principle used in the robotic assessment under static conditions in robotic gait trainers could provide the tools to assess abnormal joint torque coupling during walking.
Definition of the measure
In the clinical field, the term joint stiffness has been used to express the sensation of difficulty in moving a joint . While this term is commonly used in the clinical practice, the notion of stiffness used in this context does not match the definition of stiffness in classical mechanics. To describe all the mechanisms that contribute to the resistance of motion, the term impedance is usually preferred. In motor control literature, the term mechanical impedance is defined as the dynamic operator that specifies the force an object generates in response to an imposed motion . The latter definition includes all motion-dependent effects, i.e. those terms that specify the force generated by changes in position (e.g. stiffness, non-elastic forces), in velocity (e.g. viscosity, damping) and in acceleration (e.g. inertia) . In biomechanics, the term joint impedance relates the motion of the joint and the torque acting about it . Joint impedance is usually estimated by applying a torque or force perturbation and measuring the resulting change in position or applying a position perturbation and measuring the resulting change in torque of force.
Joint impedance is mainly determined by three sources: i) the passive biomechanical properties of the muscles, tendons and tissue around the joint and limb inertia – passive components; ii) the resistance produced by the muscles in response to reflexes [134–137] – reflexive components; and iii) the resistance produced by the muscle fibers due to non-reflexive, neural-driven contractions – intrinsic components . Since the reflexive and intrinsic component are both related to muscle activation, their sum is commonly referred to as active componentFootnote 1.
In neurological populations, an abnormal increase in joint impedance can result from spasticity, rigidity or dystonia . The intrinsic and reflexive components have also been shown to be affected in neurological populations .
Joint impedance varies with muscle contraction , joint position [141–143], rotation amplitude , and the duration of the applied perturbation, since after approximately 30 ms cross-bridges break  and the contribution of cross bridge stiffness to the overall joint impedance will diminish. Joint position affects joint impedance measurements because the intrinsic component increases towards the extreme joint angles as the ligaments get more stretched. Additionally the different muscles vary their active contribution to the joint impedance depending on their length (and therefore on the corresponding joint configuration), due to the particular shape of the length-tension curve of the muscle . The reflex activity is also known to be speed dependent  and only contributes above a threshold . Finally, the task instruction given to the subject will also shape the joint impedance . Most common task instructions are ‘relax’,’resist the perturbation’, or ‘keep the force constant’.
Clinical assessment and open issues
The Modified Ashworth Scale  is the most widely used clinical assessment to quantify an abnormal increase in joint impedance due to excessive muscle tone. The MAS consists of moving the limb of the patient through its range of motion and rating the resistance on a 6-point scale. The MAS is widely accepted, even though the validity and reliability of the measure are questionable  since especially inter-rater reliability was slight to fair. Moreover, the MAS may also lack sensitivity. The MAS assess joint impedance only in passive conditions, where the subject is asked to relax, which might not be indicative for how spasticity influences dynamic movements. Another test to assess the increased resistance to movement in a more quantitative way is the pendulum test, first described by Wartenberg . This test quantifies movements of the lower leg following its drop from a horizontal position by deriving the angle of first reversal, the maximal angular velocity or number of oscillations. The pendulum test has shown good convergent validity, reliability and sensitivity [153, 154]. Some limitations of this test are that it is done in relaxed conditions – which is difficult to achieve - and can only be used for the knee. Additionally, measuring equipment (electrogoniometers, inertial sensors) are needed to record the leg motion and to extract the variables.
While measurement of joint impedance in not commonly performed on the everyday clinical practice, it has implications in understanding a potential cause of impairment. For instance, Mirbagheri et al.  was able to isolate abnormal active contributions in spinal cord injury patients based on measurement of joint impedance of the ankle. Such measurements can also point out to different pathologies such as spasticity, rigidity or dystonia .
State of the art in rehabilitation robotics
As mentioned earlier, joint impedance is dependent on joint position, muscle contraction levels, and amplitude, velocity and duration of the perturbation. Therefore, the use of robotic devices is advantageous because these factors can be precisely controlled at the same time relevant signals are been recorded. Several instrumented and robotic measures have been developed to asses either the reflexive and/or intrinsic components of joint impedance [45, 49, 136, 138, 155–162]. We will not review all devices and methods. In particular for the ankle joint many devices have been developed, which have recently been reviewed . To assess passive joint impedance, the joint of the participant is moved by a robotic manipulator or manually over a certain angle often measured using a potentiometer while the resisting force is measured using force sensors integrated in the (robotic) device. For accessing the passive joint impedance it is important that no muscle activity is present. Therefore, the participant is asked to (try to) relax and the angular velocity is kept low to avoid the excitation of reflex contractions. In the push and pull test, the joint is moved with small increments and kept static for approximately 5 s in every position. The net moment (after removing gravity) provided by an external device to keep the segment in equilibrium is retained for each incremental position . Both isokinetic dynamometers and custom made joint actuators have been used as assessment devices. With a manually operated device the passive ankle impedance could be estimated reliably in healthy subjects (ICC values between 0.71 and 0.85,) and in CP children (ICC = 0.82, ). In the study of Chesworth et al.  a custom made torque motor system was used to assess passive joint impedance of the ankle with a comparable reliability (ICC: 0.77–0.94).
The contribution of active components (i.e. intrinsic and reflexive) to joint impedance have also been investigated using similar experimental setups. In a typical setup, the subject is either asked to actively resist an angular displacement or to (try to) exert a constant force. At some point, either an angular position perturbation is applied while the resisting force is measured or a force perturbation is applied and the resulting angle is measured. Impedance measured under this condition contains the three components: passive, intrinsic and reflexive. To be able to distinguish between these components, different strategies have been used. For example, in the study of McHugh et al.  the passive component is subtracted from the total impedance to determine the active component. Also more complex methods exist, which are based on system identification techniques. In the method of Mirbagheri et al. , a system identification method is applied to distinguish between intrinsic and reflexive components. In this method, pseudo-random continuous rotations of the ankle are applied, and the ankle torque and EMG of involved muscle groups are recorded. The model consists of an intrinsic component and a unidirectional delayed velocity feedback pathway representing the reflexive component. Input to the model is ankle rotation, and the model parameters are optimized to minimize the error between the predicted and recorded torque. The EMG is used to determine the latency of the reflex component. In healthy subjects a good intra-rater (r > 0.8) reliability was found . De Vlugt et al.  used similar techniques but instead of continuous rotations, they applied ramp-and-hold ankle dorsiflexion rotations with different speed profiles. They employed a nonlinear neuromuscular model that is more complex than the one used by Mirbagheri to predict the recorded ankle torque. Results showed that stroke survivors could be distinguished from control subjects by tissue stiffness and viscosity and to a lesser extent by reflexive torque from the soleus muscle. These parameters were also sensitive to discriminate different patients, who were clinically graded by the MAS. In a subsequent study  these researchers adapted their model and protocol slightly by applying both ankle plantar and dorsiflexion rotations. The estimated model parameters could discriminate between patients with CP and control subjects. Soleus background activity was sensitive to MAS spasticity severity, but reflex activity was not. Preliminary data indicated that reflex activity was reduced after spasticity treatment. The between-trial (ICC: 0.76–0.99) and between-day repeatability (ICC: 0.64–0.95) was moderate to substantial for tissue stiffness and background activity, but not for reflex parameters.
A shortcoming of most of the studies on joint impedance is that the assessment is done for static or passive tasks where the participants are in a supine, prone-lying or sitting position. The ankle impedance has also been determined in more natural active conditions, such as stance [166, 167] using very fast dorsi- and plantar-flexion rotations with a motorized footplate and non-parametric impedance estimates.
Aforementioned studies and approaches all made use of dedicated assessment setups. However also robotic gait trainers can be used to derive measures of joint impedance. For instance, the Lokomat has a built-in function to assess overall joint impedance of the hip and knee joints by passively moving the limbs at different speed profiles and recording the resulting joint torque. Using this technique, a moderate correlation between joint impedance and MAS scores could be seen . Koopman et al.  used the LOPES robotic gait trainer and multi input multi output system identification techniques to assess joint impedance of the hip and knee. Healthy subjects were assessed while in the toe-off or heel strike position and were asked to resist the movement of the device or apply no force at all. Results showed that the effect of biarticular muscles on the inter-joint impedance could not be ignored.
Future developments in rehabilitation robotics
Although research on the accuracy and reliability of robotic devices to assess joint impedance is not available for all developed devices and methods, it can be argued that the use of integrated sensors and robotic actuators will show better psychometric properties compared to the MAS score. Another advantage of robotic measures is that they can help to develop methods to estimate the active and passive or intrinsic from reflexive components, while the MAS only measures the resistance of motion but not the underlying cause. The pendulum test could be implemented in combination with transparent devices that do not hinder the natural oscillation of the shank (e.g. soft exoskeletons). However, the reviewed robotic assessments are still performed under non-functional and static or passive conditions. Therefore, further development is necessary to be able to assess joint impedance during a dynamic task such as walking. A method to estimate joint impedance during gait is to use musculoskeletal models and using optimization techniques to estimate muscle forces that are related in the model to muscle impedance . An alternative method is to apply time-varying system identification algorithms to estimate the changing impedance of the human knee over the gait cycle . The ensemble-based correlation technique averages over repetitions instead as over the time cycle . Averaging over repetitions and over time within a short data segment within repetition can also be combined  with the advantage that less repetitions are needed. A testing platform consisting of a knee perturbator has been built in order to deliver velocity perturbations during walking and record reaction torques, with the aim of determining the knee impedance using system identification techniques . The ensemble-based correlation technique has also been applied to estimate the modulation of the ankle impedance from the end of the stance phase to heel contact with MIT’s AnkleBot . However, comparing the estimated knee impedance of the ensemble-based correlation method with the model-based method  shows order of magnitude differences in the estimated knee impedance. Hence, more research is needed to reliably estimate the impedance of multiple joints during gait.
Walking function/Gait pattern
Definition of the measure
Walking can be defined as a repetitious sequence of limb motions that move the body forward while simultaneously maintaining stance stability . Gait refers to the manner or style of walking . Gait is composed by a cyclic series of motion patterns performed by the hip, knee and ankle. The gait cycle can be divided in phases, the main ones being swing and stance [174, 176, 177]. Walking can be described according to different domains: i) the capacity of performing activities related to walking (e.g. walking without assistance, sit-to-stand); ii) the spatio-temporal characteristics (e.g. speed, step length, cadence, stance/swing ratio); iii) the “quality” of gait pattern, which concerns the ability to coordinate lower-limb segments and joints (e.g., simultaneous coordination of hip and knee angles) .
Clinical assessment and open issues
In clinical practice, walking is mainly assessed by examining the spatio-temporal characteristics and the capacity of performing walking-related tasks. Like the other assessments discussed in this paper, measuring walking is also influenced by time constraints. Therefore, measures that are relatively easy and fast to administer are normally chosen.
Among these, the capacity of performing functional walking activities is commonly assessed using ordinal-based clinical scores. These tests have a low administrative burden and they can be useful to grossly categorize the patients according to their walking capacity, but they are not sensitive enough to detect small improvements in locomotion . The Walking Index for Spinal Cord Injury (WISCI II), for example, assigns a score between 0 and 20 based on the amount of assistance required for walking (e.g., walking with one/two crutches). Consistent floor and ceiling effects and a low responsiveness were reported . Moreover, the different levels are unevenly spaced, meaning that a change of 1 point in the score has a different relevance depending on the position along the scale . Several other activity-based tests were developed (e.g., Functional Ambulation Category (FAC), Dynamic Gait Index (DGI)) to assess walking function but, although very useful for gaining information on the overall walking process, they are unable to provide any detailed information on the way it is realized.
Time-based tests are often performed, since they provide quantitative measures and have shown substantial inter- and intra-rater reliability . For example, in the 10-m-Walking-Test (10MWT) a stopwatch is used to measure the time required to walk 10 m . Thus, the test provides a measure of short-duration walking speed and it has substantial correlation with other time-based walking tests and with other walking-related functions like muscle strength of the lower limb (Table 3) . However, the information obtained with this test is limited to gait speed, which, although normally used as a surrogate measure for gait quality , is not able to provide information on complex alterations of walking (e.g., compensatory strategies) . 10MWT and other time-based walking tests (e.g., Time-Up-and-Go, 6-min-Walking-Test) present floor and ceiling effects since non-ambulatory subjects score 0 and mildly impaired patients could walk longer distances at the same speed . Other spatio-temporal parameters can be obtained using more sophisticated instruments like IMUs [185–187] and pressure mats . Heel strike events can be detected from an IMU placed at the lower back . A more detailed step segmentation is possible if the IMUs are placed directly on the feet [190, 191]. Parameters such as step duration, step length and swing/stance time ratio can provide important additional information on gait impairments and on the progresses during recovery. For example, there is evidence that step variability (i.e. variability in stride time, stride length and gait speed) is altered in patients with neurodegenerative diseases . In stroke patients, asymmetry in right and left step time and altered stance/swing time ratio were reported using IMUs [193–195]. IMU-based systems are not yet widely integrated in clinical practice, even if new systems are now commercialized (e.g. McRoberts DynaPort , GaitUp ).
It is important to evaluate the patient’s gait pattern to understand whether the person is using compensatory strategies. These strategies might indeed not be visible in the spatio-temporal gait characteristics, which can be similar to physiological ones even in presence of an aberrant muscle activity . This is especially important in longitudinal studies which aim at demonstrating whether improvements in walking speed are attained either by using compensatory strategies or by restoration of the pre-morbid gait patterns . By using measurements able to capture the quality of the gait pattern it is possible to discriminate between the two different recovery strategies - compensation or restoration of physiological gait. However, at present the quality of the gait pattern can only be accurately assessed using a motion tracking system and force plates. This instrumented gait analysis provides an accurate measure of joint angles, moments and powers but requires a costly equipment and a long administration time.
A major issue related to walking assessment is that non-ambulatory subjects are often assigned the lowest score in every test (e.g. 0 m/s in the 10MWT, 0 score in the WISCI II), irrespective of their residual lower limb functions. These subjects are therefore excluded from it because most of the scales’ floor effect. It would be possible to assess non-ambulatory subjects indirectly by measuring other variables that correlate with walking ability, like muscle strength or balance. However, these tests are performed usually while sitting or lying, in contexts very dissimilar to walking.
State of the art in rehabilitation robotics
Driven gait robots for treadmill walking and exoskeletons for overground assistance can be used to record joint kinematics while walking in order to obtain information on the quality of the gait pattern. Robotic exoskeletons equipped with angular position sensors have been utilized to record joint kinematics during treadmill or overground walking [36, 42, 199–201].
The Lokomat and the LOPES have been used to measure hip and knee angles in various studies, where the reduced impedance of the joints allowed the subjects to impose their own gait pattern. The joints’ kinematics was evaluated mainly by comparing it with a reference angular trajectory: e.g. timing error within a tunnel around the desired spatial path  or spatial tracking error . A method to assess retraining in stroke patients based on the areal difference between a healthy reference and the patient’s trajectory during the swing phase was implemented in the ALEX gait trainer . However, at present, robotic gait trainers might not be the most suitable devices for performing an assessment equivalent to camera-based gait analysis, due to the influence that their mechanical constraints have on the gait pattern. Wearable and lightweight devices that do not hinder human movements are required for this purpose. Particularly suitable for this condition would be the soft lower limb exoskeletons (“exosuits”) that have been recently developed to improve human-robot interaction and to allow a more natural walking pattern [61, 204, 205]. In an active soft orthotic ankle device two IMUs placed on the shank and on the foot are used to compute the ankle joint angle . Alternatively, strain sensors embedded in the suit spanning over a joint are used . Although this is a promising approach for measurements in dynamic conditions, the sensing accuracy is at present not high enough for accurate measurements, due to relative movements between the suit and the skin of the subject. Moreover, sensor calibration is required every time a user wears the suit.
The robotic assistance required for walking has been proposed as an alternative method for assessing the walking function. For example, adaptive algorithms automatically adjust the support provided by the device based on the patient’s ability to follow a predefined trajectory or to perform a specific task (e.g. foot clearance) [200, 201, 206, 207]. The algorithms update a control parameter K (usually the impedance of the joints and the unloading of the body weight) at each walking step s based on a forgetting factor γ < 1 and on the weighted error g ⋅ e calculated in the previous step:
After a certain number of steps, the parameter K converges to a value that can be retained as a measure of the subject’s impairment . For example, an overall score can be obtained summing the torques required at each joint averaged during the last 10 steps .
Future developments in rehabilitation robotics
Robotic gait trainers and exoskeletons for overground assistance can be easily instrumented to provide kinematic and kinetic data that can be used to derive metrics useful for assessing the gait pattern and the walking function. Since these devices enable non-ambulatory patients to walk in a safe and functional manner, they allow the assessment of these category of subjects, limiting unwanted floor effects of the tests. Although these systems are expensive, they are already used in many clinical centers worldwide for providing gait training. In these contexts, subjects can be tested during gait training, requiring no or little additional time. Repeatable assessment procedures can be programmed in order to standardize the testing conditions (e.g., speed, unloading of body weight). Accurate measurements of the gait pattern can be obtained if the effects of the device dynamics (i.e. weight, inertia) are minimized. Moreover, the exoskeletons should have enough degrees of freedom to avoid constraining physiological walking movements. The compliant fixation of the patient’s leg to the orthosis could lead to measurement inaccuracy and errors , therefore standardized procedures need to be established in order to make the patient’s setup in the device as independent as possible of the operator. When the transparency of the device is guaranteed by hardware design (e.g. soft exoskeletons) or by software compensation [52, 53], the robot can be used for measuring joint kinematics or spatio-temporal gait parameters. When this condition is not met or when the subject is too impaired to be able to walk without the support of the device, other assessment methods must be used. It would be misleading, in fact, to measure standard gait parameters in a robotic gait trainer that affects the patient’s walking pattern. To address this problem, new outcome measures can be proposed. For example, the amount of support (i.e. joint impedance or unloading of the body weight) required to achieve a functional walking pattern can be used as an indicator of the subject’s impairment. Further studies in this direction are needed to establish the concurrent validity of this outcome measure with existing clinical scores. It can be hypothesized that a correlation with clinical scores that address the amount of support required for walking (e.g. WISCI II, FAC) exists. Moreover, if the algorithm adapts the support of the device to the particular needs of the single gait phases, it would be also possible to identify specific impairments localized within the gait cycle . However, the results of this method depend on the performance metric used. If a measure of the deviation from a reference trajectory is applied, the resulting support will depend also on the similarity between the prescribed trajectory and the patient’s individual gait pattern. A dead band around the reference trajectory, as in , could partially address this problem.
Discussion and conclusion
In this review we have discussed how novel robot-aided functional assessments can address the current issues related to the evaluation of the health-related status of a patient in clinical practice. Although essential for maximizing the individual therapy outcomes, the use of assessment methods in routine practice is at present insufficient. Among the reasons that contribute to this dearth, poor quality of the existing assessment scores and high administrative burden have been identified . In the different sections of this review we have highlighted additional issues in current clinical assessments specific for different lower limb functions. We have explained how robotic devices for rehabilitation have the potential to solve these issues by providing high quality assessments (i.e. objective, reliable and valid) and by integrating the assessment procedure in a training program. Based on the existing shortcomings and on the possibilities offered by robotic technologies, we have proposed solutions and recommendations for the development of novel robot-aided assessment tools. The quality of the assessment methods must be determined by studying their psychometric properties, as discussed in the section Assessments validation. We believe that the increasing use of robots for rehabilitation is not only beneficial for the therapy outcome, but also represents a huge opportunity for improving the assessment quality and increasing their frequency of administration. Indeed, robotic devices can be equipped with sensors for recording data useful for developing quantitative and objective assessment metrics. Secondly, robots can potentially assure the standardized execution of the assessment procedure, which is essential for reducing the measurement error and increasing the reliability. Moreover, robot-based assessments can reach higher inter-rater and intra-rater reliability if the robotic device is designed to limit fixation errors and to reduce inter-operator differences. Cuffs positioning, misalignments and different tightening of the fixation to the patient’s limbs may have a huge impact on the reliability of the assessment outcomes. User-friendly and ergonomic robotic device, along with a rigorous training of the operators may contribute to solve this problem.
A known issue of the current assessments used in clinical practice is their administrative burden (mainly time-wise) that limits the frequency at which they are administered. Assessments executed with rehabilitation robotic devices can be performed during the therapy session, measuring relevant parameters directly during the training, while the patient is using the device. Robotic assessments are able not only to complement existing clinical measurements, but also to enlarge the measurable range of an impairment: because of the quantifiable assistance that robots can provide, robotic assessments can be administered even if the patient is not able to perform the movement without support . Moreover, measurements that have been only subjectively addressed before (e.g., proprioception) can now be targeted by instrumented tests. New variables that were not readily accessible before (e.g., smoothness, joint coupling) become available. Further research on neurophysiological mechanisms must be encouraged to determine how these variables relate to sensorimotor functions and whether they can provide information on recovery . The increased sensitivity and the reduced measurement error of the robot-based assessments can be of utmost importance when the outcome measures are used in a clinical trial aimed at demonstrating the efficacy of a new therapy. Often little can be concluded because the effects of the therapy under study are masked by high inter-subject variability or they are not captured by conventional clinical assessments . Assessments able to distinguish the contribution of restoration of physiological patterns and the effect of compensatory mechanisms to the recovery will help to orient future therapeutic approaches . Not least, more sensitive measurements could also contribute to increasing the motivation of the patients, when even a slight improvement can be documented.
Before starting a research study aiming at developing a new assessment method, researchers must consider several issues. First of all, an inter-disciplinary approach involving research institutes, clinical facilities and medical device manufacturers is encouraged in all the phases of development: researchers must take into account the clinical relevance of the proposed measure (is the information provided by the measure useful for adjusting the therapy?), the interpretability of the outcome parameter (what is its physiological meaning?), the feasibility of the method for both its use in clinical practice (is it safe? Are its administrative and respondent burdens reasonable?) and the manufacturability and large-scale implementation. If these steps are missing, the risk is that the assessment method will never be routinely used in the clinical practice. On the other side, it is also important that researchers go “beyond” the limits of established clinical tests by developing new and independent standards based on robotic measurements. The final aim of robotic assessments, indeed, should not be to reproduce existing clinical scales that, even if widely accepted in the clinical practice, are not comparable by their nature to instrumented and robot-based assessments . Lastly, when developing assessment metrics of a same variable for different lower extremities devices, researchers should try to make the results independent of the platform on which they are obtained. In this way the same metric could be implemented in different devices and results from several studies could be compared. Even if the dynamics of the device will most likely influence some of the assessment metrics, comparative measures can be used (e.g. normalizing patient’s data against healthy normative data recorded in the same device).
A crucial requirement for the acceptance of a new assessment method in the rehabilitation community is its clinical validation: reliability must be assessed with an adequate sample size and validity should be established either by comparing the score with a gold-standard - if it exists - (concurrent validity) or by studying the relationship of the new assessment score with the underlying constructs of interest (construct validity). Guidelines for the validation of new robot-based assessments should be developed to help the researchers to define adequate clinical validation studies and to use the correct statistical tools. Moreover, it is necessary to develop indications for interpreting the different scoring systems: clinicians must be able to identify whether a change in score is clinically significant or it is due to measurement error .
We think that robotic assessments represent a challenging “green field” where researchers have the possibility – and the urgency – to develop methods that will have a strong impact on rehabilitation outcomes. Better assessments of lower extremities functions will allow the clinicians to prescribe therapeutic and rehabilitation plans that optimize the individual recovery while minimizing unnecessary effort and costs . We believe, therefore, that research for developing valid, reliable and responsive assessment methods is strongly needed for clinical practice, for studies on new therapies and, overall, for improving the rehabilitation outcome and decreasing the time of recovery.
10MWT, 10-m-walking-test; 1-RM, one repetition maximum; 6MWT, 6-min-walking-test; AB, able-bodied; CI, confidence interval; CP, cerebral palsy; CPM, continuous passive motion; DF, dorsiflexion; DGI, dynamic gait index; DOF, degree of freedom; EMG, electromyography; FAC, functional ambulation category; HHD, hand-held dynamometer; ICC, intra-class correlation coefficient; ICF, international classification of function, disability and health; JPR, joint position reproduction; LOA, limits of agreement; MAS, modified Ashworth scale; MCID, minimum clinically important difference; MDC, minimum detectable change; MMT, manual muscle test; MVC, maximum voluntary contraction; OA, osteoarthritis; PF, plantarflexion; ROM, range of motion; SD, standard deviation; SEM, standard error of measurement; TTDPM, time to detection of passive motion; TUG, timed-up-and-go; WISCI II, walking index for spinal cord injury (2nd version)
The terminology used in different publications is not always consistent. In the work of Kearny and colleagues  the intrinsic component includes the passive properties of the joint and the properties of the active muscle fibers.
Duncan EA, Murray J. The barriers and facilitators to routine outcome measurement by allied health professionals in practice: a systematic review. BMC Health Serv Res. 2012;12:96.
Jette DU, Halbert J, Iverson C, Miceli E, Shah P. Use of standardized outcome measures in physical therapist practice: perceptions and applications. Phys Ther. 2009;89:125–35.
Copeland J. Outcome measures: why physiotherapists must use them. Phys Ther Rev. 2009;14:367–8.
Van Hedel HJA. Improvement in function after spinal cord injury : the black-box entitled rehabilitation. Swiss Med Wkly. 2012;142:w13673.
Steeves JD, Lammertse D, Curt A, Fawcett JW, Tuszynski MH, Ditunno JF, Ellaway PH, Fehlings MG, Guest J, Kleitman N, Bartlett P, Blight A, Dietz V, Dobkin B, Grossman R, Short D, Nakamura M, Coleman W, Gaviria M, Privat A. Guidelines for the conduct of clinical trials for spinal cord injury (SCI) as developed by the ICCP panel: clinical trial outcome measures. Spinal Cord. 2007;45:206–21.
Lambercy O, Maggioni S, Lünenburger L, Gassert R, Bolliger M. Robotic and wearable sensor technologies for measurements/clinical assessments. In: Neurorehabilitation Technology 2nd edition. Edited by Dietz V, Reinkensmeyer DJ. Springer International; 2016.
Mehrholz J, Elsner B, Werner C, Kugler J, Pohl M. Electromechanical-assisted training for walking after stroke (Review). Cochrane Libr. 2013;44(10):e127.
Diaz I, Gil JJ, Sanchez E. Lower-Limb Robotic Rehabilitation : Literature Review and Challenges. J Robot. 2011.
Keller U, Schölch S, Albisser U, Rudhe C, Curt A, Riener R, Klamroth-Marganska V. Robot-assisted Arm assessments in spinal cord injured patients: a consideration of concept study. PLoS One. 2015;10:e0126948.
Bolliger M, Banz R, Dietz V, Lünenburger L. Standardized voluntary force measurement in a lower extremity rehabilitation robot. J Neuroeng Rehabil. 2008;5:23.
Tiffreau V, Ledoux I, Eymard B, Thévenon A, Hogrel J-Y. Isokinetic muscle testing for weak patients suffering from neuromuscular disorders: a reliability study. Neuromuscul Disord. 2007;17:524–31.
Ditunno PL, Patrick M, Stineman M, Ditunno JF. Who wants to walk? Preferences for recovery after SCI: a longitudinal and cross-sectional study. Spinal Cord. 2008;46:500–6.
Bohannon RW, Horton MG, Wikholm JB. Importance of four variables of walking to patients with stroke. Int J Rehabil Res. 1991;14:246–50.
Zhang M, Davies TC, Zhang Y, Xie S, Eng P. Reviewing effectiveness of ankle assessment techniques for use in robot-assisted therapy. J Rehabil Res Dev. 2014;51:517–34.
World Health Organization. Towards a Common Language for Functioning, Disability and Health ICF. 2002.
Raghavendra P, Bornman J, Granlund M, Björck-Akesson E. The world health Organization’s international classification of functioning, disability and health: implications for clinical and research practice in the field of augmentative and alternative communication. Augment Altern Commun. 2007;23:349–61.
Atkinson G, Nevill A. Statistical methods for assessing measurement error (reliability) in variables relevant to sports medicine. Sport Med. 1998;26:217–38.
Weir JP. Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J Strength Cond Res. 2005;19:231–40.
Altman DG, Bland JM. Measurement in medicine : the analysis of method comparison studies. Stat. 1983;32(July 1981):307–17.
Andresen EM. Criteria for assessing the tools of disability outcomes research. Arch Phys Med Rehabil. 2000;81:S15–20.
Roach KE. Measurement of health outcomes : reliability, validity and responsiveness. J Prosthetics Orthot. 2006;18:1–5.
Cook CE. Clinimetrics corner: the minimal clinically important change score (MCID): a necessary pretense. J Man Manip Ther. 2008;16:E82–3.
Clarkson H. Joint motion and function assessment: a research-based practical guide. Philadelphia: Lippincott Williams and Wilkins; 2000.
Rowe PJ, Myles CM, Walker C, Nutton R. Knee joint kinematics in gait and other functional activities measured using flexible electrogoniometry: how much knee motion is sufficient for normal daily life? Gait Posture. 2000;12:143–55.
Charbonnier C, Chagué S, Schmid J, Kolo FC, Bernardoni M, Christofilopoulos P. Analysis of hip range of motion in everyday life: a pilot study. Hip Int. 2015;25:82–90.
Anderson FC, Goldberg SR, Pandy MG, Delp SL. Contributions of muscle forces and toe-off kinematics to peak knee flexion during the swing phase of normal gait : an induced position analysis. J Biomech. 2004;37:731–7.
Ribbers G. Brain injury: long term outcome after traumatic brain injury. In: Stone J, Blouin M, editors. International Encyclopedia of Rehabilitation. 2010.
Pohl M, Mehrholz J, Rockstroh G, Rückriem S, Koch R. Contractures and involuntary muscle overactivity in severe brain injury. Brain Inj. 2007;21:421–32.
Brosseau L, Balmer S, Tousignant M, O’Sullivan JP, Goudreault C, Goudreault M, Gringras S. Intra- and intertester reliability and criterion validity of the parallelogram and universal goniometers for measuring maximum active knee flexion and extension of patients with knee restrictions. Arch Phys Med Rehabil. 2001;82:396–402.
Gajdosik RL, Bohannon RW. Clinical measurement of range of motion. Review of goniometry emphasizing reliability and validity. Phys Ther. 1987;67:1867–72.
Piriyaprasarth P, Morris ME. Psychometric properties of measurement tools for quantifying knee joint position and movement: a systematic review. Knee. 2007;14:2–8.
van Trijffel E, van de Pol RJ, Oostendorp RB, Lucas C. Inter-rater reliability for measurement of passive physiological movements in lower extremity joints is generally low: a systematic review. J Physiol Lond. 2010;56:223–35.
Dijkstra PU, de Bont LG, Van der Weele LT BG. Joint mobility measurements: reliability of a standardized method. Cranio J Craniomandib Pract. 1994;12:52–7.
Jung I-G, Yu I-Y, Kim S-Y, Lee D-K, Oh J-S. Reliability of ankle dorsiflexion passive range of motion measurements obtained using a hand-held goniometer and Biodex dynamometer in stroke patients. J Phys Ther Sci. 2015;27:1899–901.
Bok S-K, Lee TH, Lee SS. The effects of changes of ankle strength and range of motion according to aging on balance. Ann Rehabil Med. 2013;37:10–6.
Banala SK, Kim SH, Agrawal SK, Scholz JP. Robot assisted gait training with Active Leg Exoskeleton (ALEX). IEEE Trans Neural Syst Rehabil Eng. 2009;17:2–8.
Emken JL, Wynne JH, Harkema SJ, Reinkensmeyer DJ. Robotic device for manipulating human stepping. IEEE Trans Robot. 2006;22:185–9.
Farris RJ, Quintero H, Goldfarb M. Preliminary evaluation of a powered lower limb orthosis to aid walking in paraplegic individuals. IEEE Trans Neural Syst Rehabil Eng. 2011;19:652–9.
del-Ama AJ, Gil-Agudo A, Pons JL, Moreno JC. Hybrid FES-robot cooperative control of ambulatory gait rehabilitation exoskeleton. J Neuroeng Rehabil. 2014;11:27.
Esquenazi A, Talaty M, Packel A, Saulino M. The ReWalk powered exoskeleton to restore ambulatory function to individuals with thoracic-level motor-complete spinal cord injury. Am J Phys Med Rehabil. 2012;91:911–21.
Strausser KA, Swift TA, Zoss AB, Kazerooni H, Bennett BC. Mobile Exoskeleton for Spinal Cord Injury: Development and Testing. Arlington: ASME 2011 Dynamic Systems and Control Conference; 2011.
Bortole M, Venkatakrishnan A, Zhu F, Moreno JC, Francisco GE, Pons JL, Contreras-Vidal JL. The H2 robotic exoskeleton for gait rehabilitation after stroke: early findings from a clinical study. J Neuroeng Rehabil. 2015;12:54.
Bartenbach V, Wyss D, Seuret D, Riener R. A Lower Limb Exoskeleton Research Platform to Investigate Human-Robot Interaction. Singapore: IEEE International Conference on Rehabilitation Robotics; 2015.
Riener R, Lünenburger L, Maier IC, Colombo G, Dietz V. Locomotor training in subjects with sensori-motor deficits : an overview of the robotic gait orthosis lokomat. J Healthc Eng. 2010;1:197–216.
Peng Q, Park H-S, Shah P, Wilson N, Ren Y, Wu Y-N, Liu J, Gaebler-Spira DJ, Zhang L-Q. Quantitative evaluations of ankle spasticity and stiffness in neurological disorders using manual spasticity evaluator. J Rehabil Res Dev. 2011;48:473–81.
Waldman G, Yang C-Y, Ren Y, Liu L, Guo X, Harvey RL, Roth EJ, Zhang L-Q. Effects of robot-guided passive stretching and active movement training of ankle and mobility impairments in stroke. NeuroRehabilitation. 2013;32:625–34.
Zhang L-Q, Chung SG, Bai Z, Xu D, Rey EMT V, Rogers MW, Johnson ME, Roth EJ. Intelligent stretching of ankle joints with contracture/spasticity. IEEE Trans Neural Syst Rehabil Eng. 2002;10:149–57.
Giacomozzi C, Cesinaro S, Basile F, De Angelis G, Giansanti D, Maccioni G, Masci E, Panella a, Paolizzi M, Torre M, Valentini P, Macellari V. Measurement device for ankle joint kinematic and dynamic characterisation. Med Biol Eng Comput. 2003;41:486–93.
Roy A, Krebs HI, Williams DJ, Bever CT, Forrester LW, Macko RM, Hogan N. Robot-aided neurorehabilitation: a novel robot for ankle rehabilitation. IEEE Trans Robot. 2009;25:569–82.
Borsa PA, Sauers EL, Herling DE, Manzour WF. In vivo quantification of capsular End- point in the nonimpaired glenohumeral joint using an instrumented measurement system. J Orthop Sports Phys Ther. 2001;31:419–31.
McQuade K, Price R, Liu N, Ciol MA. Objective assessment of joint stiffness: a clinically oriented hardware and software device with an application to the shoulder joint. J Nov Physiother. 2012;2:7.
Jarrassé N, Proietti T, Crocher V, Robertson J, Sahbani A, Morel G, Roby-Brami A. Robotic exoskeletons: a perspective for the rehabilitation of Arm coordination in stroke patients. Front Hum Neurosci. 2014;8(December):1–13.
van Dijk W, van der Kooij H, Koopman B, van Asseldonk EHF. Improving the transparency of a rehabilitation robot by exploiting the cyclic behaviour of walking. IEEE Int Conf Rehabil Robot. 2013;2013:6650393.
Weiselfish-Giammatteo S, Giammatteo T. Integrative Manual Therapy for Biomechanics: Application of Muscle Energy and “Beyond” Technique : Treatment of the Spine, Ribs, and Extremities. North Atlantic Books; 2003. [Integrated Manual Therapy Series].
Veneman JF, Kruidhof R, Hekman EEG, Ekkelenkamp R, Asseldonk EHF V. Design and evaluation of the LOPES exoskeleton robot for interactive gait rehabilitation. IEEE Trans Neural Syst Rehabil Eng. 2007;15:379–86.
Colombo G, Wirz M, Dietz V. Driven gait orthosis for improvement of locomotor training in paraplegic patients. Spinal Cord. 2001;39:252–5.
Rowe P, Myles C, Hillmann S, Hazlewood M. Validation of flexible electrogoniometry as a measure of joint kinematics. Physiotherapy. 2001;87:479–88.
Tesio L, Monzani M, Gatti R, Franchignoni F. Flexible electrogoniometers: kinesiological advantages with respect to potentiometric goniometers. Clin Biomech (Bristol, Avon). 1995;10:2–4.
Bronner S, Agraharasamakulam S, Ojofeitimi S. Reliability and validity of electrogoniometry measurement of lower extremity movement. J Med Eng Technol. 2010;34:232–42.
Piriyaprasarth P, Morris ME, Winter A, Bialocerkowski AE. The reliability of knee joint position testing using electrogoniometry. BMC Musculoskelet Disord. 2008;9:6.
Menguc Y, Park YL, Martinez-Villalpando E, Aubin P, Zisook M, Stirling L, Wood RJ, Walsh CJ. Soft wearable motion sensing suit for lower limb biomechanics measurements. Proc - IEEE Int Conf Robot Autom. 2013;5309–16.
Donno M, Palange E, Nicola F Di, Member S, Bucci G, Ciancetta F. A New Flexible Optical Fiber Goniometer for Dynamic Angular Measurements : Application to Human Joint Movement Monitoring. IEEE Trans. Instrum. Meas. 2008;57:1614–20.
Favre J, Jolles BM, Aissaoui R, Aminian K. Ambulatory measurement of 3D knee joint angle. J Biomech. 2008;41:1029–35.
Luinge HJ, Veltink PH. Inclination measurement of human movement using a 3-D accelerometer with autocalibration. IEEE Trans Neural Syst Rehabil Eng. 2004;12:112–21.
Dejnabadi H, Jolles BM, Aminian K. A new approach to accurate measurement of uniaxial joint angles based on a combination of accelerometers and gyroscopes. IEEE Trans Biomed Eng. 2005;52:1478–84.
Djurić-Jovičić MD, Jovičić NS, Popović DB, Djordjević AR. Nonlinear optimization for drift removal in estimation of gait kinematics based on accelerometers. J Biomech. 2012;45:2849–54.
MeSH - Muscle Strength. [http://www.ncbi.nlm.nih.gov/mesh/68053580]. Accessed 26 July 2016.
Neckel N, Pelliccio M, Nichols D, Hidler J. Quantification of functional weakness and abnormal synergy patterns in the lower limb of individuals with chronic stroke. J Neuroeng Rehabil. 2006;3:17.
Wirz M, van Hedel HJ, Rupp R, Curt A, Dietz V. Muscle force and gait performance: relationships after spinal cord injury. Arch Phys Med Rehabil. 2006;87:1218–22.
Kim C, Eng J. The relationship of lower-extremity muscle torque to locomotor performance in people with stroke. Phys Ther. 2003;83:49–57.
Cho KH, Bok SK, Kim YJ, Hwang SL. Effect of lower limb strength on falls and balance of the elderly. Ann Rehabil Med. 2012;36:386–93.
Newham DJ, Hsiao SF. Knee muscle isometric strength, voluntary activation and antagonist co-contraction in the first six months after stroke. Disabil Rehabil. 2001;23:379.
Noreau L, Vachon J. Comparison of three methods to assess muscular strength in individuals with spinal cord injury. Spinal Cord. 1998;36:716–23.
Paternostro-Sluga T, Grim-Stieger M, Posch M, Schuhfried O, Vacariu G, Mittermaier C, Bittner C, Fialka-Moser V. Reliability and validity of the Medical Research Council (MRC) scale and a modified scale for testing muscle strength in patients with radial palsy. J Rehabil Med. 2008;40:665–71.
Beasley W. Quantitative muscle testing: principles and applications to research and clinical services. Arch Phys Med Rehabil. 1961;42:398–425.
Herbison GJ, Isaac Z, Cohen ME, Ditunno JF. Strength post-spinal cord injury: myometer vs manual muscle test. Spinal Cord. 1996;34:543–8.
Bohannon RW. Manual muscle testing: does it meet the standards of an adequate screening test? Clin Rehabil. 2005;19:662–7.
Merlini L, Mazzone ES, Solari A, Morandi L. Reliability of hand-held dynamometry in spinal muscular atrophy. Muscle Nerve. 2002;26:64–70.
Stark T, Walker B, Phillips JK, Fejer R, Beck R. Hand-held dynamometry correlation with the gold standard isokinetic dynamometry: a systematic review. PM R. 2011;3:472–9.
Marmon AR, Pozzi F, Alnahdi AH, Zeni J. The validity of plantarflexor strength measures obtained through hand-held dynamometry measurements of force. Int J Sports Phys Ther. 2013;8:820–7.
Kim WK, Kim DK, Seo KM, Kang SH. Reliability and validity of isometric knee extensor strength test with hand-held dynamometer depending on its fixation: a pilot study. Ann Rehabil Med. 2014;38:84–93.
Meldrum D, Cahalane E, Keogan F, Hardiman O. Maximum voluntary isometric contraction: investigation of reliability and learning effect. Amyotroph Lateral Scler Other Motor Neuron Disord. 2003;4:36–44.
Colombo R, Mazzini L, Mora G, Parenzan R, Creola G, Pirali I, Minuco G. Measurement of isometric muscle strength: a reproducibility study of maximal voluntary contraction in normal subjects and amyotrophic lateral sclerosis patients. Med Eng Phys. 2000;22:167–74.
Lauermann SP, Lienhard K, Item-Glatthorn JF, Casartelli NC, Maffiuletti N. Assessment of quadriceps muscle weakness in patients after total knee arthroplasty and total hip arthroplasty: methodological issues. J Electromyogr Kinesiol. 2014;24:285–91.
Lienhard K, Lauermann SP, Schneider D, Item-Glatthorn JF, Casartelli NC, Maffiuletti N. Validity and reliability of isometric, isokinetic and isoinertial modalities for the assessment of quadriceps muscle strength in patients with total knee arthroplasty. J Electromyogr Kinesiol. 2013;23:1283–8.
Lum PS, Patten C, Kothari D, Yap R. Effects of velocity on maximal torque production in poststroke hemiparesis. Muscle Nerve. 2004;30:732–42.
Knapik JJ, Wright JE, Mawdsley RH, Braun J. Isometric, isotonic, and isokinetic torque variations in four muscle groups through a range of joint motion. Phys Ther. 1983;63:938–47.
Whiteley R, Jacobsen P, Prior S, Skazalski C, Otten R, Johnson A. Correlation of isokinetic and novel hand-held dynamometry measures of knee flexion and extension strength testing. J Sci Med Sport. 2012;15:444–50.
Van Campen A, De Groote F, Jonkers I, De Schutter J. An extended dynamometer setup to improve the accuracy of knee joint moment assessment. IEEE Trans Biomed Eng. 2013;60:1202–8.
Clark DJ, Condliffe EG, Patten C. Reliability of concentric and eccentric torque during isokinetic knee extension in post-stroke hemiparesis. Clin Biomech (Bristol, Avon). 2006;21:395–404.
Hidler J. Robotic-assessment of walking in individuals with gait disorders. Conf Proc Annu Int Conf IEEE Eng Med Biol Soc IEEE Eng Med Biol Soc Annu Conf. 2004;7:4829–31.
Neckel ND, Blonien N, Nichols D, Hidler J. Abnormal joint torque patterns exhibited by chronic stroke subjects while walking with a prescribed physiological gait pattern. J Neuroeng Rehabil. 2008;5:19.
Veneman JF, Kruidhof R, Hekman EEG, Ekkelenkamp R, Van Asseldonk EHF, Van Der Kooij H. Design and evaluation of the LOPES exoskeleton robot for interactive gait rehabilitation. Neural Syst Rehabil Eng IEEE Trans. 2007;15:379–86.
Forrester LW, Roy A, Goodman RN, Rietschel J, Barton JE, Krebs HI, Macko RF. Clinical application of a modular ankle robot for stroke rehabilitation. NeuroRehabilitation. 2013;33:85–97.
Bohannon RW. Knee extension strength and body weight determine sit-to-stand independence after stroke. Physiother Theory Pract. 2007;23:291–7.
Bohannon RW. Relevance of Muscle Strength to Gait Performance in Patients with Neurologic Disability. Neurorehabil Neural Repair. 1989.
Nadler SF, DePrince ML, Hauesien N, Malanga G, Stitik TP, Price E. Portable dynamometer anchoring station for measuring strength of the hip extensors and abductors. Arch Phys Med Rehabil. 2000;81:1072–6.
Drouin JM, Valovich-mcLeod TC, Shultz SJ, Gansneder BM, Perrin DH. Reliability and validity of the Biodex system 3 pro isokinetic dynamometer velocity, torque and position measurements. Eur J Appl Physiol. 2004;91:22–9.
Rothstein JM, Lamb RL, Mayhew TP. Clinical uses of isokinetic measurements. Critical issues Phys Ther. 1987;67:1840–4.
Han J, Waddington G, Adams R, Anson J, Liu Y. Assessing proprioception: A critical review of methods. J Sport Heal Sci. 2015.
Suetterlin KJ, Sayer AA. Proprioception: where are we now? A commentary on clinical assessment, changes across the life course, functional implications and future interventions. Age Ageing. 2013;43:1–6.
Goble DJ. Proprioceptive acuity assessment via joint position matching: from basic science to general practice. Phys Ther. 2010;90:1176–84.
Proske U, Gandevia SC. The proprioceptive senses: their roles in signaling body shape, body position and movement, and muscle force. Physiol Rev. 2012;92:1651–97.
Cammarata ML, Dhaher YY. Proprioceptive acuity in the frontal and sagittal planes of the knee: a preliminary study. Eur J Appl Physiol. 2011;111:1313–20.
Allet L, Kim H, Ashton-Miller J, De Mott T, Richardson JK. Frontal plane hip and ankle sensorimotor function, not age, predicts unipedal stance time. Muscle Nerve. 2012;45:578–85.
Domingo A, Lam T. Reliability and validity of using the Lokomat to assess lower limb joint position sense in people with incomplete spinal cord injury. J Neuroeng Rehabil. 2014;11:167.
Elangovan N, Herrmann A, Konczak J. Assessing proprioceptive function: evaluating joint position matching methods against psychophysical thresholds. Phys Ther. 2014;94:553–61.
Clark NC, Röijezon U, Treleaven J. Proprioception in musculoskeletal rehabilitation. Part 2: clinical assessment and intervention. Man Ther. 2015;20:378–87.
Sanford J, Moreland J, Swanson LR, Stratford PW, Gowland C. Reliability of the Fugl-Meyer assessment for testing motor performance in patients following stroke. Phys Ther. 1993;73:447–54.
Hillier S, Immink M, Thewlis D: Assessing Proprioception: A Systematic Review of Possibilities. Neurorehabil Neural Repair 2015;29:933–49.
Refshauge KM, Taylor JL, Mccloskey DI, Gianoutsos M, Mathews P, Fitzpatrick RC. Movement detection at the human big toe. J Physiol. 1998;513:307–14.
Refshauge KM, Chan R, Taylor JL, Mccloskey DI. Detection of movements imposed on human hip, knee, ankle and toe joints. J Physiol. 1995;488:231–41.
Hurkmans EJ, van der Esch M, Ostelo RWJG, Knol D, Dekker J, Steultjens MPM. Reproducibility of the measurement of knee joint proprioception in patients with osteoarthritis of the knee. Arthritis Rheum. 2007;57:1398–403.
Chisholm AE, Domingo A, Jeyasurya J, Lam T. Quantification of lower extremity kinesthesia deficits using a robotic exoskeleton in people with a spinal cord injury. Neurorehabil Neural Repair. 2015;30:199–208.
Brunnstrom S. Movement therapy in hemiplegia—a neurophysiological approach. New York: Harper & Row Publishers, Inc.; 1970.
Fugl-Meyer AR, Jääskö L, Leyman I, Olsson S, Steglind S. The post-stroke hemiplegic patient. 1. a method for evaluation of physical performance. Scand J Rehabil Med. 1975;7:13–31.
Duncan PW, Propst M, Nelson SG. Reliability of the fugl-meyer assessment of sensorimotor recovery following cerebrovascular accident. Phys Ther. 1983;63:1606–10.
Hsueh I-P, Hsu M-J, Sheu C-F, Lee S, Hsieh C-L, Lin J-H. Psychometric comparisons of 2 versions of the fugl-meyer motor scale and 2 versions of the stroke rehabilitation assessment of movement. Neurorehabil Neural Repair. 2008;22:737–44.
Dettmann MA, Linder MT, Sepic SB. Relationships among walking performance, postural stability, and functional assessments of the hemiplegic patient. Am J Phys Med. 1987;66:77–90.
Tan AQ, Dhaher YY. Evaluation of lower limb cross planar kinetic connectivity signatures post-stroke. J Biomech. 2014;47:1–8.
Cruz TH, Dhaher Y. Evidence of abnormal lower-limb torque coupling after stroke: an isometric study. Stroke. 2008;39:139–47.
Krishnan C, Dhaher Y. Corticospinal responses of quadriceps are abnormally coupled with hip adductors in chronic stroke survivors. Exp Neurol. 2012;233:400–7.
Thelen DD, Riewald S, Asakawa DS, Sanger TD, Delp SL. Abnormal coupling of knee and hip moments during maximal exertions in persons with cerebral palsy. Muscle Nerve. 2003;27:486–93.
Hidler JM, Carroll M, Federovich EH. Strength and coordination in the paretic Leg of individuals following acute stroke. J Biomech. 2007;15:526–34.
Sanchez N, Dewald JPA. Constraints imposed by the lower extremity extensor synergy in chronic hemiparetic stroke: Preliminary findings. Conf Proc Annu Int Conf IEEE Eng Med Biol Soc IEEE Eng Med Biol Soc Conf. 2014;2014:5804–7.
Lunardini F, Casellato C, D’Avella A, Sanger T, Pedrocchi A. Robustness and reliability of synergy-based myocontrol of a multiple degree of freedom robotic Arm. IEEE Trans Neural Syst Rehabil Eng. 2015.
Sanchez N, Acosta A, Stienen A, Dewald J: A Multiple Degree of Freedom Lower Extremity Isometric Device to Simultaneously Quantify Hip, Knee and Ankle Torques. IEEE Trans Neural Syst Rehabil Eng 2015;23:765–75.
Cruz TH, Lewek MD, Dhaher YY. Biomechanical impairments and gait adaptations post-stroke: multi-factorial associations. J Biomech. 2009;42:1673–7.
Dewald JP, Beer R. Abnormal joint torque patterns in the paretic upper limb of subjects with hemiparesis. Muscle Nerve. 2001;24(2):273–83.
Ellis MD, Sukal T, DeMott T, Dewald JPA. Augmenting clinical evaluation of hemiparetic arm movement with a laboratory-based quantitative measurement of kinematics as a function of limb loading. Neurorehabil Neural Repair. 2008;22:321–9.
Latash M, Zatsiorsky VM. Biomechanics and Motor Control: Defining Central Concepts. Academic Press; 2015.
Hogan N. Mechanical impedance of single- and multi-articular systems. In: Winters JM, Woo S-Y, editors. Multiple Muscle Systems. New York: Springer; 1990. p. 149–64.
Latash ML, Zatsiorsky VM. Joint stiffness: Myth or reality? Hum Mov Sci. 1993;12(6):653–92.
Ludvig D, Perreault EJ. Estimation of joint impedance using short data segments. Conf Proc Annu Int Conf IEEE Eng Med Biol Soc IEEE Eng Med Biol Soc Conf. 2011;2011:4120–3.
Kearney RE, Hunter IW. System identification of human joint dynamics. Crit Rev Biomed Eng. 1990;18:55–87.
Mirbagheri MM, Barbeau H, Kearney RE. Intrinsic and reflex contributions to human ankle stiffness: variation with activation level and position. Exp Brain Res. 2000;135:423.
Sinkjaer T, Magnussen I. Passive, intrinsic and reflex-mediated stiffness in the ankle extensors of hemiparetic patients. Brain. 1994;117:355–63.
Chung SG, Van Rey E, Bai Z, Roth EJ, Zhang L-Q. Biomechanic changes in passive properties of hemiplegic ankles with spastic hypertonia. Arch Phys Med Rehabil. 2004;85:1638–46.
Mirbagheri MM, Barbeau H, Ladouceur M, Kearney RE. Intrinsic and reflex stiffness in normal and spastic, spinal cord injured subjects. Exp Brain Res. 2001;141:446.
Hunter I, Kearney R. Dynamics of human ankle stiffness: variation with mean ankle torque. J Biomech. 1982;15:747–52.
Gottlieb GL, Agarwal GC. Response to sudden torques about ankle in man: myotatic reflex. J Neurophysiol. 1979;42(1 Pt 1):91–106.
Weiss P, Kearney R, Hunter I. Position dependence of ankle joint dynamics-II. Active mechanics. J Biomech. 1986;19:737–51.
Weiss P, Kearney R, Hunter I. Position dependence of ankle joint dynamics-I. Passive mechanics. J Biomech. 1986;19:727–35.
Kearney R, Hunter I. Dynamics of human ankle stiffness: variation with displacement amplitude. J Biomech. 1982;15:753–6.
de Vlugt E, van Eesbeek S, Baines P, Hilte J, Meskers CGM, de Groot JH. Short range stiffness elastic limit depends on joint velocity. J Biomech. 2011;44:2106–12.
Zhang LQ, Nuber G, Butler J, Bowen M, Rymer WZ. In vivo human knee joint dynamic properties as functions of muscle contraction and joint position. J Biomech. 1998;31:71–6.
Kearney RE, Hunter IW. Nonlinear identification of stretch reflex dynamics. Ann Biomed Eng. 1988;16:79–94.
Powers RK, Marder-Meyer J, Rymer WZ. Quantitative relations between hypertonia and stretch reflex threshold in spastic hemiparesis. Ann Neurol. 1988;23:115–24.
Mugge W, Abbink DA, van der Helm FCT. Reduced power method: how to evoke low-bandwidth behaviour while estimating full-bandwidth dynamics. 2007. p. 575–81.
Bohannon RW, Smith MB. Interrater reliability of a modified ashworth scale of muscle spasticity. Phys Ther. 1987;67:206–7.
Blackburn M, van Vliet P, Mockett SP. Reliability of measurements obtained with the modified Ashworth scale in the lower extremities of people with stroke. Phys Ther. 2002;82:25–34.
Wartenberg R. Pendulousness of the legs as a diagnostic test. Neurology. 1951;1:18.
Bohannon RW, Harrison S, Kinsella-Shaw J. Reliability and validity of pendulum test measures of spasticity obtained with the Polhemus tracking system from patients with chronic stroke. J Neuroeng Rehabil. 2009;6:30.
Fowler EG, Nwigwe AI, Ho TW. Sensitivity of the pendulum test for assessing spasticity in persons with cerebral palsy. Dev Med Child Neurol. 2000;42:182–9.
Kearney E, Weiss L, Morier R. System identification of human ankle dynamics: Intersubject variability and intrasubject reliability. Clin Biomech. 1990;5:205–17.
Wilken J, Rao S, Estin M, Saltzman CL, Yack HJ. A new device for assessing ankle dorsiflexion motion: reliability and validity. J Orthop Sports Phys Ther. 2011;41:274–80.
Lünenburger L, Colombo G, Riener R, Dietz V. Clinical assessments performed during robotic rehabilitation by the gait training robot Lokomat. In: Proceedings of the 2005 IEEE 9th International Conference on Rehabilitation Robotics. 2005. p. 345–8.
Chesworth BM, Vandervoort BM. Reliability of a torque motor system for measurement of passive ankle joint stiffness in control subjects. Physiother Canada. 1988;40:300–3.
Franzoi AC, Castro C, Cardone C. Isokinetic assessment of spasticity in subjects with traumatic spinal cord injury (ASIA A). Spinal Cord. 1999;37:416–20.
McHugh MP, Hogan DE. Effect of knee flexion angle on active joint stiffness. Acta Physiol Scand. 2004;180:249–54.
Blackburn JT, Padua DA, Riemann BL, Guskiewicz KM. The relationships between active extensibility, and passive and active stiffness of the knee flexors. J Electromyogr Kinesiol. 2004;14:683–91.
de Vlugt E, de Groot JH, Schenkeveld KE, Arendzen JH, van der Helm FCT, Meskers CGM. The relation between neuromechanical parameters and Ashworth score in stroke patients. J Neuroeng Rehabil. 2010;7:35.
Androwis GJ, Michael PA, Strongwater A, Foulds RA. Estimation of intrinsic joint impedance using quasi-static passive and dynamic methods in individuals with and without Cerebral Palsy. In: IEEE Engineering in Medicine and Biology Society. Annual Conference. 2014. p. 4403–6.
Mirbagheri MM, Kearney RE, Barbeau H. Quantitative, objective measurement of ankle dynamic stiffness: Intrasubject reliability and intersubject variability. In: Annual International Conference of the IEEE Engineering in Medicine and Biology - Proceedings. 1996. p. 585–6.
Sloot LH, van der Krogt MM, de Gooijer-van de Groep KL, van Eesbeek S, de Groot J, Buizer AI, Meskers C, Becher JG, de Vlugt E, Harlaar J. The validity and reliability of modelled neural and tissue properties of the ankle muscles in children with cerebral palsy. Gait Posture. 2015;42:7–15.
Loram I, Lakie M, Maganaris C. Reply from Ian D. Loram, Constantinos N. Maganaris and Martin Lakie. J Physiol. 2005;569:706.
Vlutters M, Boonstra TA, Schouten AC, Van der Kooij H. Direct measurement of the intrinsic ankle stiffness during standing. J Biomech. 2015;48:1258–63.
Koopman B, van Asseldonk EHF, Van Der Kooij H. In vivo measurement of human knee and hip dynamics using MIMO system identification. Buenos Aires: Proceedings of 32nd Annual Internation conference of the IEEE EMBS; 2011. p. 3426–9.
Pfeifer S, Vallery H, Hardegger M, Riener R, Perreault EJ. Model-based estimation of knee stiffness. IEEE Trans Biomed Eng. 2012;59:2604–12.
Ludvig D, Pfeifer S, Hu X, Perreault EJ. Time-varying system identification for understanding the control of human knee impedance. In: IFAC Proceedings Volumes (IFAC-PapersOnline). 2012. p. 1306–10.
Ludvig D, Starret Visser T, Giesbrecht H, Kearney R. Identification of time-varying intrinsic and reflex joint stiffness. IEEE Trans Biomed Eng. 2011;58:1.
Ludvig D, Perreault EJ. System identification of physiological systems using short data segments. IEEE Trans Biomed Eng. 2012;59:3541–9.
Tucker MR, Moser A, Lambercy O, Sulzer J, Gassert R. Design of a wearable perturbator for human knee impedance estimation during gait. IEEE Int Conf Rehabil Robot 2013. 2013;2013:6650372.
Perry J. Gait Analysis. Normal and Pathological Function. Thorofare: SLACK Incorporated; 1992.
Keane-Miller, O’Toole MT. Miller-Keane Encyclopedia & Dictionary of Medicine, Nursing & Allied Health. 7th ed. 2005.
Winter DA. Biomechanics and Motor Control of Human Gait: Normal, Elderly and Pathological. 2nd edition. Waterloo Biomechanics; 1991.
Baker R: Measuring Walking: A Handbook of Clinical Gait Analysis. 1st edition. London: Mac Keith Press; 2013.
Awai L, Curt A. Intralimb coordination as a sensitive indicator of motor-control impairment after spinal cord injury. Front Hum Neurosci. 2014;8(March):148.
Hedel HJA V, Wirz M, Curt A. Improving walking assessment in subjects with an incomplete spinal cord injury : responsiveness. 2006. p. 352–6.
van Hedel HJ, Wirz M, Dietz V. Standardized assessment of walking capacity after spinal cord injury: the European network approach. Neurol Res. 2008;30:61–73.
Van Hedel HJ, Wirz M, Dietz V. Assessing Walking Ability in Subjects With Spinal Cord Injury : Validity and Reliability of 3 Walking Tests. Arch Phys Med Rehabil. 2005;86:190–6.
Jackson AB, Carnel CT, Ditunno JF, Read MS, Boninger ML, Schmeler MR, Apt OTRL, Williams SR, Donovan WH. Outcome measures for gait and ambulation in the spinal cord injury population. J Spinal Cord Med. 2008;31:487–99.
Schmid A, Duncan PW, Studenski S, Lai SM, Richards L, Perera S, Wu SS. Improvements in speed-based gait classifications are meaningful. Stroke. 2007;38:2096–100.
Awai L, Bolliger M, Ferguson AR, Courtine G, Curt A. Influence of spinal cord integrity on gait control in human spinal cord injury. Neurorehabil Neural Repair. 2015.
Jasiewicz JM, Allum JHJ, Middleton JW, Barriskill A, Condie P, Purcell B, Li RCT. Gait event detection using linear accelerometers or angular velocity transducers in able-bodied and spinal-cord injured individuals. Gait Posture. 2006;24:502–9.
Rueterbories J, Spaich EG, Larsen B, Andersen OK. Methods for gait event detection and analysis in ambulatory systems. Med Eng Phys. 2010;32:545–52.
Aminian K, Najafi B, Büla C, Leyvraz P-F, Robert P. Spatio-temporal parameters of gait measured by an ambulatory system using miniature gyroscopes. J Biomech. 2002;35:689–99.
Bilney B, Morris M, Webster K. Concurrent related validity of the GAITRite walkway system for quantification of the spatial and temporal parameters of gait. Gait Posture. 2003;17:68–74.
Micó-Amigo ME, Kingma I, Ainsworth E, Walgaard S, Niessen M, van Lummel RC, van Dieën JH. A novel accelerometry-based algorithm for the detection of step durations over short episodes of gait in healthy elderly. J Neuroeng Rehabil. 2016;13:38.
Boutaayamou M, Schwartz C, Stamatakis J, Denoël V, Maquet D, Forthomme B, Croisier JL, Macq B, Verly JG, Garraux G, Brüls O. Development and validation of an accelerometer-based method for quantifying gait events. Med Eng Phys. 2015;37:226–32.
Mariani B, Rouhani H, Crevoisier X, Aminian K. Quantitative estimation of foot-flat and stance phase of gait using foot-worn inertial sensors. Gait Posture. 2013;37:229–34.
Hausdorff JM. Gait variability: methods, modeling and meaning. J Neuroeng Rehabil. 2005;9:1–9.
Allen JL, Kautz S, Neptune RR. Step length asymmetry is representative of compensatory mechanisms used in post-stroke hemiparetic walking. Gait Posture. 2011;33:538–43.
Chen G, Patten C, Kothari DH, Zajac FE. Gait differences between individuals with post-stroke hemiparesis and non-disabled controls at matched speeds. Gait Posture. 2005;22:51–6.
Olney SJ, Richards C. Hemiparetic gait following stroke.Part I : Characteristics. Gait Posture. 1996;4:136–48.
McRoberts. [http://www.mcroberts.nl/]. Accessed 26 July 2016.
GaitUp. [http://www.gaitup.com/]. Accessed 26 July 2016.
Awai L, Curt A. Intralimb coordination as a sensitive indicator of motor-control impairment after spinal cord injury. Front Hum Neurosci. 2014;8(March):1–8.
Hidler J, Wisman W, Neckel N. Kinematic trajectories while walking within the Lokomat robotic gait-orthosis. Clin Biomech (Bristol, Avon). 2008;23:1251–9.
Koopman B, van Asseldonk EHF, van der Kooij H. Selective control of gait subtasks in robotic gait training: foot clearance support in stroke survivors with a powered exoskeleton. J Neuroeng Rehabil. 2013;10:3.
Emken JL, Harkema SJ, Beres-jones JA, Ferreira CK, Reinkensmeyer DJ. Feasibility of manual teach-and-replay and continuous impedance shaping for robotic locomotor training following spinal cord injury. IEEE Trans Biomed Eng. 2008;55:322–34.
Duschau-Wicke A, von Zitzewitz J, Caprez A, Luenenburger L, Riener R. Path control: a method for patient-cooperative robot-aided gait rehabilitation. IEEE Trans neural Syst Rehabil Eng. 2010;18:38–48.
Krishnan C, Ranganathan R, Dhaher YY, Rymer WZ. A pilot study on the feasibility of robot-aided Leg motor training to facilitate active participation. PLoS One. 2013;8:e77370.
Park YL, Chen BR, Young D, Stirling L, Wood RJ, Goldfield E, Nagpal R. Bio-inspired active soft orthotic device for ankle foot pathologies. IEEE Int Conf Intell Robot Syst. 2011;4488–95.
Bartenbach V, Schmidt K, Naef M, Wyss D, Riener R. Concept of a Soft Exosuit for the Support of Leg Function in Rehabilitation. Singapore: IEEE International Conference on Rehabilitation Robotics; 2015.
Maggioni S, Lunenburger L, Riener R, Melendez-Calderon A. Robot-aided assessment of walking function based on an adaptive algorithm. In: Rehabilitation Robotics (ICORR), 2015 IEEE International Conference on. 2015. p. 804–9.
Duschau-Wicke A, Felsenstein S, Riener R. Adaptive body weight support controls human activity during robot-aided gait training. Kyoto: 2009 IEEE 11th International Conference on Rehabilitation Robotics; 2009. p. 413–8.
Wolbrecht ET, Chan V, Reinkensmeyer DJ, Bobrow JE. Optimizing compliant, model-based robotic assistance to promote neurorehabilitation. IEEE Trans Neural Syst Rehabil Eng. 2008;16:286–97.
Neckel ND, Blonien N, Nichols D, Hidler J. Abnormal joint torque patterns exhibited by chronic stroke subjects while walking with a prescribed physiological gait pattern. J Neuroeng Rehabil. 2008;13:1–13.
Niu X, Varoqui D, Kindig M, Mirbagheri MM. Prediction of gait recovery in spinal cord injured individuals trained with robotic gait orthosis. J Neuroeng Rehabil. 2014;11:1–9.
ICF Browser. [http://apps.who.int/classifications/icfbrowser/].
Hallgren KA. Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol. 2012;8:23–34.
Shrout PE, Fleiss JL. Intraclass correlations : uses in assessing rater reliability. Psychon Bull. 1979;86:420–8.
Shrout PE. Measurement reliability and agreement in psychiatry. Stat Methods Med Res. 1998;7:301–17.
McHugh ML. Interrater reliability: the kappa statistic. Biochemia Medica. 2012;276–282.
Sivan M, Connor RJO, Makower S, Hons BA, Phys D, Levesley M, Bhakta B. Systematic review of outcome measures used in the evaluation of robot-assisted upper limb exercise in stroke. J Rehabil Med. 2011;43:181–9.
Husted JA, Cook RJ, Farewell VT, Gladman DD. Methods for assessing responsiveness : a critical review and recommendations. J. Clin. Epidemiol. 2000;53:459–68.
Wright A, Hannon J, Hegedus EJ, Kavchak AE. Clinimetrics corner : a closer look at the minimal clinically important difference (MCID). J Man Manip Ther. 2012;20:160–6.
Gogia PP, Braatz JH, Rose SJ, Norton BJ. Reliability and validity of goniometric measurements at the knee. Phys Ther. 1987;67:192–5.
Currier LL, Froehlich PJ, Carow SD, McAndrew RK, Cliborne AV, Boyles RE, Mansfield LT, Wainner RS. Development of a clinical prediction rule to identify patients with knee pain and clinical evidence of knee osteoarthritis Who demonstrate a favorable short-term response to Hip mobilization. Phys Ther. 2007;87:1106–19.
Poulsen E, Christensen HW, Penny JØ, Overgaard S, Vach W, Hartvigsen J. Reproducibility of range of motion and muscle strength measurements in patients with hip osteoarthritis - an inter-rater study. BMC Musculoskelet Disord. 2012;13:242.
Elveru RA, Rothstein JM, Lamb RL. Goniometric reliability in a clinical setting. Phys Ther. 1988;68:672.
Watkins MA, Riddle DL, Lamb RL, Personius WJ. Reliability of goniometric measurements and visual estimates of ankle joint active range of motion obtained in a clinical setting. Phys Ther. 1991;71:90–6.
Wakefield CB, Halls A, Difilippo N, Cottrell GT. Reliability of goniometric and trigonometric techniques for measuring Hip-extension range of motion using the modified thomas test. J Athl Train. 2015;50:150219105224004.
Hayes KW, Petersen CM. Reliability of assessing end-feel and pain and resistance sequence in subjects with painful shoulders and knees. J Orthop Sports Phys Ther. 2001;31:432–45.
Fan E, Ciesla ND, Truong AD, Bhoopathi V, Zeger SL, Needham DM. Inter-rater reliability of manual muscle strength testing in ICU survivors and simulated patients. Intensive Care Med. 2010;36:1038–43.
Escolar DM, Henricson EK, Mayhew J, Florence J, Leshner R, Patel KM, Clemens PR. Clinical evaluator reliability for quantitative and manual muscle testing measures of strength in children. Muscle Nerve. 2001;24:787–93.
Jain M, Smith M, Cintas H, Koziol D, Wesley R, Harris-Love M, Lovell D, Rider LG, Hicks J: Intra-Rater and Inter-Rater Reliability of the 10- Point Manual Muscle Test (MMT) of Strength in Children with Juvenile Idiopathic Inflammatory Myopathies (JIIM). Phys Occup Ther Pediatr 2006;26:1541–3144.
Schache MB, McClelland JA, Webster KE. Reliability of measuring hip abductor strength following total knee arthroplasty using a hand-held dynamometer. Disabil Rehabil. 2015;38:597.
www.rehabmeasures.org. [http://www.rehabmeasures.org]. Accessed 26 July 2016.
Lam T, Noonan VK, Eng JJ. SCIRE research team: a systematic review of functional ambulation outcome measures in spinal cord injury. Spinal Cord. 2008;46:246–54.
Zhang M, Davies TC, Nandakumar A, Quan S. A novel assessment technique for measuring ankle orientation and stiffness. J Biomech. 2015;48:3527–9.
Chesworth BM, Vandervoort AA. Age and passive ankle stiffness in healthy women. Phys Ther. 1989;69:217–24.
This work was conceived among the actions of “State of the Art Robot-Supported assessments (STARS)” in the frame of the COST Action TD1006 “European Network on Robotics for NeuroRehabilitation”. We would like to thank Rick Bosveld for the initial contribution to the conception of the paper.
SM at the time of writing of this work was employed within the Industrial Academic Initial Training Network “Moving Beyond” (www.moving-beyond.eu) that received funding from the People Programme (Marie Curie Actions) of the European Union’s Seventh Framework Programme (FP7/2007-2013) under REA grant agreement n° 316639.
Availability of data and supporting materials
SM wrote the manuscript. AM-C contributed substantially to the discussion and the section on Joint Impedance and to drafting of the introduction and conclusion sections. EVA and HVDK contributed substantially to the section on Joint Impedance and wrote the section on Abnormal Joint Torque Coupling and Synergies. All the authors participated in the conception of the scope and structure of the paper, edited, reviewed and approved the manuscript.
At the time of writing, SM, AM-C and LL are employed by Hocoma AG, a manufacturer of robotic devices for rehabilitation.
Consent for publication
Ethics approval and consent to participate
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
About this article
Cite this article
Maggioni, S., Melendez-Calderon, A., van Asseldonk, E. et al. Robot-aided assessment of lower extremity functions: a review. J NeuroEngineering Rehabil 13, 72 (2016). https://doi.org/10.1186/s12984-016-0180-3
- Robotic rehabilitation
- Muscle force
- Range of motion
- Joint impedance
- Translational research