Skip to main content

Augmented reality for stroke rehabilitation during COVID-19



The lack of the rehabilitation professionals is a global issue and it is becoming more serious during COVID-19. An Augmented Reality Rehabilitation System (AR Rehab) was developed for virtual training delivery. The virtual training was integrated into the participants’ usual care to reduce the human trainers’ effort so that the manpower scarcity can be eased. This also resulted in the reduction of the contact rate in pandemics.


To investigate the feasibility of the AR Rehab-based virtual training when integrated into the usual care in a real-world pandemic setting, by answering questions of whether the integrated trials can help fulfill the training goal and whether the trials can be delivered when resources are limited because of COVID-19.


Chronic stroke participants were randomly assigned to either a centre-based group (AR-Centre) or a home-based group (AR-Home) for a trial consisting of 20 sessions delivered in a human–machine integrated intervention. The trial of the AR-Centre was human training intensive with 3/4 of each session delivered by human trainers (PTs/OTs/Assistants) and 1/4 delivered by the virtual trainer (AR Rehab). The trial of the AR-Home was virtual training intensive with 1/4 and 3/4 of each session delivered by human and virtual trainers, respectively. Functional assessments including Fugl-Meyer Assessment for Upper Extremity (FMA-UE) and Lower Extremity (FMA-LE), Functional Ambulation Category (FAC), Berg Balance Scale (BBS), Barthel Index (BI) of Activities of Daily Living (ADL), and Physical Component Summary (SF-12v2 PCS) and Mental Component Summary (SF-12v2 MCS) of the 12-Item Short Form Health Survey (SF-12v2), were conducted before and after the intervention. User experience (UX) using questionnaires were collected after the intervention. Time and human resources required to deliver the human and virtual training, respectively, and the proportion of participants with clinical significant improvement were also used as supplementary measures.


There were 129 patients from 10 rehabilitation centres enrolled in the integrated program with 39 of them were selected for investigation. Significant functional improvement in FMA-UE (AR-Centre: p = 0.0022, AR-Home: p = 0.0043), FMA-LE (AR-Centre: p = 0.0007, AR-Home: p = 0.0052), SF-12v2 PCS (AR-Centre: p = 0.027, AR-Home: p = 0.036) were observed in both groups. Significant improvement in balance ability (BBS: p = 0.0438), and mental components (SF-12v2 MCS: p = 0.017) were found in AR-Centre group, while activities of daily living (BI: p = 0.0007) was found in AR-Home group. Contact rate was reduced by 30.75–72.30% within AR-All, 0.00–60.00% within AR-Centre, and 75.00–90.00% within AR-Home.


The human–machine integrated mode was effective and efficient to reduce the human rehabilitation professionals’ effort while fulfilling the training goals. It eased the scarcity of manpower and reduced the contact rate during the pandemics.


Stroke is the second commonest cause of death and one of the leading causes of disability [1,2,3]. Researches show that most of these deaths can be prevented through early interventions with rehabilitation training programs [4]. However, this requires intensive participation of physiotherapists (PTs), occupational therapists (OTs), and rehabilitation assistants, which are the scarce resources in most of the countries/regions [4,5,6]. In Hong Kong, patients are facing the similar scarcity, indicated by the statistics of Strategic Review on Healthcare Manpower Planning and Professional Development of Hong Kong in 2017 [7] in which the manpower gap of PTs (OTs) is 17% (6.6%) in 2016 and will gradually increase to 26.8% (11.2%) by 2030 because of the large ageing population (1.40 million or 18.3% of the total population of age 75 or above is predicted). However, the situation is becoming even worse since 2020 as a consequence of the COVID-19 [8, 9]. Rehabilitation centres limited their service to reduce the contact rate so as to reduce the risk of infections of this most vulnerable group of populations. In a period from February 2020 to February in 2021, many rehabilitation centres were suspended because of the first to fourth waves of pandemic [10, 11]. Many patients have missed their golden period of the rehabilitation [12]. The traditional and manpower intensive rehabilitation mode is challenged by both the scarcity of human resources and the strict regulations for contact rate reduction [13]. The emerging technology of Virtual Reality (VR) or Augmented Reality (AR), which has been validated as a promising way to assist the rehabilitation training in previous studies [14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46], has a great potential to save the efforts of rehabilitation professionals so as to ease the scarcity issue. The study of VR/AR stroke rehabilitation can date back to 2004, when Jaffe et al. built a VR training environment with a motorised treadmill, a head-mounted display (HMD), and shoes with switches on the bottom for training the patients to step over the virtual obstacles. The experimental results provided evidence with enhanced clinical performance [45]. Later on, many systems have been developed and reported with encouraging results for stroke recovery. The most recent work include the AR-augmented wheelchairs by Daniel et al. in 2020 [44], the Kinect-based training by Adyasha et al. [41] and Au\({\check{s}}\)ra et al. [42] in 2019. In Fig. 1, we have summarized the representative AR stroke rehabilitation studies/systems in recent 10 years. We can see that a diverse range of studies/systems have been conducted/built. Some are for the upper extremity including those for arms [14, 16, 17, 20, 21, 24, 25, 27, 29, 31, 34, 41,42,43], hands/fingers [14, 15, 18, 21, 24, 27, 29,30,31,32, 34,35,36,37, 42], and shoulders [14,15,16,17,18, 20, 24,25,26,27, 31, 33, 34, 39, 42, 43], while the others are on the lower extremity including those for ankles [22, 23, 40], gait [19, 23, 28, 40], balance [19, 23, 27, 28, 40], and postural [23, 27, 28]. However, most of these studies are conducted in laboratory setting and thus are less feasible in real clinical scenariosFootnote 1.

Fig. 1
figure 1

AR rehabilitation studies/systems in recent 10 years. Lab lLaboratory, H home, ADL activity of daily living, HMD head-mounted display, EMG electromyography, FES functional electric stimulation

First of all, those studies focus on the specific aspects of the recovery for research purpose (e.g., 24 of them are for upper extremity, 8 for muscle strength, and 4 for gait), while in a real clinical setting, the training needs to be designed as practical programs which are comprehensive enough to cover not only as many parts as necessary, but also the whole-body coordination and the quality of life. Furthermore, those systems may require complicated ad-hoc environmental setup which are not practical in rehabilitation centres or hospitals. For example, a precise camera calibration has to be done prior to the mirror training [14,15,16, 18,19,20,21]. Some systems require the patients to wear the HMD during training [19, 22, 23, 28, 29] which is with limited acceptability to patients, not to mention that most of the post-stroke patients are with balance impairments and it is dangerous for them to wear HMD during training. Moreover, none of previous studies has provided a quantitative analysis on how the AR systems can help to reduce the rehabilitation manpower while fulfilling the training goals, which is critical to rehabilitation centres and hospitals. It is especially important during COVID-19 pandemics.

The purpose of this study is to investigate the feasibility of the AR rehabilitation system in a real-world setting, in which it is used as a part of an integrated training program of the usual care. The setting of the integrated mode rather than applying AR training alone (like in most of the previous studies) is due to a conclusion made by Kate et al. [47]. Based on the survey including 72 trials of 2470 participants, Kate et al. concluded that the AR/VR for stroke rehabilitation, when was used alone, was not more beneficial than conventional therapy approaches in improving upper limb functions, but it could make a statistical difference when used as an adjunct to usual care [36]. Our study follows this thread. Furthermore, the study has been conducted under the emergency situations of COVID-19 in Hong Kong when the demand for reducing the manpower and contact rate reached the pandemic level. The following questions were addressed: (a) can the integrated program fulfil the training goals when a considerable part of the training (25 to 75%) has been replaced with AR guided training? (b) how much and what types of manpower can be saved while maintaining the efficacy of training with the assistance of the AR rehabilitation? (c) how can the contact rate be reduced with the AR training? (d) how do therapists/patients experience the AR training?


The study was an assessor-blind, randomized controlled trial and was approved by the Joint Chinese University of Hong Kong—New Territories East Cluster Clinical Research Ethics Committee. Participants were recruited through 10 local public rehabilitation centres, which are distributed over 8 different districts of Hong Kong. Written informed consents were obtained from participants prior to the enrollment.

Fig. 2
figure 2

The AR rehabilitation system for virtual training. a System architecture and training scene; b Training in progress; c Training plan customization; d Body tracking joints and the Kinect sensor; e Real-time report; f Training statistics


Chronic stroke survivors were selected with inclusion criteria: (1) age between 18 to 90 years who are diagnosed with ischemic brain injury, intracerebral haemorrhage shown by magnetic resonance imaging, or computed tomography after the onset of stroke; (2) with motor impairment in upper-limb, lower-limb, and/or balance; (3) have no or mild spasticity on the lower-limb or upper-limb assessed by Modified Ashworth Scale (MAS \(\le\) 2); and 4) have sufficient cognition to follow the instructions provided by the therapists and the computer. The exclusion criteria included: (1) any additional medical or psychological condition that would affect their ability to comply with the study protocol, e.g., a significant orthopaedic or chronic pain condition, major post-stroke depression, epilepsy, artificial cardiac pacemaker/joint; (2) have severe shoulder/arm or hip/knee contracture/pain and; (3) Pregnant women. Potential participants were referred by their doctors or therapists, and screened at the nearest centres. A brief tutorial of using the AR training system was given, and then the participants were required to use the system as a test to see if they can follow the instructions and complete the exercises independently. Participants who failed in the test were excluded.

Randomization and blinding

Randomization of group allocation occurred before the baseline assessment at centre level, in the way that participants who met the eligibility were first assigned to the nearest centre, and then randomly allocated to the AR-Centre or AR-Home group with a ratio of 1:1. The assignment of centre was due to the consideration of their limited mobility and the government’s regulations for prevention and control of disease during COVID-19. Baseline assessments were conducted after the assignments. This study was blinded at two levels. For the general objective of investigating whether AR rehabilitation worked, it was single blinded to assessors, because it was easy for recruiters, therapists and patients to see the difference from the usual care and was thus impossible to make this objective blind [34, 39, 48]. All assessors obtained their qualifications for the clinical assessments before joining the study. Furthermore, the objective in this study was not only to answer a question of whether AR training worked, but also to answer a more practical question of HOW it worked in a real and pandemic setting. The hypothesis was to use an integrated trial of human and virtual trainers. This was blinded to all recruiters, assessors, therapists, and patients.

Fig. 3
figure 3

CONSORT diagram of this study. Centres dropped out because their services were limited due to the government’s regulation in the pandemic. Users dropped because of quarantine or not being able to return to Hong Kong. Assessment data were lost because the regular handover process was disrupted in the pandemic


Each participant received a standard trial consisting of 20 training sessions (2–5 sessions per week, 120 min per session) in consecutive 4–10 weeks. Each session consisted of 2–10 exercises selected by the trainer from a pool of 46 exercises. The selected exercises were tailored based on the participant’s condition and progress. The exercise pool was designed as much comprehensive as possible to cover the upper-limb (22 exercises) and lower-limb (7 exercises) motor functions, balance ability (11 exercises), or/and coordination (6 exercises). Each exercise was with a duration ranging from 3 to 23 min, and was determined by a set of parameters that can be adjusted to the individuals dynamically according to the participant’s performance.

The training was delivered in an integrated manner of a human PT/OT/Assistant trainer (called human trainer hereafter) and a AR rehabilitation system (called virtual trainer hereafter).

The two groups of AR-Centre and AR-Home were designed with different configurations of trainers to study the best way to integrate the AR training into usual care.

AR-Centre group: The training was conducted at the rehabilitation centres following the same procedure as their regular usual care, with 3/4 of each session delivered by a human trainer and the rest of 1/4 delivered by a virtual trainer.

AR-Home group: The training was conducted mainly at home (3/4 of each session) and delivered by the virtual trainers. However, at the beginning of each session, there was a short (half an hour or 1/4 time of a regular usual care session) in-centre/at-home training delivered by human trainers to make the participants familiar with the subjects and procedures. To this end, upon the recruitment, a human trainer was randomly assigned to a participant for tailoring her training plan according to the baseline assessment. While the participant was taking the training at home, the human trainer could monitor the training process remotely. Reports of the training were sent to the human trainer on a weekly basis. The human trainer could adjust the training plan based on the participant’s performance.

The AR rehabilitation system for virtual training

As shown in Fig. 2, the AR rehabilitation system consists of a Microsoft Kinect V2 RGB-D camera for sensing the body movements of the patients, a TV/Monitor for displaying instructions and giving real-time assessments, and an AR rehabilitation software platform which has been implemented with the latest arts of AR and AI technology for delivering the training. The system is installed on a portable TV frame (NB MOUNT AVA 1500-60-IP) and is capable of communicating with our rehabilitation sever for synchronizing the latest training materials (videos, exercises, schedules, and training reports).

The system is able to deliver the exercises following the same procedure as in the usual care. The exercises performed by therapists are prerecorded. Every exercise starts by a brief voice introduction followed by a video demonstration with detailed steps. The participant will then follow the steps to do the exercise. Her body movements will be sensed into 3D skeletons, on the basis of which parameters such as the centre of Mass-COM, body parts, joint angles, reaching distances, speed and directions of the motions, and trajectory are evaluated. At the end of exercise, immediate feedback will be given as a report with the performance details and compared with her early performance. In addition, the participant can review her performance through the skeleton videos recorded during the exercise to locate the exact steps which she needs to pay more attention for further improvement. The performance details with the skeleton videos are synchronized to the cloud so that the human trainer could review her performance and adjust her training plan accordingly.

Table 1 Demographic and clinical characteristics of participant groups

Outcome measures

Participants received the pre-assessment within two weeks before the first training, and the post-assessment within two weeks after the last training. Assessments were conducted at centres for the AR-Centre group and at homes for the AR-Home group.

Primary outcome measures

The primary outcome measures focused on the motor function, balance, and functional ambulatory, which include (1) Fugl-Meyer Assessment with Upper Extremity and Lower Extremity (FMA-UE and FMA-LE); (2) Berg Balance Scale (BBS); (3) Functional Ambulation Category (FAC).

Secondary outcome measures

The secondary outcome measures included the assessments of (1) Quality of life (QOL): physical and mental health using the 12-Item Short Form Health Survey (SF-12v2); (2) User experience (UX) using questionnaires (designed based on UEQ-S); and (3) Activities of Daily Living (ADL) using Barthel Index (BI).

To evaluate the human effort required to deliver the training, the time of human training, the time of virtual training, the ratio of human training over the total training time, the number of human trainers required for each participant were calculated. The in-centre contact rate was calculated based on the numbers of PTs, OTs, training assistants, supporting staff, teammates the participant encountered at the centre. The numbers varied from centres and individuals. We used the numbers estimated by the PTs/OTs/Assistants and calculated the average across centres.

In addition, to study the clinical significance, we identified participants who were with improvement exceeding the Minimal Clinically Important Difference (MCID) or Minimally Detectable Change (MDC), on the basis of which we used the proportion of participants with clinical performance improvement as a supplementary measure for the efficacy of the training. More specifically, MCIDs reported in [49,50,51] were used for FMA-UE, BI, and SF12v2 PCS, and MDCs reported in [52], [53] were used for FMA-LE and BBS. For SF12v2 MCS, we used the \(10\%\) of the maximum as the threshold, because there is no MCID/MDC reported in literature. Similarly, for FAC, the clinical significance was evaluated by whether a participant of limited mobility (baseline FAC\(<4\)) become an independent walker (post-assessment FAC\(>=4\)) [54].

Statistical analysis

The normality of all demographic, clinical variables was tested by using the Kolmogorov-Smirnov method or Shapiro-Wilk method and histogram inspection. To assess the group differences on the baseline data, we used Independent Sample t tests for numerical data with a normal distribution, and Mann-Whitney U tests for data with a skewed distribution. For categorical data, we used Pearson Chi-square tests or Fisher Exact tests. To assess the differences between pre- and post-assessments, Paired t-test was used for data with a normal distribution, and Wilcoxon Signed-Ranks tests were used for data with a skewed distribution.

All outcomes measures were analyzed using both per-protocol analysis (PP) and intention-to-treat principle (ITT), and missing data were dealt with the last observation carried forward method. All outcomes aim to evaluate whether the results demonstrate feasibility of assistance with virtual rehabilitation when integrated into the usual care in a real-world pandemic setting. Statistical results were reported with the effect size in confidence interval of \(95\%\) (95% CI). Two-tailed levels of significance at 5% was used.


Participants were recruited from 10 centres across 8 districts of Hong Kong between September 2019 and August 2021. A total of 936 patients were screened, resulting in 129 from 10 centres. Eligible participants were randomly allocated to AR-Centre (\(n=89)\) and AR-Home (\(n=40\)) groups. The numbers of participants in the two groups were with discrepancy, because during the pandemic, our capacity for home system deployment has been limited by both government’s regulation for disease control and the stock-out of the core hardware (Kinect sensors and TV monitors). Meanwhile, 45 patients withdrew after the baseline (pre-) assessment including 7 with incomplete baseline assessments, 5 were under the quarantine or were not able to return after leaving for abroad, 29 due to the limited service availability of the corresponding centres during the pandemic, and 4 with data lost in the disrupted (by the pandemic) handover process. This resulted in 84 participants who completed the baseline assessment. Data of these participants (AR-Centre: \(n=59\), AR-Home: \(n=25\)) were used for ITT analysis. After that, 55 participants completed all 20 sessions, with 29 left with incomplete training because of the pandemic. There were 16 of them with incomplete data including 6 who missed from assessments/questionnaires, 6 who have only participated the upper limb training (instead of the comprehensive training of upper and lower limb, and balance), and 4 of data lost. This finally resulted in 39 participants with completed training and assessments. Data of these participants (AR-Centre: \(n=23\), AR-Home: \(n=16\)) were used for PP analysis. The CONSORT diagram is shown in Fig. 3, and the demographic data of the participants is shown in Table 1.

Although group sizes of AR-Centre and AR-Home were with discrepancy, no significant difference was found on the baseline data in both PP and ITT analysis. It indicated that the selection and chance bias have been addressed by the randomization and blinding. In this section, we included the results of both PP and ITT analysis in tables. However, due to space limitation, we mainly referred to the statistics in PP analysis when describing the results in text. The statistical results of the two analysis methods were consistent to each other.

Primary outcomes

The primary outcomes were shown as the first blocks in Table 2 (Tables 3) and  4 (Table 5). In Table 2, we presented the changes of all the participants without grouping to investigate the efficacy of the integrated intervention schemes. Significant difference of the post-assessment from the pre-assessment (baseline) were observed on both upper and lower extremity in FMA-UE (\(3.64\pm 4.69\), \(p=0.00002\)) and FMA-LE (\(3.92\pm 4.84\), \(p=0.00001\)), and on the balance in BBS (\(2.23 \pm 5.70\), \(p=0.019\)). However, there was no significant difference on the functional ambulation in FAC (\(0.22\pm 0.92\), \(p=0.149\)).

In Table 4 (Table 5), we presented the changes with comparisons between the AR-Home and AR-Centre groups. Significant differences from the baseline were observed on both groups, including the FMA-UE (\(3.57\pm 4.93\), \(p=0.0022\) for AR-Centre, \(3.75\pm 4.46\), \(p=0.0043\) for AR-Home), FMA-LE (\(4.52\pm 5.48\), \(p=0.0007\) for AR-Centre, \(3.06\pm 3.75\), \(p=0.0052\) for AR-Home). The improvement on balance was inconsistent on two groups with significant difference in BBS observed on AR-Centre (\(1.87\pm 5.42\), \(p=0.0438\)), but insignificant on AR-Home (\(2.75\pm 6.21\), \(p=0.097\)). Neither group was observed with significant difference on walk ability in FAC (\(0.13\pm 0.87\), \(p=0.48\) for AR-Centre, \(0.34\pm 1.01\), \(p=0.194\) for AR-Home). Comparing between the two groups, there was no significant difference on these parameters.

Table 2 Primary and secondary outcomes of all participants (AR-ALL, n=39) after 20 training sessions (PP analysis)
Table 3 Primary and secondary outcomes of all participants (AR-ALL, n=84) after 20 training sessions (ITT analysis)
Table 4 Primary and secondary outcomes of AR-Centre and AR-Home participants after 20 training sessions (PP analysis)
Table 5 Primary and secondary outcomes of AR-Centre and AR-Home participants after 20 training sessions (ITT Analysis)

Secondary outcomes

The secondary outcomes were shown as the second blocks in Table 2 (Tables 3) and 4 (Table 5). In Table 2, significant improvement was observed on all secondary parameters when considering all participants as a whole, with BI (\(1.25\pm 1.18\), \(p=0.0007\)), SF-12v2 PCS (\(4.82\pm 9.11\), \(p=0.002\)), and SF-12v2 MCS (\(4.94\pm 13.05\), \(p=0.023\)).

In Table 4 (Table 5), the BI was missing from the AR-Centre while was observed with significant difference on the AR-Home group (\(1.25\pm 1.18, p=0.0007\)). Consistent improvement was observed on two groups in SF-12v2 PCS (\(4.83\pm 9.79\), \(p=0.027\) for AR-Centre, \(4.81\pm 8.35\), \(p=0.036\) for AR-Home). The observation in SF-12v2 MCS was also inconsistent which is significant on AR-Centre (\(8.01\pm 14.87\), \(p=0.017\)) and insignificant on AR-Home (\(0.53\pm 8.47\), \(p=0.807\)). Comparing between two groups, there was no significant difference observed on the secondary outcomes, which is consistent with the primary outcomes.

The training effort and contact rate were presented in Table 6. By integrating AR virtual training, the trial could save the human effort from the regular usual care by \(33.19\%\) when considering all participants as a whole (AR-All), by \(19.05\%\) within the AR-Centre group, and by \(57.78\%\) within the AR-Home group. The in-centre contact rate could be reduced by 30.75–72.30% (2.46–7.23 persons) within AR-All, 0.00–60.00% (0–6 persons) within AR-Centre, and 75.00–90.00% (6–9 persons) within AR-Home.

Table 6 The human effort for delivering the training and the in-centre contact rate of participants
Table 7 User experience of participants and centre staff
Table 8 The proportions of participants with performance improvement exceeding MCIDs/MDCs

The results of the user experience were shown in Table 7. The rate for user acceptance is 81.09% and 80.77% on average among participants and human trainers, respectively.

To measure the clinical significance, the proportions of participants with performance improvements exceeding the MCIDs/MDCs/\(10\%\) max. were shown in Table 8. In the AR-Centre (AR-Home) group, the proportions were 34.78% (37.50%) on upper limb, 52.17% (62.50%) on lower limb, 43.48% (37.50%) on BBS, 4.35% (6.25%) on FAC, N/A (37.50%) on BI, 47.83% (68.75%) on SF-12v2 PCS, and 30.43% (12.50%) on SF-12v2 MCS, respectively.

Adverse events

There is no study-related adverse events reported during the process of the rehabilitative intervention in either the AR-Centre or AR-Home group.


The results give clear evidence that the trial integrated with the AR training is feasible in a real-world setting to fulfil the training goals. In terms of the intervention, significant improvement on the upper/lower extremity (in FMA-UE/LE), balance (in BBS), and quality of life (in SF-12v2 PCS/MCS) are observed. The improvement is comprehensive when compared with those of the previous studies which focus on specific functions. This is also different from the conclusion made in [47] that the majority of AR/VR intervention methods are reported with no significant improvement. However, it is not surprising because the essential difference of our integrated training from those methods is that the AR training is designed as an extension of the PT/OT rather than video games or training for specific functions. This makes the training plans are more practical.

First of all, the training plans are tailored to the same extent of that of the usual care. The plans can thus dynamically address (and be adjusted to) the specific needs of the participants, and provide a comprehensive intervention by paying a balanced attention on the participants’ functions. Secondly, the integrated scheme draws a boundary between the training efforts on the teaching and practising. It makes the human trainers can focus more on the teaching and explaining of the trials. This part needs more perceptual intelligence that is still an open question for AR research. Meanwhile, the integrated scheme leaves the AR training mainly for the practising part, in which the key is the repetition of what being taught. The human trainers are thus released from this time-consuming part and their general availability to participants are maximized. The efficacy of the balanced efforts is evident in Table 6. It shows that the 20 training sessions were completed in AR-Home groups within \(61.17\%\) of the time of that in AR-Centre group (10.35 wks vs. 16.92 wks) with the same significant improvement observed (see Table 4). The reason is that the AR-Home participants are able to receive the training sessions at least twice every week (which reaches the same level of the regular care before the pandemic) because of the increased availability of human trainers. At the same time, without the dependence on the human trainers, the participants are self-driven to practise 2.51 times longer than those in AR-Centre group (24.62 hrs vs. 9.82 hrs), so that their training effect is still ensured with the compact schedule.

This leads to the reduction of contact rate as well. It makes the training possible even during the COVID-19 regulation from February 2020 to February 2021 in Hong Kong when the government imposed strict restriction on group gathering of more than 4 (sometimes 2) persons. The rehabilitation services of seven centres were suspended at that time because it was difficult to deliver the training with the conventional scheme (which usually requires a team of a PT/OT and a few assistants). By contrast, 3 centres delivering our integrated program were still open for the services during that period. The AR-Home group was not affected because the training could be conducted at home by the participants with remote assistance of human trainers. Our records show that the average practice time of AR-Home group reached 2.64 hrs/week, which is 2.29 times longer than that of the less strict periods. Several participants reveal in our random interviews that their worries of missing the recovery period due to the regulation had drove them to conduct more self-training. The AR system installed at their homes made it happen.

Fig. 4
figure 4

The pre- and post-assessment details of participants. No. H5 (H9) and No. C20 (C22) are from AR-Home and AR-Centre groups, respectively

The feasibility of the integrated scheme also comes from its user acceptance. Despite the fact that the integrated scheme was one of a few possible solutions during the COVID-19, the results on the Table 7 show AR training is generally accepted by both participants and human trainers. One reason is that the AR training is an extension of their usual care so that the participants can work with the virtual trainers seamlessly. In addition, the design of the AR training with focus on practice has simplified the use of the system, because the training content is already delivered by the human trainers. The gait independence (FAC) is a measure in which significant improvement is not observed. It might result from our selection criteria which requires the participants are capable of using the AR training system independently. This requirement is an indirect filter that excludes the candidates with baseline below FAC 4 “Dependent, Supervision” (Ambulation occurs on level surfaces without manual contact of another person. Requires stand-by guarding of one person because of poor judgment, questionable cardiac status, or the need for verbal cuing to complete the task.). Consequently, there is a limited room for improvement in FAC among our participants. Nonetheless, this helps clarify that target users of this type of AR training systems.

The BBS is a measure with inconsistent observations (significant for participants as a whole or in the AR-Centre group, not significant in the AR-Home group). This might be due to the human trainers’ concern of safety that it is dangerous for the participants to conduct the balance related training independently at home. Therefore, they intentionally reduced the amount of balance training for those in the AR-Home group, which may result in the less significant improvement. However, the statistical evidence in both PP and ITT do not support this hypothesis well, because there is no significant difference on BBS between the AR-Centre (with full training) and AR-Home (with reduced mobility training). The evidence is too weak to make the conclusion because the two groups are with different configurations of trainers. To answer this question, we need to conduct a specific experiment in the future by aligning the configurations.

The SF-12v2 PCS is significant improved (for participants of all groups). However, the SF-12v2 MCS is with inconsistent observations (significant for participants as a whole or in the AR-Centre group, not significant in the AR-Home group). This is due to the fact that AR-Home users have less (face-to-face) communication with their PTs/OTs and friends than those of the AR-Centre, which is critical to how they feel about their quality of life (QOL) on mental well-being. In Fig. 4, we compare the assessment details of 4 participants. The AR-Home participant No. H5 (H9) is with similar improvement (on all parameters except SF-12v2 MCS) with that of the AR-Centre participant No. C20 (C22). This is an indication that similar rehabilitation efficacy of their physical functions have been archived. However, the inconsistent observations on SF-12v2 MCS indicate that AR-Centre participants may feel better about their mental QOL. By comparing the improvement of BI which is an index for activities of daily living (ADL), this is more evident that the participants’ feeling about the improvement is not necessarily consistent with the facts. Again, we can learn from the results that mental care should be improved in the future design.

In this study, we include the comparison to MCID or MDC to investigate the clinical significance of the improvements. The method is indeed rarely used in AR rehabilitation for chronic stroke recovery (i.e., it is only used in 1 out of 31 previous studies published in recent 10 years).

The reason might be that the chronic stroke recovery is a long-term, slow, and non-linear process [55,56,57,58,59] which may span years. A study with a period of several weeks or months thus has less chance to observe clinically significant improvement. However, the results in Table 8 shows more than 1/3 of our participants have achieved improvements exceeding the MCIDs or MDCs. The proportion on SF12v2 PCS is as high as \(68.75\%\) in PP analysis. This is encouraging. We believe it is another indication of the efficacy of the integrated trials. Nonetheless, this is not conclusive, given the fact that the sample size is still small. Experiments on a larger scale are expected for further verification.


While encouraging results have been observed, there are several limitations. First of all, the attrition in this study is high. It is mainly due to the limited availability of on-site services at centres during the pandemic and the quarantine of participants. It reflects one of the nature of such studies when facing a force majeure like COVID-19 [60]. One possible solution is to recruit more home users which rely less on the on-site services. However, it raises the challenges to system deployment at the same time and might be beyond the capacity of a research lab. We may need partners from industry in the future when the study scales up.

Furthermore, we have seen that the human trainers reduced the balance training because of the safety concern. It reveals the limitation of AR rehabilitation for mobility-impaired individuals. In a more general sense, this is in fact an open question for most of the Tele-rehabilitation systems, but not sufficiently discussed because a majority of previous studies are conducted in a lab setting (as shown in Fig. 1). Another closely related limitation is that mobility training often requires equipment that is rarely available at home. Our system has made the training available to the maximum extent by including exercises in sitting positions to move the legs or shift the body. For those who have difficulty to travel to hospitals or clinics, this makes the training possible for them better than having nothing at all. However, there is still a gap from the training using specific equipment. A better design of experiments/systems is needed in the future to address this issue.

Moreover, some home participants have raised the privacy concerns to the cameras, although we technically only save the 3D skeletons. This might be addressed in the future when non-RGB motion sensors (e.g., event cameras, WiFi-based motion sensors) became a mature technology.

In addition, frequency of training of two groups are different from each other, although we have fixed the training to 20 sessions. This might contribute to the efficacy of training and needed to be further investigated in the future.


We have conducted a feasibility study to investigate the efficacy of our rehabilitation programs which integrate the AR training into the participants’ usual care. The results showed that the integrated mode can significantly reduce the human (PTs/OTs/Assistants) trainers’ effort on the training as well as to reduce the contact rate during COVID-19 pandemics. The AR training system turns the time-consuming the monotonous practising part of the trials into an automated process which can be conducted by the participants independently. This also releases the human trainers so that they can focus more on the teaching and explaining part. We believe this is a practical rehabilitation mode with well-balanced human–machine coordination which is effective and efficient during the pandemics.

Availability of data and materials

All data generated or analysed during this study are included in this published article. Information on this clinical trial can be found at:


  1. Although there are several studies [14, 17, 19, 22, 34, 36, 42] are under hospital/centre setting, they were designed for research rather than daily rehabilitation programs. In [17], the experiment was conducted under home setting, but the study was conducted with only 2 users.



Augmented Reality

AR Rehab:

Our Augmented Reality Rehabilitation System


AR-Centre and AR-Home


Rehabilitation with AR Rehab in centre


Telerehabilitation with AR Rehab at home


Fugl-Meyer Assessment for Upper Extremit


Fugl-Meyer Assessment for Lower Extremit


Berg balance scale


Functional ambulation category


Functional ambulatory category


Barthel Index of Activities of Daily Living


Activities of daily living


The 12-item short form health survey

SF-12v2 PCS:

Physical component summary of SF-12v2

SF-12v2 MCS:

Mental component summary of SF-12v2


User experience




Occupational therapists


Virtual reality


Head-mounted display




Functional electric stimulation






Modified Ashworth Scale


Centre of mass


Quality of life


User Experience Questionnaire


A short 8-item version of the UEQ


Minimal clinically important difference


Minimally detectable change


  1. Avan A, Digaleh H, Di Napoli M, Stranges S, Behrouz R, Shojaeianbabaei G, et al. Socioeconomic status and stroke incidence, prevalence, mortality, and worldwide burden: an ecological analysis from the Global Burden of Disease Study 2017. BMC Med. 2019;17(1):1–30.

    Article  Google Scholar 

  2. Donkor ES. Stroke in the 21st century: a snapshot of the burden, epidemiology, and quality of life. Stroke research and treatment. 2018: 3238165.

  3. Norlander A, Iwarsson S, Jönsson AC, Lindgren A, Månsson Lexell E. Living and ageing with stroke: an exploration of conditions influencing participation in social and leisure activities over 15 years. Brain Inj. 2018;32(7):858–66.

    Article  Google Scholar 

  4. Cieza A, Causey K, Kamenov K, Hanson SW, Chatterji S, Vos T. Global estimates of the need for rehabilitation based on the Global Burden of Disease study 2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet. 2020;396(10267):2006–17.

    Article  CAS  Google Scholar 

  5. Heinemann AW, Feuerstein M, Frontera WR, Gard SA, Kaminsky LA, Negrini S, et al. Rehabilitation is a global health priority. Los Angeles, CA: SAGE Publications Sage CA; 2020.

    Google Scholar 

  6. Rehabilitation. Accessed 10 Nov 2021.

  7. Food, (FHB) HB. Strategic review on healthcare manpower planning and professional development. Author Hong Kong; 2017.

  8. Ganesan B, Fong K, Meena S, Prasad P, Tong R. Impact of COVID-19 pandemic lockdown on occupational therapy practice and use of telerehabilitation—a cross sectional study. Eur Rev Med Pharmacol Sci. 2021;25(9):3614–22.

    CAS  Google Scholar 

  9. Venketasubramanian N, Anderson C, Ay H, Aybek S, Brinjikji W, De Freitas GR, et al. Stroke care during the COVID-19 pandemic: international expert panel review. Cerebrovasc Dis. 2021;50(3):245–61.

    Article  CAS  Google Scholar 

  10. COVID-19 pandemic in Hong Kong. 2020.

  11. Health & Community: Social distancing measures extended. 2020.

  12. Ku BPS, Tse AWS, Pang BCH, Cheung NT, Pang JYW, Chan JKY, et al. Tele-rehabilitation to combat rehabilitation service disruption during COVID-19 in Hong Kong: observational study. JMIR Rehabil Assist Technol. 2021;8(3): e19946.

    Article  Google Scholar 

  13. Ganesan B, Al-Jumaily A, Fong KN, Prasad P, Meena SK, Tong RKY. Impact of coronavirus disease 2019 (COVID-19) outbreak quarantine, isolation, and lockdown policies on mental health and suicide. Front Psychol. 2021;12: 565190.

    Article  Google Scholar 

  14. King M, Hale L, Pekkari A, Persson M, Gregorsson M, Nilsson M. An affordable, computerised, table-based exercise system for stroke survivors. Disabil Rehabil Assist Technol. 2010;5(4):288–93.

    Article  Google Scholar 

  15. Burke JW, McNeill M, Charles DK, Morrow PJ, Crosbie JH, McDonough S, Augmented reality games for upper-limb stroke rehabilitation. In Second international conference on games and virtual worlds for serious applications. IEEE. 2010;2010:75–8.

  16. Aung YM, Al-Jumaily A. Development of augmented reality rehabilitation games integrated with biofeedback for upper limb. In: i-CREATe 2011-International Convention on Rehabilitation Engineering and Assistive Technology; 2011.

  17. Zhang H, Austin H, Buchanan S, Herman R, Koeneman J, He J. Feasibility studies of robot-assisted stroke rehabilitation at clinic and home settings using RUPERT. In: 2011 IEEE International Conference on Rehabilitation Robotics. IEEE; 2011. p. 1–6.

  18. Dinevan A, Aung YM, Al-Jumaily A. Human computer interactive system for fast recovery based stroke rehabilitation. In: 2011 11th International Conference on Hybrid Intelligent Systems (HIS). IEEE; 2011. p. 647–652.

  19. Kim IC, Lee BH. Effects of augmented reality with functional electric stimulation on muscle strength, balance and gait of stroke patients. J Phys Ther Sci. 2012;24(8):755–62.

    Article  Google Scholar 

  20. Da Gama A, Chaves T, Figueiredo L, Teichrieb V, Poster: improving motor rehabilitation process through a natural interaction based system using kinect sensor. In,. IEEE Symposium on 3D User Interfaces (3DUI). IEEE. 2012;2012:145–6.

  21. Khademi M, Hondori HM, Lopes CV, Dodakian L, Cramer SC. Haptic augmented reality to monitor human arm’s stiffness in rehabilitation. In: 2012 IEEE-EMBS Conference on Biomedical Engineering and Sciences. IEEE; 2012. p. 892–895.

  22. Jung GU, Moon TH, Park GW, Lee JY, Lee BH. Use of augmented reality-based training with EMG-triggered functional electric stimulation in stroke rehabilitation. J Phys Ther Sci. 2013;25(2):147–51.

    Article  Google Scholar 

  23. Park YH, Lee Ch, Lee BH. Clinical usefulness of the virtual reality-based postural control training on the gait ability in patients with stroke. J Exerc Rehabil. 2013;9(5):489.

    Article  Google Scholar 

  24. Mousavi Hondori H, Khademi M, Dodakian L, Cramer SC, Lopes CV. A spatial augmented reality rehab system for post-stroke hand rehabilitation. In: Medicine meets virtual reality 20. IOS Press; 2013. p. 279–285.

  25. Aung YM, Al-Jumaily A. Augmented reality based illusion system with biofeedback. In: 2nd Middle East Conference on Biomedical Engineering. IEEE; 2014. p. 265–268.

  26. Aung YM, Al-Jumaily A. Augmented reality-based RehaBio system for shoulder rehabilitation. Int J Mechatron Autom. 2014;4(1):52–62.

    Article  Google Scholar 

  27. De Leon NI, Bhatt SK, Al-Jumaily A. Augmented reality game based multi-usage rehabilitation therapist for stroke patients. Int J Smart Sensing Intel Syst. 2014;7(3).

  28. Lee CH, Kim Y, Lee BH. Augmented reality-based postural control training improves gait function in patients with stroke: randomized controlled trial. Hong Kong Physiother J. 2014;32(2):51–7.

    Article  Google Scholar 

  29. Trojan J, Diers M, Fuchs X, Bach F, Bekrater-Bodmann R, Foell J, et al. An augmented reality home-training system based on the mirror training and imagery approach. Behav Res Methods. 2014;46(3):634–40.

    Article  Google Scholar 

  30. Corrêa A, de Assis G, do Nascimento M, de Deus Lopes R. Perceptions of clinical utility of an Augmented Reality musical software among health care professionals. Disabil Rehabil Assist Technol. 2015;12(3):205–216.

  31. Ponte S, Gabrielli S, Jonsdottir J, Morando M, Dellepiane S. Monitoring game-based motor rehabilitation of patients at home for better plans of care and quality of life. In 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE. 2015;2015:3941–4.

  32. Collins J, Hoermann S, Regenbrecht H. Comparing a finger dexterity assessment in virtual, video-mediated, and unmediated reality. Int J Child Health Hum Dev. 2016;9(3):333–42.

    Google Scholar 

  33. Da Gama AEF, Chaves TM, Figueiredo LS, Baltar A, Meng M, Navab N, et al. MirrARbilitation: a clinically-related gesture recognition interactive tool for an AR rehabilitation system. Comput Methods Programs Biomed. 2016;135:105–14.

    Article  Google Scholar 

  34. Colomer C, Llorens R, Noé E, Alcañiz M. Effect of a mixed reality-based intervention on arm, hand, and finger function on chronic stroke. J Neuroeng Rehabil. 2016;13(1):1–11.

    Article  Google Scholar 

  35. Liu J, Mei J, Zhang X, Lu X, Huang J. Augmented reality-based training system for hand rehabilitation. Multimed Tools Appl. 2017;76(13):14847–67.

    Article  Google Scholar 

  36. Hoermann S, Ferreira dos Santos L, Morkisch N, Jettkowski K, Sillis M, Devan H, et al. Computerised mirror therapy with augmented reflection technology for early stroke rehabilitation: clinical feasibility and integration as an adjunct therapy. Disabil Rehabil. 2017;39(15):1503–14.

    Article  Google Scholar 

  37. Pinches J, Hoermann S. Automated instructions and real time feedback for upper limb computerized mirror therapy with augmented reflection technology. J Altern Med Res. 2018;10(1):37–46.

    Google Scholar 

  38. Syed Ali Fathima SJ , Shankar S. AR using NUI based physical therapy rehabilitation framework with mobile decision support system: a global solution for remote assistance. J Global Info Manag. 2018;26(4):36–51.

  39. Liao W, McCombe Waller S, Whitall J. Kinect-based individualized upper extremity rehabilitation is effective and feasible for individuals with stroke using a transition from clinic to home protocol. Cogent Med. 2018;5(1):1428038.

  40. An CM, Kim DH. Clinical application of AR system in early rehabilitation program after stroke: 2 case study. J Korean Phys Therapy. 2019;31(3):141–6.

    Article  Google Scholar 

  41. Dash A, Yadav A, Chauhan A, Lahiri U. Kinect-assisted performance-sensitive upper limb exercise platform for post-stroke survivors. Front Neurosci. 2019;13:228.

    Article  Google Scholar 

  42. Adomavičienė A, Daunoravičienė K, Kubilius R, Varžaitytė L, Raistenskis J. Influence of new technologies on post-stroke rehabilitation: a comparison of armeo spring to the kinect system. Medicina. 2019;55(4):98.

    Article  Google Scholar 

  43. Orand A, Erdal Aksoy E, Miyasaka H, Weeks Levy C, Zhang X, Menon C. Bilateral tactile feedback-enabled training for stroke survivors using microsoft kinecttm. Sensors. 2019;19(16):3474.

    Article  Google Scholar 

  44. Caetano DSD, Valentini C, Mattioli F, Camargos P, Sá T, Cardoso A, et al. The augmented reality telerehabilitation system for powered wheelchair user’s training. J Commun Inf Syst. 2020;35(1):51–60.

    Google Scholar 

  45. Jaffe DL, Brown DA, Pierson-Carey CD, Buckley EL, Lew HL. Stepping over obstacles to improve walking in individuals with poststroke hemiplegia. J Rehabil Res Dev. 2004;41(3A):283-92.

  46. Ganesan B, Gowda T, Al-Jumaily A, Fong K, Meena S, Tong R. Ambient assisted living technologies for older adults with cognitive and physical impairments: a review. Eur Rev Med Pharmacol Sci. 2019;23:10470–81.

    CAS  Google Scholar 

  47. Laver KE, Lange B, George S, Deutsch JE, Saposnik G, Crotty M. Virtual reality for stroke rehabilitation. Cochrane Database Syst Rev. 2017;11(11):CD008349.

  48. Saywell NL, Vandal AC, Mudge S, Hale L, Brown P, Feigin V, et al. Telerehabilitation after stroke using readily available technology: a randomized controlled trial. Neurorehabil Neural Repair. 2021;35(1):88–97.

    Article  Google Scholar 

  49. Page SJ, Fulk GD, Boyne P. Clinically important differences for the upper-extremity Fugl-Meyer Scale in people with minimal to moderate impairment due to chronic stroke. Phys Ther. 2012;92(6):791–8.

    Article  Google Scholar 

  50. Hsieh YW, Wang CH, Wu SC, Chen PC, Sheu CF, Hsieh CL. Establishing the minimal clinically important difference of the Barthel Index in stroke patients. Neurorehabil Neural Repair. 2007;21(3):233–8.

    Article  Google Scholar 

  51. Fu V, Weatherall M, McNaughton H. Estimating the minimal clinically important difference for the Physical Component Summary of the Short Form 36 for patients with stroke. J Int Med Res. 2021;49(12):03000605211067902.

    Article  Google Scholar 

  52. Hiengkaew V, Jitaree K, Chaiyawat P. Minimal detectable changes of the Berg Balance Scale, Fugl-Meyer Assessment Scale, Timed “Up & Go’’ Test, gait speeds, and 2-minute walk test in individuals with chronic stroke with different degrees of ankle plantarflexor tone. Arch Phys Med Rehabil. 2012;93(7):1201–8.

    Article  Google Scholar 

  53. Alghadir AH, Al-Eisa ES, Anwer S, Sarkar B. Reliability, validity, and responsiveness of three scales for measuring balance in patients with chronic stroke. BMC Neurol. 2018;18(1):1–7.

    Article  Google Scholar 

  54. Mehrholz J, Wagner K, Rutte K, Meissner D, Pohl M. Predictive validity and responsiveness of the functional ambulation category in hemiparetic patients after stroke. Arch Phys Med Rehabil. 2007;88(10):1314–1319.

  55. Demain S, Wiles R, Roberts L, McPherson K. Recovery plateau following stroke: fact or fiction? Disabil Rehabil. 2006;28(13–14):815–21.

    Article  Google Scholar 

  56. Almenkerk SV, Smalbrugge M, Depla MF, Eefsting JA, Hertogh CM. What predicts a poor outcome in older stroke survivors? A systematic review of the literature. Disabil Rehabil. 2013;35(21):1774–82.

    Article  Google Scholar 

  57. Hatem SM, Saussez G, Della Faille M, Prist V, Zhang X, Dispa D, et al. Rehabilitation of motor function after stroke: a multiple systematic review focused on techniques to stimulate upper extremity recovery. Front Hum Neurosci. 2016;10:442.

    Article  CAS  Google Scholar 

  58. Stinear CM, Lang CE, Zeiler S, Byblow WD. Advances and challenges in stroke rehabilitation. Lancet Neurol. 2020;19(4):348–60.

    Article  CAS  Google Scholar 

  59. Kundert R. Improving inpatient stroke rehabilitation: proportional recovery. ETH Zurich: Neural Coupling and Performance Feedback; 2020.

    Google Scholar 

  60. Bersano A, Kraemer M, Touzé E, Weber R, Alamowitch S, Sibon I, et al. Stroke care during the COVID-19 pandemic: experience from three large European countries. Eur J Neurol. 2020;27(9):1794–800.

    Article  CAS  Google Scholar 

Download references


We thank the support of the rehabilitation teams, administrative staff, and participants of the 10 rehabilitation centres who were enrolled in this study, and the help of the teachers and student helpers at School of Medical and Health Science of Tung Wah College. Thank you for all the research assistants and student helpers of CUHK that make this project possible.


This study was supported by the Hong Kong Jockey Club Charity Trust (Project ID: 6904735) and the Innovation and Technology Fund – Public Sector Trial Scheme for the Prevention and Control of Coronavirus Disease 2019 in Hong Kong (Project ID: PSTS-COVID-19 (SST/105/20GP)).

Author information

Authors and Affiliations



The authors confirm contribution to the paper as follows: study conception and design: Z.Q. Yang and R.K.Y. Tong; AR Rehab system design and implementation, and data collection: Z.Q. Yang and D. Du ; data analysis and interpretation of results: Z.Q. Yang, D. Du, X.Y. Wei, and R.K.Y. Tong; draft manuscript preparation: Z.Q. Yang, X.Y. Wei, and R.K.Y. Tong. All authors reviewed the results, read and approved the final version of the manuscript.

Corresponding author

Correspondence to Raymond Kai-Yu Tong.

Ethics declarations

Ethics approval and consent to participate

This study was approved by The Joint Chinese University of Hong Kong-New Territories East Cluster Clinical Research Ethics Committee (CREC Ref.No.: 2019.377-T and 2020.538-T). This trial was designed following the principle of the Declaration of Helsinki. All recruited subjects read and signed consent form before participation.

Consent for publication

The person with stroke problem in Fig. 2a has consented to the publication of the photograph.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, ZQ., Du, D., Wei, XY. et al. Augmented reality for stroke rehabilitation during COVID-19. J NeuroEngineering Rehabil 19, 136 (2022).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: