- Open Access
What is the impact of user affect on motor learning in virtual environments after stroke? A scoping review
Journal of NeuroEngineering and Rehabilitation volume 16, Article number: 79 (2019)
The purported affective impact of virtual reality (VR) and active video gaming (AVG) systems is a key marketing strategy underlying their use in stroke rehabilitation, yet little is known as to how affective constructs are measured or linked to intervention outcomes. The purpose of this scoping review is to 1) explore how motivation, enjoyment, engagement, immersion and presence are measured or described in VR/AVG interventions for patients with stroke; 2) identify directional relationships between these constructs; and 3) evaluate their impact on motor learning outcomes.
A literature search was undertaken of VR/AVG interventional studies for adults post-stroke published in Medline, PEDro and CINAHL databases between 2007 and 2017. Following screening, reviewers used an iterative charting framework to extract data about construct measurement and description. A numerical and thematic analytical approach adhered to established scoping review guidelines.
One hundred fifty-five studies were included in the review. Although the majority (89%; N = 138) of studies described at least one of the five constructs within their text, construct measurement took place in only 32% (N = 50) of studies. The most frequently described construct was motivation (79%, N = 123) while the most frequently measured construct was enjoyment (27%, N = 42). A summative content analysis of the 50 studies in which a construct was measured revealed that constructs were described either as a rationale for the use of VR/AVGs in rehabilitation (76%, N = 38) or as an explanation for intervention results (56%, N = 29). 38 (76%) of the studies proposed relational links between two or more constructs and/or between any construct and motor learning. No study used statistical analyses to examine these links.
Results indicate a clear discrepancy between the theoretical importance of affective constructs within VR/AVG interventions and actual construct measurement. Standardized terminology and outcome measures are required to better understand how enjoyment, engagement, motivation, immersion and presence contribute individually or in interaction to VR/AVG intervention effectiveness.
An increasing evidence base supports the use of virtual reality (VR) and active video gaming (AVG) systems to promote motor learning in stroke rehabilitation [1,2,3,4]. However, practical and logistical barriers to VR/AVG implementation in clinical sites have been well described [5,6,7]. To support their use, researchers and developers often emphasize the potential advantages of VR/AVG systems over conventional interventions, including that these technologies may enhance a patient’s affective experience in therapy for the purpose of facilitating recovery [8,9,10,11]. Examining the role of affective factors for motor learning is an emerging area of emphasis in rehabilitation [2, 12,13,14,15].
VR/AVG use may enhance patients’ motivation to participate in rehabilitation as well as their engagement in therapeutic tasks. Motivation encourages action toward a goal by eliciting and/or sustaining goal-directed behavior . Motivation can be intrinsic (derived from personal curiosity, importance or relevance of the goal) or extrinsic (elicited via external reward) . Engagement is a cognitive and affective quality or experience of a user during an activity . Many characteristics of VR/AVG play can contribute to user motivation and engagement, such as novelty, salient audiovisual graphics, interactivity, feedback, socialization, optimal challenge , extrinsic rewards, intrinsic curiosity or desire to improve in the game, goal-oriented tasks, and meaningful play .
Motivation and engagement are hypothesized to support motor learning either indirectly, through increased practice dosage leading to increased repetitive practice, or directly, via enhanced dopaminergic mechanisms influencing motor learning processes [15, 16]. Yet evidence is required to support these claims. A logical first step is to understand how these constructs are being measured within VR/AVG intervention studies. Several studies have used practice dosage or intensity as an indicator of motivation or engagement [19,20,21]. To the authors’ knowledge, few have specifically evaluated the indirect mechanistic pathway by correlating measurement of patient motivation or engagement in VR/AVGs with practice dosage or intensity. While participants in VR/AVG studies report higher motivation as compared to conventional interventions [22,23,24], conclusions regarding the relationship between motivation and intervention outcomes are limited by lack of consistency and rigour in measurement, including the use of instruments with poor psychometric properties [22, 23].
The body of research exploring the direct effects of engagement or motivation on motor learning is still in its infancy. Lohse et al.  were the first to evaluate whether a more audiovisually enriched as compared to more sterile version of a novel AVG task contributed to skill acquisition and retention in typically developing young adults, finding that participants who played under the enriching condition had greater generalized learning and complex skill retention. Self-reported engagement (User Engagement Scale; UES) was higher in the enriched group, but the only difference in self-reported motivation was in the Effort subscale of the Intrinsic Motivation Inventory (IMI), where the enriched group reported less effort as compared to the sterile group. The authors did not find a significant correlation between engagement, motivation and retention scores. A follow-up study using electroencephalography did not replicate the finding that the more enriched practice condition enhanced learning, it did show that more engaged learners had increased information processing, as measured by reduced attentional reserve .
Enjoyment, defined as ‘the state or process of taking pleasure in something’ , has less frequently been the subject of study in motor learning research, but has become popular as a way of describing patient interaction with VR/AVGs. Enjoyment may be hypothesized to be a precursor to both motivation and engagement. Given that the prevailing marketing of VR/AVGs is that they are ‘fun’ and ‘enjoyable’ [1, 3, 14, 27], it is important to evaluate its measurement in the context of other constructs.
Motivation, engagement and enjoyment in VR/AVGs may be influenced by the additional constructs of immersion and presence. Immersion is defined as “the extent to which the VR system succeeds in delivering an environment which refocuses a user’s sensations from the real world to a virtual world” [13, 28]. Immersion is considered as an objective construct referring to how the computational properties of the technology can deliver an illusion of reality through hardware, software, viewing displays and tracking capabilities [29, 30]. A recent systematic review  could not conclusively state effect of immersion on user performance. Immersion is distinct from presence, defined as the “psychological product of technological immersion” . Presence is influenced by many factors, including the characteristics of the user, the VR/AVG task, and the VR/AVG system . While presence is thought to be related to enhanced motivation and performance , relationships between this and other constructs of interest require exploration. Table 1 outlines definitions of constructs of interest to this scoping review.
The purpose of this scoping review is to explore the impact of these affective constructs on motor learning after stroke. This greater understanding will enhance the clinical rationale for VR/AVG use and inform directions for subsequent research. Specifically, our objectives were to:
Describe how VR/AVG studies measure or report client enjoyment, motivation, engagement, immersion and presence.
Evaluate the extent to which motivation, enjoyment, engagement, immersion, and presence impact motor learning.
Propose directional relationships between enjoyment, motivation, engagement, immersion, presence and motor learning.
Scoping reviews synthesize knowledge about an exploratory research question to map a field of literature . They are useful methodologies to address questions beyond effectiveness and to describe how a particular subject has been conceptualized or studied . The study is structured according the original methodological framework for conducting scoping reviews  and the updated recommendations proposed by Levac et al. . The updated methodological framework consists of six stages: stage 1) Identifying the research question; Stage 2) Searching for relevant studies; 3) Selecting studies; 4) Charting the data; 5) Collating, summarizing, and reporting the results, and 6) Consulting with stakeholders to inform or validate study findings. The consultation stage is optional and was omitted here. The review follows the PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) reporting guidelines .
Protocol and registration
The protocol was registered with the Open Science Framework on 8 November 2017 (OSF, http://osf.io/3x6y5) after the development of the search strategy and before data extraction and analysis.
Stage 1. Identifying the research questions
Our research questions (RQ) were as follows:
RQ1: How have studies of VR/AVGs in stroke rehabilitation measured or described motivation, enjoyment, engagement, immersion or presence?
RQ2: What is known as to the extent to which motivation, enjoyment, engagement, immersion and presence impact training outcomes?
RQ3: What are the proposed relationships between motivation, enjoyment, engagement, immersion, presence and motor learning?
Stage 2. Identifying relevant studies
CINAHL Complete, MEDLINE, and PEDro were searched for articles published from 2007 up to November 2017. This timeline was chosen on the basis of the rapid development in the field following the release of the active video gaming system Nintendo Wii/WiiFit in 2007. The specific search strategy was created by one author (NR) and peer reviewed by another author (DL) with expertise in conducting scoping reviews. A combination of medical sub-headings (MeSH) and key words on “stroke” and “virtual reality”, were adapted as needed for each database and combined using boolean operators.
The MEDLINE search strategy is exemplarily presented in Additional file 1: Table S1 (Date of last search: Friday, October 13, 2017 4:16:49 PM).
Stage 3. Study selection
The Population-Intervention-Comparison-Outcome (PICO) approach was applied to systematically define our eligibility criteria. Inclusion criteria were studies of any design published in the last 10 years in English or German describing rehabilitation interventions using virtual reality (VR) and/or active video games (AVG) including both commercially available systems (i.e. Nintendo Wii console, Sony PlayStation or Xbox Kinect consoles) and custom designed games for stroke rehabilitation (including interfaces such as handheld controllers, gloves, treadmill, etc.) for motor skill improvement in adult patients with stroke. We did not specifically search for studies measuring any of the five constructs. We included studies with any type of control as long as data provided for stroke population were reported separately. Exclusion criteria were robot-based interventions and robot-assisted training (exoskeletons, fixed manipulandum), functional electrical stimulation/ transcranial current stimulation; brain computer interface, electromyography-controlled interventions; outcomes other than motor-based (i.e. energy expenditure, metabolism, cognitive, memory, communication, neglect), and usability/reliability studies, reviews or meta-analyses. Robotic devices, even those that include VR simulations, were excluded due to the additional potential influences on our constructs of interest resulting from the physically assistive nature of these devices.
Selection of sources of evidence
Prior to the formal screening process, a calibration exercise with two reviewers (JZ, NR) on a subset of articles (n = 10) was undertaken to pilot the screening questions and eligibility criteria. Based on that calibration exercise, the first selection was made by title and abstract screening of each study by one reviewer (NR). Studies that did not meet the eligibility criteria on the basis of the content of their abstracts were excluded. If required, the full-text versions were obtained to determine whether the studies met our eligibility criteria. In case of uncertainty, another reviewer (DL) blind to the first reviewers’ comments reviewed the study. Any disagreements concerning the inclusion/exclusion were collaboratively discussed until the authors met consensus. Figure 1 outlines the study selection process .
Stage 4. Charting the data
Data charting process
Data from the identified studies were extracted using a charting framework developed a priori by the authors. The charting framework was pilot-tested with a random sample of five articles to check agreement among reviewers. As a result of this process, charting rules were developed to guide a group of five reviewers, including the original two (NR, DL, KP, JZ, MS), who independently charted data from each eligible article. The construct ‘enjoyment’ was added following data extraction from the first 50 papers, as reviewers noted that it was a frequently mentioned construct and should be included in the review; these papers were re-reviewed. The full process required frequent review and discussion within the core research team (NR, EC, and DL) to resolve any uncertainties, and ensure that data extraction was in line with the research questions. All charted data from each reviewer was reviewed by one of the authors (NR). EndNote X8 was used as a reference management software and to avoid multiple reports of the same study. Microsoft Excel was used to manage data within the review team.
The main data extracted were 1) study characteristics (journal name, year of publication, study design, type of VR/AVG technology, study sample, duration of intervention); 2) text in which the author mentioned or described each of the five constructs (motivation, enjoyment [also described as ‘fun’], engagement, immersion, presence); 3) nature of measurement of any of the five constructs (inferential or qualitative), and 4) text related to the authors’ proposed relationships between the constructs.
Characteristics and critical appraisal of individual sources of evidence
We did not appraise the methodological quality or risk of bias of the included studies, which is consistent with guidance on scoping review conduct .
Stage 5. Collating and summarizing the results
Synthesis of results
Numerical analysis (counts, frequencies, proportions) was used to map the studies included in the review in terms of study design, type of VR/AVG system and viewing medium, intervention focus (upper vs lower extremities vs postural control), frequency of mention of constructs, and frequency of measurement of constructs. In a first step, we screened the articles for the constructs of interest (if, and where, the construct was mentioned). In a second step, the studies that mentioned one or more constructs were checked to clarify whether and how the authors measured these constructs (Additional file 2). If measurement was undertaken, the article was included for further analysis (Additional file 3). A numerical summary was used to describe the frequency and type of inferential statistics per construct, and the results of the statistical analyses were summarized. Summative content analysis  of the authors’ construct description within the text was undertaken for the subset of studies in which a construct was measured. The goal of summative content analysis is to understand and identify how words are used in context (in this case, we were interested in the specific labels of each of our 5 constructs) . In addition to counting frequencies of use, this approach interprets how words are used and how they relate to each other. Summative content analysis was also used to identify how authors’ described relationships between motivation, engagement, enjoyment, immersion, presence and motor learning in their texts. This analysis resulted in frequency counts of each relationship, which we illustrated in a Fig. 2.
Selection of sources of evidence
Figure 1 outlines the study selection process.
Overview of the included studies
The articles included in this review employed a range of methodologies: 45.2% were Randomized Controlled Trials (RCTs, n = 70), 18.7% were pilot-studies (n = 29), 9.7% were pilot-RCTs (n = 15), 10.3% used a pre-post design (n = 16), 5.8% were case reports/series (n = 9), and 10.4% (n = 16) applied other designs such as mixed-methods, interviews, non-randomized controlled or crossover trials, overviews of work, case control or descriptive observational studies. Table 2 illustrates that most studies (47%, n = 73) involved customized, rehabilitation-specific devices in which tracking/interaction took place indirectly via a controller (e.g., the YouGrabber system) or directly via motion capture (e.g., using the Microsoft Kinect sensor) and visual display of the virtual environment was on a 2D flat-screen monitor, while only 5 % of studies (n = 8) investigated any type of stereoscopic glasses or head mounted display. Most (66.45%) of included studies (n = 103) focused on upper extremity impairments after stroke while 29.67% (n = 46) focused on lower extremities/balance and 3.8% focused on both (n = 6). Additional File 2 provides a complete list of the 155 studies included in the review.
RQ1: How have studies of VR/AVGs in stroke rehabilitation described or measured motivation, engagement, enjoyment, presence or immersion?
One hundred fifty-five studies were included in the review. 89% (n = 138/155) of studies mentioned at least one of the five constructs within their text, but only 32% (N = 50/155) measured a construct using a standardized or non-standardized outcome measure (Table 3).
Table 4 lists the outcome measures used per construct. Examples of standardized measurements include the Intrinsic Motivation Inventory (IMI) and the Presence Questionnaire (PQ). Self-designed questionnaires were applied in 15 studies, where authors mostly used Likert scales or dichotomous yes/no answer formats to evaluate motivation, engagement and enjoyment/fun. For example, Chen et al.  designed specific questions to assess motivation and enjoyment that were scored on a 5-point Likert-type scale, with 1 signifying “strongly disagree” and 5 being “strongly agree”. Another example can be found in Schuck et al.  who asked questions requiring yes/no answers such as “Was the game fun to play? Did the game increase your motivation to perform your exercise?”, Summative content analysis  of the authors’ construct description within the text was undertaken for the 50 studies in which a construct was measured. Additional File 3 provides a complete list of the 50 studies included in summative content analysis.
Two themes emerged from the content analysis. In the first theme, represented in 76% of studies (n = 38/50), authors described the construct as a rationale for use of VR/AVGs in rehabilitation. In the second theme, represented in 58% of studies (n = 29/50), authors used the construct to explain why the VR/AVG intervention was successful. The two themes are described below. Table 5 depicts the quantitative breakdown of both themes for each individual construct.
Theme 1: Construct described as a rationale for use of VR/AVG
Each of the five constructs was described under this theme. Engagement and motivation were described almost identically. Authors described engagement as a rationale for VR/AVG use because of its potential to influence practice dosage and adherence, greater amounts of which were felt to promote functional improvements [19, 20, 56, 57, 68]. Motivation was also described as a rationale for use of VR/AVG interventions for its potential to increase training intensity, influencing motor learning and neuroplasticity [20, 39, 41,42,43,44,45, 49, 60]. The ability to motivate clients in this way was identified as unique to this treatment method [40, 42, 45, 48]. Rationales presented for VR/AVG use included the potential to engage and motivate users by involving them in game selection  or individualization of game features , the ability to elicit multiplayer competition or cooperation [39, 44, 48] the provision of individualized challenge [21, 44, 48, 54, 57], and the delivery of feedback [43, 45, 50, 53, 61, 62, 73] or of a rewarding sense of achievement [45, 55]. For example, Subramanian et al.  stated that “Motivation and interactivity of the VE were enhanced by the added visual effects and game score that enabled participants to track success.”
The potential for VR/AVG use to increase patient enjoyment was described as a strong rationale for use in rehabilitation. Specifically, authors outlined patient enjoyment related to playing games [60, 62, 64, 66, 70,71,72, 82] which differed from traditional rehabilitation approaches (e.g. [52, 73]. Enjoyment was also seen as essential to the flow experience induced by VR/AVG play [57, 72]. Flow was defined as the “feeling of complete and energized engagement in an activity, with a high level of enjoyment and fulfillment” and described as supportive of adherence to VR/AVG-based rehabilitation . Enjoyment was described as facilitating motivation [20, 39,40,41,42,43,44,45, 47,48,49, 68, 74], engagement [57, 68, 78], and training intensity [42, 60, 66, 79] in VR/AVG use. Finally, patient enjoyment due to rewards and feedback provided in VR/AVG games was described as a rationale for their use in clinical practice [57, 66, 78].
Immersion was described as a rationale for VR/AVG use because of its influence on user performance and the fact that it differentiates VR/AVG interventions from conventional rehabilitation [76, 84]. Similarly, authors described presence as an essential component separating the advantages of VR/AVG use over other interventions [75, 76, 78, 84]. All authors interpreted immersion as a subjective characteristic, i.e. defining it as “the perception of the setting as real” [76, 84], or “the feeling of being in the virtual world, rather than looking at it” . For example, Crosbie et al.  stated: “A person with a positive immersive tendency [as measured by the ITQ instrument] may be more likely to be successful in the performance of virtual tasks.” Authors also described the need to measure side effects associated with immersion to justify the burden of VR/AVG use [19, 78, 84].
Theme 2: Construct described as an explanation for successful intervention
Motivation, enjoyment and engagement were the only constructs described under this theme. Engagement and motivation were described as contributing to intervention success by promoting adherence and contributing to a higher training intensity [20, 41, 43, 47, 48, 52, 55, 58] as well as by distracting participants’from therapeutic intent [50, 60]. For example, Lewis et al.  state that “the level of engagement and motivation in performing tasks is posited as factor in determining the success of rehabilitation interventions using VR”. Another example is Sampson et al.  who describe that “(…) perhaps the main benefit found in this study was that the VR system successfully motivated participants to practice using their affected arms and engage in and enjoy therapy for sustained periods of time.” Game design features such as individualized challenge levels [21, 42, 47, 48, 51,52,53, 57, 58, 61, 62], meaningful tasks [20, 52, 53, 57, 58], multiplayer platforms [39, 48], and feedback [42, 43, 46, 48, 49, 51, 52, 58, 61,62,63] were described as promoting motivation and influencing successful outcomes. For example, Friedmann et al.  suggested that “...sensory-rich visual and auditory feedback motivated high effort levels”.
Enjoyment achieved through VR/AVG play was described as important to intervention outcomes because it is a critical factor for rehabilitation success [53, 68, 72], lowers stress levels  and induces flow [57, 60, 72]. Enjoyment was also described as an explanation for the success of the interventions due to its effects on patient motivation and engagement [20, 45, 52, 64, 73], particularly in patients who otherwise lacked interest or motivation to complete normal exercise regimes [39, 58]. Gorsic et al.  stated enjoyment led to effort, stating that “participants enjoyed competitive exercises more than exercising alone, and that this also increased self-reported effort put into the exercise.” Schuck et al.  referred their intervention success to previous literature “that implicated the importance of fun, motivation, and engagement as critical factors for success in rehabilitation.”
RQ2: What do we know about the extent to which motivation, engagement, presence, enjoyment and level of immersion impact training outcomes?
While 74% of studies (n = 37) in which a construct was measured reported only descriptive statistics or qualitative summaries, 26% (n = 13) used inferential statistical analysis to evaluate hypotheses related to a construct in different arms of the intervention, e.g. to compare differences in motivation or enjoyment between two studied practice conditions [40, 42, 43, 45, 47, 49, 52, 65, 71, 73]. None of the studies used statistical inference to link any of the five constructs to motor learning outcomes.
RQ3: What are the proposed relationships between motivation, engagement, enjoyment, immersion, and presence and motor learning?
Summative content analysis was used to explore authors’ interpretations of construct relationships in their texts. Figure 2 illustrates the frequency and direction of identified relationships.
The most frequently described relationship was motivation leading to motor learning (N = 24). For example, Hale et al.  state: “One of the rationales for computer-based rehabilitation is the use of the motivational aspects of the technology to stimulate people to practice repetitive movement to facilitate neuroplasticity and enhance functional movement.” The second most frequently reported relationship was the influence of patient enjoyment on motivation (N = 15). Enjoyment is seen as “a key factor for increasing motivation” . However, authors measured this construct using an instrument designed to assess motivation: the Intrinsic Motivation Inventory (IMI). For instance, Lloréns et al.  conclude: “In terms of motivation, the results of the IMI showed that most of the participants found the system enjoyable (...)”.
The combination of two constructs was suggested to influence a third construct. For example, Kottink et al.  suggest that the combination of fun and motivation together lead to engagement, stating: “The application of videogames in rehabilitation (rehab games) can be regarded as a specific form of VR training, in which the fun and motivational elements of the exercises are emphasized to engage people during their activity.” Multidimensional relationships, in which constructs are chained, are listed in Table 6.
For example, Flynn et al.  state that engagement and motivation lead to immersion and this impacts therapeutic outcomes: “Moreover, the virtual environment (VE) provides an engaging and motivating framework for feedback allowing the participant to become immersed in the virtual world and to experience the emotional sense of “winning” in a particular game.” Turkbey et al.  suggest that enjoyment leads to motivation, which leads to engagement: “It should be remembered that patients’ enjoyment and belief in benefits of a treatment may improve engagement in a therapy and intensity of training as a reflection of increased motivation.” Finally, Hung et al.  also describe this relationship: “One of the most important successes of the Wii Fit training may lie in the pleasure component, which motivates subjects to engage more fully in the program.”
This scoping review explored how motivation, enjoyment, engagement, immersion and presence were described or measured in VR/AVG studies in stroke rehabilitation. We also sought to identify potential links between these constructs and motor learning outcomes. Although the majority of studies mentioned at least one of the five constructs within their text, construct measurement took place in only 1/3 of studies. Multiple relational links between two or more constructs or between any construct and motor learning were described, though statistical analyses were not used to examine these links.
The emphasis by authors on enjoyment was a surprising finding of this review. Enjoyment was described as important because it underlies both engagement and motivation, and because it is central to essential game design principles of VR/AVG games. However, although it was the most frequently measured construct, it is important to note that measurement of this construct was undertaken with the use of instruments designed for other purposes. This included using instruments measuring flow or intrinsic motivation [85, 86] or self-designed subjective questionnaires lacking psychometric properties . Hung et al.  were the only ones to use an enjoyment-specific scale (PACES), although its psychometric properties have not yet been validated in the stroke population or for exercise modalities other than sports . Given that authors appear to consider this construct foundational both to the affective impact of VR/AVGs and to the mechanics of game design, it will be important to achieve consensus on optimal measurement.
A second important finding of the review was the inconsistency with which constructs were mentioned, described, defined and measured in these studies, and the fact that despite lack of tests of statistical inference or even measurement, authors stated assumptions or conclusions about constructs as fact. For example, Shin et al.  conclude that their device “encouraged the patient’s skill development, improved immersion, and motivated further rehabilitation by providing meaningful play, optimal challenge, and a flow experience” while acknowledging that they did not measure motivation. In addition, definitions did not consistently align with our a-priori understanding of the terms, and were often vague and interchangeable. Indeed, these terms are differentially operationalized and defined in various fields (e.g. psychology, sports medicine, rehabilitation). This issue of ill-defined terminology was identified by some authors . For example, immersion was often described as a synonym for presence, as follows: “This allows users to experience a high degree of immersion; they feel as if they are in the virtual world, rather than looking at it.”  Presence was also described as an indicator of subjective immersion, for example in : “(...) presence is a subjective measure used in VR studies to quantify how immersed a user is in a VE.” Also problematic is the fact that authors use a single instrument to measure several different constructs. For example, immersion was measured using the Presence Questionnaire, the same instrument as that used to measure presence, which was also used to measure what authors’ labelled as engagement . Overall, the inconsistent and varying use of terms, as well as the use of single instruments to quantify different constructs presents a challenge for readers and should be addressed through the development of consistent terminology and a consensus on optimal outcome measures .
Among the studies in which a construct was measured, 44% of studies (N = 22) used validated instruments (e.g. IMI, IM-TEQ, PQ, ITQ, TSFQ, PACES and SFQ), however, most measures were not verified yet for the targeted purpose (e.g., the PACES), or population (e.g. PQ). Most used either indirect tools (e.g. taking training time or practice duration as a measure of motivation and engagement, N = 3), study-specific subjective questionnaires with untested psychometric properties (N = 15), or exclusively qualitative assessments (e.g. interviews or comments, N = 12) with varying rigour in data analysis (Table 4). Tatla et al. [22, 23] also found a lack of valid instruments used to measure motivation in pediatric interventions for children with cerebral palsy and acquired brain injury. As such, consensus is clearly required on instruments in order to align the field and facilitate interpretation and the advancement of knowledge. Existing instruments could be adapted and validated for use in VR/AVG interventions and with specific target populations. For example, Gil-Gómez et al.  have proposed the SEQ (Suitability Evaluation Questionnaire) that is based on the SFQ (Short Feedback Questionnaire) but has been updated to cover specific VR-related items.The use of direct or indirect objective measures of motivation, enjoyment, engagement is an option to overcome challenges of subjective self-report. Indirect measures include recording time spent interacting with the VR/AVG game (as undertaken by [19,20,21], counting the frequency of repetitions, or measuring the intensity of physical activity (for example, using EMG measurement, as in Zimmerli et al. who considered physical activity intensity as an indicator of engagement in VR/AVG interventions) . Clearly, this indirect approach is not without limitations, as there will always be a multitude of influences besides affective state on adherence, dosage and intensity (for example, the expectation of external rewards, or the pressure to maintain a strict treatment schedule). As such, more direct objective measures are also warranted . Examples include electroencephalography, including use of event related potentials to evaluate attentional demand , spectra analysis for indicators of engagement, or other measures such as galvanic skin response, heart rate variability or functional near-infrared spectroscopy . The use of such objective measures may elucidate the neurophysiological processes by which affective state influences motor learning .
Perez-Marcos  suggests that authors should distinguish between VR hardware and software to evaluate user experiences. Specifically, authors should be more specific about describing the components of their VR/AVG interventions to differentiate between systems, the games themselves, and the resulting user experience . Results of our review indicate that game mechanics such as rewards, feedback, challenge, choice/interactivity, clear goals, and socialization  were frequently lauded for their influence on motivation, engagement and enjoyment. These game design features are different from the features of the VR system that is delivering the intervention, and can likely be delivered across different platforms. Interestingly, authors did not link these game design features to immersion or presence, indicating that these constructs are more aligned with the game context than with the viewing medium or interaction modality. Further unpacking the ‘active ingredients’ of VR/AVG interventions, and how they may be attached to game characteristics as opposed to hardware components, is a key area for future research [92, 93].
Results of the review illustrate the discrepancy between the frequency of construct description or actual measurement. One potential explanation is that these constructs are universally accepted as inherent to VR/AVG interventions, and as such, researchers are not compelled to measure them. No conclusions can be made about the potential impact of motivation, enjoyment, engagement, degree of immersion and level of presence on the motor improvements achieved in a VE. We recommend including these analyses in future work, where power analyses permit. Such calculations should be facilitated as the field continues to grow and study designs move beyond the feasibility and pilot study stage in which authors’ focus on demonstrating an effect or differentiating the intervention from traditional care.
This scoping review had several limitations. We identified studies in which the apparent goal of VR/AVG interventions was motor skill improvement; however, the assumption of motor learning as an intervention goal was our own. We used summative content analysis to analyze article text, but did not record nor assign speculative or other intent to authors’ words. As such, and particularly since no inferential statistics were performed in the original articles to support identified relationships, we can assign no weight to relational links identified in this review. While our literature search included the three main rehabilitation-specific databases, literature may have been missed from other databases. In particular, we did not search the IEEE Xplore database, which may have led to more studies on immersion and presence, though perhaps not in a rehabilitation context. In keeping with scoping review conduct recommendations, we did not undertake a quality appraisal of the included studies.
The construct of ‘Flow’ was mentioned in relation to motivation, enjoyment, engagement, and immersion, e.g. by stating that “flow experience results from a combination of intrinsic motivation and complete immersion in the intervention”  and flow was often described as an indicator of engagement [72, 94]. As such, the omission of flow as a construct relevant to affective state in VR/AVG interventions is a scoping review limitation. Finally, we did not differentiate our analyses between non-customized and customized rehabilitation-specific VR/AVG systems. Non-customized systems are less expensive and accessible, may be easier to use and are most frequently used in clinical practice . Differentiating between these types of VR/AVGs may have helped to elicit any potential differences in the constructs that may be due to potentially more impactful game design principles (such as more abundant audiovisual feedback, or more explicit competition) of commercially-available games as compared to rehabilitation-specific games.
Next steps for research
Results of this scoping review indicate the need for greater consensus on definitions and terminology. Given the lack of psychometrically-valid outcome measures, integrating greater use of objective measures is essential. Researchers should include hypotheses as to how these constructs influence motor learning. High quality mixed methods research designs may be useful when appropriately conducted using a rigorous framework for design and interpretation , as a qualitative component can help to further elucidate what specifically participants found motivating or engaging, and can be used as a complement to explore the validity of self-report quantitative measures or objective measures. Finally, measuring sustainability and changes in these constructs over time can inform decision-making protocols for clinicians to better adjust VR/AVG intervention parameters to sustain motivation and engagement .
Greater understanding of the impact of affective state on learning will inform the design of VR interventions that can better exploit attributes found to promote motivation and engagement. Researchers can conduct experiments in VR to inform directions for development of VR-based therapeutic tasks, but they could also provide knowledge to inform conventional rehabilitation by providing greater awareness of the potential importance of affective state for learning. In addition, because VR experimental paradigms can better isolate or manipulate a single task presentation factor over others as compared to experiments in physical environments, this can support understanding of which specific factors enhance motivation and engagement for different types (e.g. ages, interests, cognitive abilities) of users. This can also provide more evidence for why therapists could consider using VR over traditional interventions as well as provide information for how to design conventional interventions that take advantage of these same attributes.
To accompany the increasing evidence of VR/AVG effectiveness in stroke rehabilitation, it is important to better understand factors that may differentiate certain systems or modulate effectiveness in clients with differing characteristics. The growing emphasis on the role of affective factors in motor learning combined with our findings that many researchers use these constructs as a rationale for VR/AVG use highlight the need to better understand and measure whether affective state differentiates VR/AVG use from traditional interventions and whether it contributes to intervention outcomes. This body of literature currently demonstrates a discrepancy between description and measurement, one that might be explained by the early stage of the literature and the current feasibility-oriented research methodologies. Results of the review provide suggestions for researchers interested in measuring these constructs and emphasize the need for consensus on terminology and outcome measures. Finally, the results point to the need to better understand, through improved measurement and inferential analyses, the potential impact of affective constructs and technical level of immersion on outcomes achieved through practice in VR environments.
Availability of data and materials
The datasets used and analysed during the current study are available from the corresponding author on reasonable request.
Active Video Game
Beck Depression Inventory
Intrinsic Motivation Inventory
Immersive Tendencies Questionnaire
Open Science Framework
Physical ACtivity Enjoyment Scale
Preferred Reporting Items for Systematic reviews and Meta-Analyses
Randomized Controlled Trial
Short Feedback Questionnaire
User Engagement Scale
Visual Analogue Scale
Laver KE, Lange B, George S, Deutsch JE, Saposnik G, Crotty M. Virtual reality for stroke rehabilitation. Cochrane Database Syst Rev. 2017;11:CD008349.
Lohse KR, Hilderman CG, Cheung KL, Tatla S, Van der Loos HM. Virtual reality therapy for adults post-stroke: a systematic review and meta-analysis exploring virtual environments and commercial games in therapy. PLoS One. 2014;9(3):e93318.
Staiano AE, Flynn R. Therapeutic uses of active videogames: a systematic review. Games Health J. 2014;3(6):351–65.
Saposnik G, Cohen LG, Mamdani M, Pooyania S, Ploughman M, Cheung D, et al. Efficacy and safety of non-immersive virtual reality exercising in stroke rehabilitation (EVREST): a randomised, multicentre, single-blind, controlled trial. Lancet Neurol. 2016;15(10):1019–27.
Levac D, Glegg S, Colquhoun H, Miller P, Noubary F. Virtual reality and active videogame-based practice, learning needs, and preferences: a cross-Canada survey of physical therapists and occupational therapists. Games Health J. 2017;6:2161–783X.
Glegg SM, Levac DE. Enhancing clinical implementation of virtual reality. In: Virtual Rehabilitation (ICVR), 2017 International Conference on: IEEE; 2017. p. 1509030530. https://ieeexplore.ieee.org/abstract/document/8007488.
Deutsch JE, McCoy SW. Virtual reality and serious games in neurorehabilitation of children and adults: prevention, plasticity, and participation. Pediatr Phys Ther. 2017;29:S23–36.
XRHealth. https://www.xr.health/. Accessed 23 Apr 2019.
Jintronix. http://www.jintronix.com/. Accessed 23 Apr 2019.
GestureTek Health. http://www.gesturetekhealth.com/solutions/rehabilitation. Accessed 23 Apr 2019.
Doctor Kinetic. https://doctorkinetic.com/. Accessed 23 Apr 2019.
Winstein C, Varghese R. Been there, done that, so what’s next for arm and hand rehabilitation in stroke? NeuroRehabilitation. 2018;43(1):3–18.
Rose T, Nam CS, Chen KB. Immersion of virtual reality for rehabilitation-review. Appl Ergon. 2018;69:153–61.
Lohse K, Shirzad N, Verster A, Hodges N, Van der Loos HM. Video games and rehabilitation: using design principles to enhance engagement in physical therapy. J Neurol Phys Ther. 2013;37(4):166–75.
Wulf G, Lewthwaite R. Optimizing performance through intrinsic motivation and attention for learning: the OPTIMAL theory of motor learning. Psychon Bull Rev. 2016;23(5):1382–414.
Lohse KR, Boyd LA, Hodges NJ. Engaging environments enhance motor skill learning in a computer gaming task. J Mot Behav. 2016;48(2):172–82.
Ryan RM, Deci EL. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am Psychol. 2000;55(1):68–78.
Burke JW, McNeill MDJ, Charles DK, Morrow PJ, Crosbie JH, McDonough SM. Optimising engagement for stroke rehabilitation using serious games. Vis Comput. 2009;25(12):1085–99.
Deutsch JE, Myslinski MJ, Kafri M, Ranky R, Sivak M, Mavroidis C, et al. Feasibility of virtual reality augmented cycling for health promotion of people poststroke. J Neurol Phys Ther. 2013;37(3):118–24.
King M, Hijmans JM, Sampson M, Satherley J, Hale L. Home-based stroke rehabilitation using computer gaming. N Z J Physiother. 2012;40(3):128–34.
Wittmann F, Held JP, Lambercy O, Starkey ML, Curt A, Hover R, et al. Self-directed arm therapy at home after stroke with a sensor-based virtual reality training system. J Neuroeng Rehabil. 2016;13(1):75.
Tatla SK, Sauve K, Virji-Babul N, Holsti L, Butler C, Van Der Loos HF. Evidence for outcomes of motivational rehabilitation interventions for children and adolescents with cerebral palsy: an American Academy for cerebral palsy and developmental medicine systematic review. Dev Med Child Neurol. 2013;55(7):593–601.
Tatla SK, Sauve K, Jarus T, Virji-Babul N, Holsti L. The effects of motivating interventions on rehabilitation outcomes in children and youth with acquired brain injuries: a systematic review. Brain Inj. 2014;28(8):1022–35.
Bryanton C, Bosse J, Brien M, McLean J, McCormick A, Sveistrup H. Feasibility, motivation, and selective motor control: virtual reality compared to conventional home exercise in children with cerebral palsy. CyberPsychol Behav. 2006;9(2):123–8.
Leiker AM, Miller M, Brewer L, Nelson M, Siow M, Lohse K. The relationship between engagement and neurophysiological measures of attention in motion-controlled video games: a randomized controlled trial. JMIR Serious Games. 2016;4(1):e4.
Merriam-Webster. https://www.merriam-webster.com/dictionary/enjoyment. Accessed 19 Dec 2018.
Lewis GN, Rosie JA. Virtual reality games for movement rehabilitation in neurological conditions: how do we meet the needs and expectations of the users? Disabil Rehabil. 2012;34(22):1880–6.
Weiss PL, Kizony R, Feintuch U, Katz N. Virtual reality in neurorehabilitation. Textbook of neural repair and rehabilitation 2006;51(8):182–197.
Slater M. A note on presence terminology. Presence connect. 2003;3(3):1–5.
Slater M. Measuring presence: a response to the Witmer and singer presence questionnaire. Presence-Teleop Virt. 1999;8(5):560–5.
Bohil CJ, Alicea B, Biocca FA. Virtual reality in neuroscience research and therapy. Nat Rev Neurosci. 2011;12(12):752–62.
Schuemie MJ, van der Straaten P, Krijn M, van der Mast CA. Research on presence in virtual reality: a survey. CyberPsychol Behav. 2001;4(2):183–201.
Colquhoun HL, Levac D, O'Brien KK, Straus S, Tricco AC, Perrier L, et al. Scoping reviews: time for clarity in definition, methods, and reporting. J Clin Epidemiol. 2014;67(12):1291–4.
Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.
Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5(1):69.
Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.
Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1.
Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.
Ballester BR, Badia SBI, Verschure PFMJ. Including social interaction in stroke VR-based motor rehabilitation enhances performance: a pilot study. Presence-Teleop Virt. 2012;21(4):490–501.
Reinthal A, Szirony K, Clark C, Swiers J, Kellicker M, Linder S. ENGAGE: guided activity-based gaming in neurorehabilitation after stroke: a pilot study. Stroke Res Treat. 2012;2012:784232.
Sampson M, Shau YW, King MJ. Bilateral upper limb trainer with virtual reality for post-stroke rehabilitation: case series report. Disabil Rehabil Assist Technol. 2012;7(1):55–62.
Fan SC, Su FC, Chen SS, Hou WH, Sun JS, Chen KH, et al. Improved intrinsic motivation and muscle activation patterns in reaching task using virtual reality training for stroke rehabilitation: a pilot randomized control trial. J Med Biol Eng. 2014;34(4):399–407.
Friedman N, Chan V, Reinkensmeyer AN, Beroukhim A, Zambrano GJ, Bachman M, et al. Retraining and assessing hand movement after stroke using the MusicGlove: comparison with conventional hand therapy and isometric grip training. J Neuroeng Rehabil. 2014;11:76.
Jordan K, Sampson M, King M. Gravity-supported exercise with computer gaming improves arm function in chronic stroke. Arch Phys Med Rehabil. 2014;95(8):1484–9.
Kottink AI, Prange GB, Krabben T, Rietman JS, Buurke JH. Gaming and conventional exercises for improvement of arm function after stroke: a randomized controlled pilot study. Games Health J. 2014;3(3):184–91.
Subramaniam S, Wan-Ying Hui-Chan C, Bhatt T. A cognitive-balance control training paradigm using wii fit to reduce fall risk in chronic stroke survivors. J Neurol Phys Ther. 2014;38(4):216–25.
Llorens R, Noe E, Colomer C, Alcaniz M. Effectiveness, usability, and cost-benefit of a virtual reality-based telerehabilitation program for balance recovery after stroke: a randomized controlled trial. Arch Phys Med Rehabil. 2015;96(3):418–25 e2.
Gorsic M, Novak D. Design and pilot evaluation of competitive and cooperative exercise games for arm rehabilitation at home. Conf Proc IEEE Eng Med Biol Soc. 2016;2016:4690–4.
Subramanian SK, Lourenco CB, Chilingaryan G, Sveistrup H, Levin MF. Arm motor recovery using a virtual reality intervention in chronic stroke: randomized control trial. Neurorehabil Neural Repair. 2013;27(1):13–23.
Flynn S, Palma P, Bender A. Feasibility of using the Sony PlayStation 2 gaming platform for an individual poststroke: a case report. J Neurol Phys Ther. 2007;31(4):180–9.
Song GB, Park EC. Effect of virtual reality games on stroke patients’ balance, gait, depression, and interpersonal relationships. J Phys Ther Sci. 2015;27(7):2057–60.
Chen MH, Huang LL, Lee CF, Hsieh CL, Lin YC, Liu H, et al. A controlled pilot trial of two commercial video games for rehabilitation of arm function after stroke. Clin Rehabil. 2015;29(7):674–82.
Schuck SO, Whetstone A, Hill V, Levine P, Page SJ. Game-based, portable, upper extremity rehabilitation in chronic stroke. Top Stroke Rehabil. 2011;18(6):720–7.
Szturm T, Peters JF, Otto C, Kapadia N, Desai A. Task-specific rehabilitation of finger-hand function using interactive computer gaming. Arch Phys Med Rehabil. 2008;89(11):2213–7.
Brokaw EB, Eckel E, Brewer BR. Usability evaluation of a kinematics focused Kinect therapy program for individuals with stroke. Technol Health Care. 2015;23(2):143–51.
Hoermann S, Ferreira Dos Santos L, Morkisch N, Jettkowski K, Sillis M, Devan H, et al. Computerised mirror therapy with augmented reflection technology for early stroke rehabilitation: clinical feasibility and integration as an adjunct therapy. Disabil Rehabil. 2017;39(15):1503–14.
Shin JH, Ryu H, Jang SH. A task-specific interactive game-based virtual reality rehabilitation system for patients with stroke: a usability test and two clinical experiments. J Neuroeng Rehabil. 2014;11:32.
Levin MF, Snir O, Liebermann DG, Weingarden H, Weiss PL. Virtual reality versus conventional treatment of reaching ability in chronic stroke: clinical feasibility study. Neurol Ther. 2012;1(1):3.
Wingham J, Adie K, Turner D, Schofield C, Pritchard C. Participant and caregiver experience of the Nintendo Wii sports after stroke: qualitative study of the trial of Wii in stroke (TWIST). Clin Rehabil. 2015;29(3):295–305.
Paquin K, Crawley J, Harris JE, Horton S. Survivors of chronic stroke - participant evaluations of commercial gaming for rehabilitation. Disabil Rehabil. 2016;38(21):2144–52.
Tsekleves E, Paraskevopoulos IT, Warland A, Kilbride C. Development and preliminary evaluation of a novel low cost VR-based upper limb stroke rehabilitation platform using Wii technology. Disabil Rehabil Assist Technol. 2016;11(5):413–22.
Lehmann I, Baer G, Schuster-Amft C. Experience of an upper limb training program with a non-immersive virtual reality system in patients after stroke: a qualitative study. Physiotherapy. 2017. https://doi.org/10.1016/j.physio.2017.03.001.
Stockley RC, O’Connor DA, Smith P, Moss S, Allsop L, Edge W. A mixed methods small pilot study to describe the effects of upper limb training using a virtual reality gaming system in people with chronic stroke. Rehabil Res Pract. 2017;2017:9569178.
Turkbey TA, Kutlay S, Gok H. Clinical feasibility of Xbox KinectTM training for stroke rehabilitation: a single-blind randomized controlled pilot study. J Rehabil Med. 2017;49(1):22–9.
Bower KJ, Clark RA, McGinley JL, Martin CL, Miller KJ. Clinical feasibility of the Nintendo Wii for balance training post-stroke: a phase II randomized controlled trial in an inpatient setting. Clin Rehabil. 2014;28(9):912–23.
Yong Joo L, Soon Yin T, Xu D, Thia E, Pei Fen C, Kuah CW, et al. A feasibility study using interactive commercial off-the-shelf computer gaming in upper limb rehabilitation in patients after stroke. J Rehabil Med. 2010;42(5):437–41.
da Silva Cameirao M, Bermudez IBS, Duarte E, Verschure PF. Virtual reality based rehabilitation speeds up functional recovery of the upper extremities after stroke: a randomized controlled pilot study in the acute phase of stroke using the rehabilitation gaming system. Restor Neurol Neurosci. 2011;29(5):287–98.
Lewis GN, Woods C, Rosie JA, McPherson KM. Virtual reality games for rehabilitation of people with stroke: perspectives from the users. Disabil Rehabil Assist Technol. 2011;6(5):453–63.
Crosbie JH, Lennon S, McGoldrick MC, McNeill MD, McDonough SM. Virtual reality in the rehabilitation of the arm after hemiplegic stroke: a randomized controlled pilot study. Clin Rehabil. 2012;26(9):798–806.
Yin CW, Sien NY, Ying LA, Chung SF, Tan May Leng D. Virtual reality for upper extremity rehabilitation in early stroke: a pilot randomized controlled trial. Clin Rehabil. 2014;28(11):1107–14.
Bower KJ, Louie J, Landesrocha Y, Seedy P, Gorelik A, Bernhardt J. Clinical feasibility of interactive motion-controlled games for stroke rehabilitation. J Neuroeng Rehabil. 2015;12:63.
Lee M, Pyun SB, Chung J, Kim J, Eun SD, Yoon B. A further step to develop patient-friendly implementation strategies for virtual reality-based rehabilitation in patients with acute stroke. Phys Ther. 2016;96(10):1554–64.
Hung JW, Chou CX, Hsieh YW, Wu WC, Yu MY, Chen PC, et al. Randomized comparison trial of balance training by using exergaming and conventional weight-shift therapy in patients with chronic stroke. Arch Phys Med Rehabil. 2014;95(9):1629–37.
Brown R, Burstin A, Sugarman H. Use of the Wii fit system for the treatment of balance problems in the elderly: a feasibility study. J Isr Phys Ther Soc (JIPTS). 2011;13(1):32.
Neil A, Ens S, Pelletier R, Jarus T, Rand D. Sony PlayStation EyeToy elicits higher levels of movement than the Nintendo Wii: implications for stroke rehabilitation. Eur J Phys Rehabil Med. 2013;49(1):13–21.
Llorens R, Gil-Gomez JA, Alcaniz M, Colomer C, Noe E. Improvement in balance using a virtual reality-based stepping exercise: a randomized controlled trial involving individuals with chronic stroke [with consumer summary]. Clin Rehabil. 2015;29(3):261–8.
Eng K, Siekierka E, Pyk P, Chevrier E, Hauser Y, Cameirao M, et al. Interactive visuo-motor therapy system for stroke rehabilitation. Med Biol Eng Comput. 2007;45(9):901–7.
Crosbie JH, McNeill MDJ, Burke J, McDonough S. Utilising technology for rehabilitation of the upper limb following stroke: the Ulster experience. Phys Ther Rev. 2013;14(5):336–47.
Proffitt RM, Alankus G, Kelleher CL, Engsberg JR. Use of computer games as an intervention for stroke. Top Stroke Rehabil. 2011;18(4):417–27.
Hale LA, Satherley JA, McMillan NJ, Milosavljevic S, Hijmans JM, King MJ. Participant perceptions of use of CyWee Z as adjunct to rehabilitation of upper-limb function following stroke. J Rehabil Res Dev. 2012;49(4):623–34.
Shiri S, Feintuch U, Lorber-Haddad A, Moreh E, Twito D, Tuchner-Arieli M, et al. Novel virtual reality system integrating online self-face viewing and mirror visual feedback for stroke rehabilitation: rationale and feasibility. Top Stroke Rehabil. 2012;19(4):277–86.
Broeren J, Claesson L, Goude D, Rydmark M, Sunnerhagen KS. Virtual rehabilitation in an activity Centre for community-dwelling persons with stroke. The possibilities of 3-dimensional computer games. Cerebrovasc Dis. 2008;26(3):289–96.
Samuel GS, Choo M, Chan WY, Kok S, Ng YS. The use of virtual reality-based therapy to augment poststroke upper limb recovery. Singap Med J. 2015;56(7):e127–30.
Llorens R, Colomer-Font C, Alcaniz M, Noe-Sebastian E. BioTrak virtual reality system: effectiveness and satisfaction analysis for balance rehabilitation in patients with brain injury. Neurologia. 2013;28(5):268–75.
Plant RW, Ryan RM. Intrinsic motivation and the effects of self-consciousness, self-awareness, and egoinvolvement: an investigation of internally controlling styles. J Pers. 1985;53(3):435–49.
Jungho P, David P, Hokyoung R. To flow and not to freeze: applying flow experience to Mobile learning IEEE transactions on Learning Technologies. 2010;3(1):56–67.
Mullen SP, Olson EA, Phillips SM, Szabo AN, Wojcicki TR, Mailey EL, et al. Measuring enjoyment of physical activity in older adults: invariance of the physical activity enjoyment scale (paces) across groups and time. Int J Behav Nutr Phys Act. 2011;8:103.
Gil-Gomez JA, Gil-Gomez H, Lozano-Quilis JA, Manzano-Hernandez P, Albiol-Perez S, Aula-Valero CSEQ. Suitability evaluation questionnaire for virtual rehabilitation systems. Application in a virtual rehabilitation system for balance rehabilitation. Int Conf Per Comp. 2013. https://doi.org/10.4108/icst.pervasivehealth.2013.252216:335-8.
Zimmerli L, Jacky M, Lunenburger L, Riener R, Bolliger M. Increasing patient engagement during virtual reality-based motor rehabilitation. Arch Phys Med Rehabil. 2013;94(9):1737–46.
Han CH, Hwang HJ, Lim JH, Im CH. Assessment of user voluntary engagement during neurorehabilitation using functional near-infrared spectroscopy: a preliminary study. J Neuroeng Rehabil. 2018;15(1):27.
Perez-Marcos D. Virtual reality experiences, embodiment, videogames and their dimensions in neurorehabilitation. J Neuroeng Rehabil. 2018;15(1):113.
Darekar A, McFadyen BJ, Lamontagne A, Fung J. Efficacy of virtual reality-based intervention on balance and mobility disorders post-stroke: a scoping review. J Neuroeng Rehabil. 2015;12(1):46.
Maier M, Rubio Ballester B, Duff A, Duarte Oller E, Verschure PF. Effect of specific over nonspecific VR-based rehabilitation on Poststroke motor recovery: a systematic meta-analysis. Neurorehabil Neural Repair. 2019;33(2):112–29.
Shin J-H, Bog Park S, Ho Jang S. Effects of game-based virtual reality on health-related quality of life in chronic stroke patients: a randomized, controlled study. Comput Biol Med. 2015;63:92–8.
Ostlund U, Kidd L, Wengstrom Y, Rowa-Dewar N. Combining qualitative and quantitative research within mixed method research designs: a methodological review. Int J Nurs Stud. 2011;48(3):369–83.
The authors thank Kayla Pinzur, Jasmine Zhang, and Marta Samokishyn for their assistance in charting the data.
This work was supported by the German Research Foundation (DFG) and the Technical University of Munich (TUM) in the framework of the Open Access Publishing Program. NR acknowledges support through a fellowship of the Bavarian State Ministry of Science and the Arts in the framework of the Centre Digitisation.Bavaria (ZD.B). DL is supported by a K01HD093838 Mentored Research Scientist Career Development (K01) award from the Eunice Kennedy Shriver National Institute of Child Health and Human Development (National Institutes of Health) and by a Pediatric Research Grant from the Charles H. Hood Foundation.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.