Skip to main content

A comparison of two personalization and adaptive cognitive rehabilitation approaches: a randomized controlled trial with chronic stroke patients

Abstract

Background

Paper-and-pencil tasks are still widely used for cognitive rehabilitation despite the proliferation of new computer-based methods, like VR-based simulations of ADL’s. Studies have established construct validity of VR assessment tools with their paper-and-pencil version by demonstrating significant associations with their traditional construct-driven measures. However, VR rehabilitation intervention tools are mostly developed to include mechanisms such as personalization and adaptation, elements that are disregarded in their paper-and-pencil counterparts, which is a strong limitation of comparison studies. Here we compare the clinical impact of a personalized and adapted paper-and-pencil training and a content equivalent and more ecologically valid VR-based ADL’s simulation.

Methods

We have performed a trial with 36 stroke patients comparing Reh@City v2.0 (adaptive cognitive training through everyday tasks VR simulations) with Task Generator (TG: content equivalent and adaptive paper-and-pencil training). The intervention comprised 12 sessions, with a neuropsychological assessment pre, post-intervention and follow-up, having as primary outcomes: general cognitive functioning (assessed by the Montreal Cognitive Assessment - MoCA), attention, memory, executive functions and language specific domains.

Results

A within-group analysis revealed that the Reh@City v2.0 improved general cognitive functioning, attention, visuospatial ability and executive functions. These improvements generalized to verbal memory, processing speed and self-perceived cognitive deficits specific assessments. TG only improved in orientation domain on the MoCA, and specific processing speed and verbal memory outcomes. However, at follow-up, processing speed and verbal memory improvements were maintained, and a new one was revealed in language. A between-groups analysis revealed Reh@City v2.0 superiority in general cognitive functioning, visuospatial ability, and executive functions on the MoCA.

Conclusions

The Reh@City v2.0 intervention with higher ecological validity revealed higher effectiveness with improvements in different cognitive domains and self-perceived cognitive deficits in everyday life, and the TG intervention retained fewer cognitive gains for longer.

Trial registration

The trial is registered at ClinicalTrials.gov, number NCT02857803. Registered 5 August 2016, .

Background

Cognitive rehabilitation after stroke

Stroke is a leading cause of long-term acquired disability in adults [1], predisposing patients toward institutionalization and poorer quality of life [2]. Over the coming decades, the incidence of post-stroke disability is expected to increase by 35% due to the rising prevalence of cerebrovascular risk and advances in medicine which are reducing post-stroke mortality rates [3]. Historically, stroke rehabilitation has been focused on motor rehabilitation [4, 5]. However, post-stroke cognitive deficits are pervasive causing disability with major impacts on quality of life and independence on everyday life activities [6, 7]. In the last years, attention to the impact of cognitive deficits has been growing [8] and finding new ways to improve cognition after stroke is considered a priority [9]. Also, more recently, the International Stroke Recovery and Rehabilitation Alliance 2018 working group has identified post-stroke cognitive impairments as a research priority [10].

Regardless of the many new developments in cognitive rehabilitation programs and applications, limited data on the effectiveness of cognitive rehabilitation is available because of the heterogeneity of participants, interventions, and outcome measures [11]. Results from recent reviews corroborate that cognitive rehabilitation has a positive impact on post-stroke cognitive outcomes [12, 13], although of small magnitude (Hedges’ g = 0.48) [12]. This result is in line with the quantitative [14] and qualitative [15,16,17] findings of previous reviews that have analyzed the effect of cognitive rehabilitation across multiple cognitive domains.

Is cognitive rehabilitation’s impact small or are we missing better cognitive rehabilitation methodologies?

Paper-and-pencil tasks are still the most widely used methods for cognitive rehabilitation because of their accessibility, ease of use, clinical validity and reduced cost [18]. In the last years, computer-based versions of these traditional tasks are also starting to become clinically accepted [19, 20]. However, there is an absence of specific methodologies that inform health professionals which tasks to apply and under what clinical conditions [21]. Consequently, rehabilitation professionals perform a selection of tasks based on their clinical experience, missing scientific foundations [22]. We have proposed an objective and quantitative framework for the creation of personalized cognitive rehabilitation tasks based on a participatory design strategy with health professionals [23]. In this work, through computational modeling, the authors operationalized 11 paper-and-pencil tasks and developed an Information and Communication Technologies based tool - the Task Generator (TG) - to tailor each of those 11 paper-and-pencil tasks to each patient in the domains of attention, memory, language and executive functions. A clinical evaluation of the TG with twenty stroke patients showed that the TG is able to adapt task parameters and difficulty levels according to patient’s cognitive assessment, and provide a comprehensive cognitive training [24]. However, although it has been shown that rehabilitation strategies based on paper-and-pencil tasks can be personalized and adapted [24, 25], this approach presents a limited transfer to performance in activities of daily living (ADL) [18].

Over the last years, rehabilitation methodologies based on virtual reality (VR) have been developed as promising solutions to improve cognitive functions [26, 27]. VR-based tools have shown potential and to be ideal environments to incorporate cognitive tasks within the simulation of ADL’s [28]. A recent trial with a VR-based simulation of everyday life activities (like going to the pharmacy, buying grocery at the supermarket, paying the water bill) suggested that an ecologically valid intervention has more impact than conventional methods (cognitive training using puzzles, calculus, problem resolution and shape sorting) in cognitive rehabilitation of stroke patients [29]. Also, some of these VR-based systems allow the integration of motor training [30] and recent studies have already shown benefits of performing simultaneous motor and cognitive training with stroke patients using VR [31, 32]. Yet, there is still an insufficient number of rigorous trials to clinically validate VR methods [12] and there are difficulties associated with the limited access which results in a low adoption by health professionals who still prefer mostly use paper-and-pencil interventions [33].

In general, existing ecologically-valid VR-based environments are simulations of cities [29, 34,35,36,37,38], kitchens [39,40,41,42,43,44,45], streets [46,47,48,49,50,51], supermarkets [52,53,54,55,56], malls and other shopping scenarios [57,58,59,60,61]. Of these, only rare cases take into account training personalization according to patient cognitive profile and session-to-session adaptation [29, 36, 38, 41]. Additionally, the results of studies comparing VR cognitive interventions with standard occupational therapy or neuropsychology cognitive paper-and-pencil training are fundamentally subjective as control interventions. OT does not consider cognition as the main training focus, and neuropsychology paper-and-pencil training tasks are too similar to the cognitive assessment scales; additionally, both approaches do not incorporate personalization and dynamic adaptation to performance. Hence, even if rehabilitation sessions last the same, these interventions are not equivalent as they are delivered with uncontrolled difficulty levels and cognitive demands. Personalized rehabilitation is defined as involving an assessment of each patient’s impairments and performing a tailored intervention to his cognitive profile in the different domains. Instead, adaptation deals with the dynamic adjustment of the tasks’ cognitive demands according to the patients’ performance along the intervention sessions, therefore avoiding boredom (tasks that are to easy to solve) or frustration (tasks that are too difficult to solve).

Here we try to address some of the existing limitations in the validation of VR-based cognitive rehabilitation tools. In this study we compared two task content equivalent rehabilitation tools developed under the same personalization and adaptation framework [23]: the TG and the Reh@City v2.0. This framework allows us to make sure that both tools deliver the same controlled adaptation and personalization of difficulty levels, and address the same cognitive demands. Hence, this comparison allows identifying the specific impact of increasing ecological validity of training through VR simulations of ADLs over the same training delivered through clinically accepted paper-and-pencil equivalent tasks. These findings will further inform on the specific benefits of ecologically valid environments delivered though VR and encourage the adoption of these technologies by health professionals.

Methods

Participants and trial design

Participants were selected based on the following inclusion criteria: no more than 75 years old; first stroke episode and at least at 6 months post-stroke (chronic phase); no hemi-spatial neglect as assessed by the clinicians with the Line Bisection test [62]; capacity to be seated; minimum of 2 years of schooling (since in Portugal there are quite high rates of illiteracy in the elder populations) and motivation to participate in the study. Patients with a total score of more than two standard deviations below the mean score for age and education in the Montreal Cognitive Assessment (MoCA) [63, 64] were excluded to ensure uniformity and enough cognitive capacity to participate in the rehabilitation interventions. Patients with severe depressive symptomatology, as assessed by the Beck Depression Inventory II [65, 66], were also excluded because its impact on cognitive functioning. Additionally, patients could not have been undergoing OT at least 2 months before the study. The study was previously approved by the Madeira Health Service Ethical Committee (reference number: 13/2016), and all the patients gave informed consent previous to participation.

The sample was selected from a list of 334 stroke patients enrolled in the cerebrovascular accidents appointment list from the Physical Medicine and Rehabilitation department of the Madeira Health Service (Portugal). They were contacted by phone by one of the researchers, and 44 declined to participate, 146 patients were excluded for not meeting the inclusion criteria and 108 were excluded for other reasons, such as transportation problems or lack of response after three phone calls. Overall, 36 patients were included meeting all inclusion criteria and were allocated to one of the two interventions (TG or Reh@City v2.0), by the two psychologists involved in the data collection. Group allocation was randomized (through a simple randomization method using a web-based application that generates a random allocation sequence) among the different rehabilitation units working under the Physical Medicine and Rehabilitation department (Fig. 1).

Fig. 1
figure1

Protocol of the intervention

Intervention protocol

The study started in January 2017 and stopped in December 2018, since the authors defined the maximum of 2 years for data collection. In total, 19 participants were allocated to the TG group (one dropped out) and 17 allocated to the Reh@City v2.0 group (three dropped out). All patients went through neuropsychological assessment pre and post-intervention and at 2 months follow-up. Each one of the assessment times had an approximate duration of 90 min.

The intervention personalization was done through the characterization of each participant with the MoCA [63, 64] assessment results: the Attention parameter was defined from MoCA’s attention component score [0–6]; the delayed recall and orientation scores [0–11] was used to parameterize memory; executive functions was parameterized through the sum of the visuospatial, executive, and abstraction MoCA subscores [0–7]; MoCA’s naming and the language scores [0–6] was used to parameterize language; and the total score [0–30] was used to parameterize the overall difficulty.

Two psychologists performed all the assessments. The same two psychologists and one occupational therapist supervised the interventions sessions. Accordingly, this study was not blind.

Interventions description

Paper-and-pencil intervention: the task generator

The TG is a free and worldwide accessible tool that is able to generate personalized paper-and-pencil cognitive rehabilitation programs in PDF format, composed by a set of 11 tasks gathered from clinical settings and parameterized through a participatory process with rehabilitation experts [24]: Cancellation; Numeric Sequences; Problem Resolution; Association; Comprehension of Contexts; Image Pairs; Word Search; Mazes; Categorization; Action Sequencing; and Memory of Stories (see Annex 1 for an example). After the characterization of each participant, the MoCA assessment data was normalized on a 1 to 10 scale, and a full training program was generated (Fig. 2). In the TG intervention, participants were instructed to use the arm they would feel more comfortable.

Fig. 2
figure2

TG training personalization parameters (on the left) and Association task generation example (on the right)

VR-based intervention: the Reh@City v2.0

Our VR-based intervention consisted of the same TG paper-and-pencil tasks contextualized in different locations of a virtual city with streets, sidewalks, buildings, shops, and parks – the Reh@City v2.0 [67] (Table 1).

Table 1 TG paper-and-pencil tasks correspondence with Reh@City v2.0 VR tasks

Reh@City v2.0 provides a more ecological training experience since patients are required to solve cognitive tasks through familiar ADL’s in a variety of commonplaces: for instance buy food in a supermarket (Fig. 3a); pick up a package in the post office; pay the electricity at the bank ATM (Fig. 3b); buy pain killers in the pharmacy; collect shirts in the clothing shop; play a game in the park (Fig. 3c); read the newspaper in the kiosk and set the table at home (Fig. 3d). These places display billboards and real products of actual spaces and trademarks commonly found in Portugal to help the patient relate the VR tasks to the real world. In addition, consistent with the actual simulated ADLs and to increase the ecological validity of the training, patients were also required to use their paretic arm to solve the tasks.

Fig. 3
figure3

Reh@City v2.0 task examples: a buying food in the supermarket; b making payments at the bank ATM; c playing a cards game at the park and; d setting the table at home

Because we were generally dealing with people of older age and low computer literacy, the interaction with the virtual environment was simplified and city was designed to have only square or rectangular building blocks and perpendicular street intersections, as well as simplified simulated environments. This simplified arrangement also allowed a more precise control of difficulty parameterization (Fig. 4).

Fig. 4
figure4

Reh@city v2.0 three-dimensional street view. Users are given goal instructions supported with a mini-map indicating the optimal path and a street arrow. Time and point counters are used to provide feedback on performance

VR simulations demand a trade-off of less realism to allow more real-time interactivity and difficulty parameterization. Notwithstanding this simplification, Reh@City has many elements attesting for its ecological validity. In terms of verisimilitude, the tasks simulate everyday life activities, although the performance of the virtual task is facilitated; the real environment and objects are minimally simulated and correspond to familiar locations and trademarks existing in Portugal; the user’s upper limb is partially represented; the specific situations in which actions take place are minimally modeled (for instance, action sequencing of baking a cake happens in a kitchen environment and reading a newspaper article happens in kiosk); it combines 2D, and 3D stimuli and; the navigation in the city is consistent with the point of view of the user. Concerning veridicality, in a previous study with Reh@City v1.0 [29], we have found a transfer of the virtual training to real-world functioning through significant improvements on a functional scale.

TG and Reh@City personalization and adaptation

The TG and the Reh@City v2.0 are two content equivalent rehabilitation tools developed under the same personalization algorithm [23]:

$$ DV= intercept+\mathrm{C}1\ast \mathrm{IV}1+\mathrm{C}2\ast \mathrm{IV}2+\dots +\mathrm{C}\mathrm{i}\ast \mathrm{IV}\mathrm{i} $$

where Ci indicates the contribution of each Independent Variable (IV) (task parameters) to the Dependent Variable (DV) (memory, executive functions, attention, language, and difficulty).

Concerning the session-to-session adaptation, when the patient finishes the first set of tasks, a score is computed using a 0 to 100% scale. Consistent with previous adaptive systems for stroke rehabilitation [68], if the mean performance is higher than 70%, the difficulty is increased by 0.5 in the next set of tasks, and if performance is from 0 to 50%, the difficulty parameter is reduced by 0.5.

In both personalization and adaptation, difficulty is one of the DV we have obtained from our algorithm [23], it is related with general cognitive demand.

Experimental setup

Task generator (TG)

The TG is an online application, accessible at neurorehabilitation.m-iti.org/TaskGenerator, and does not require to be installed on the computer. Through this tool, clinicians defined appropriate parameters of training for memory, attention, executive functions, language, and difficulty (obtained from the MoCA screening tool), and it automatically generated a set of the 11 personalized cognitive training tasks. The only required software was a PDF reader to open the downloaded paper-and-pencil tasks. After printed, the tasks were solved on a table with a pencil having the user seated.

Reh@City v2.0

Reh@City v2.0 was installed on a PC (OS: Windows 7, CPU: Intel core 2 duo E8235 at 2.80 GHz, RAM: 4 Gb, Graphics: ATI mobility Radeon HD 2600 XT). Reh@City is a cognitive rehabilitation tool, that allow clinicians to personalize parameters of training for memory, attention, executive functions, language, and difficulty according to the MoCA total and subdomains score (Fig. 5) and it enables that through upper limb movements.

Fig. 5
figure5

Reh@City v2.0 training personalization parameters according to MoCA total and subdomains score

Given the potential benefits reported in the literature of combining cognitive and motor rehabilitation through VR [30,31,32], Reh@City v2.0 implies the use of the paretic arm to solve its cognitive training tasks. The user worked on a tabletop, facing an LCD monitor (24″) and moved a customized handle with a tracking pattern on the surface of the table with his/her paretic arm (Fig. 6). 2D upper limb reaching movements were captured through a camera-based Augmented Reality (AR) pattern tracking software (AnTS) [69] connected to a PlayStation Eye camera (Sony Computer Entertainment Inc., Tokyo, Japan). For adapting the interaction to individual users, the Reh@City v2.0 implemented a built-in calibration function that normalizes the motor effort required in the task to the active range of movement of the user. The movements of the user are then mapped onto the movements of a virtual arm (in indoor tasks) or as movement directions (during outdoor navigation) in the Reh@City v2.0 environment.

Fig. 6
figure6

Reh@City v2.0 experimental setup. The user faces an LCD monitor and moves a handle on the surface of the table with his/her paretic arm to interact with the virtual content

Outcome measures

Primary outcome measures: general cognitive functioning, attention, memory, executive functions and language

As primary outcome measures we used the MoCA [63, 64] as a general cognitive functioning measure, which has been reported to have a good sensitivity and specificity in screening for cognitive impairment after stroke [70]. In addition, the decline in MoCA scores (reduction ≥2 points) was found to be associated with the decline in neuropsychological diagnosis transitional status on a sample of 275 stroke patients [71].

Moreover, we selected specific attention, memory, executive functions, and language assessments, which are the domains targeted by both TG and Reh@City v2.0. To assess attention we used the Trail Making Test A and B (TMT A and B) [72, 73], a very popular neuropsychological test that provides information on visual search, visual scanning, selective and divided attention, processing speed, mental flexibility, and also executive functioning. In part A, circles numbered from 1 to 25 need to be connected in numerical order. In part B, numbers from 1 to 13 and letters from A to L need to be connected alternating numbers and letters in ascending order. The memory assessment was performed with the Verbal Paired Associates from the Wechsler Memory Scale-III (WMS-III) [74]. To assess executive functions, namely working memory and processing speed, we used the Digit Span (forward and backward recall conditions) also from the WMS-III, and the Symbol Search and the Digit Symbol Coding (codification and incidental learning pairing conditions) from the Wechsler Adult Intelligence Scale III (WAIS) [75]. Finally, we assessed language through the Vocabulary from WAIS-III, which provides information about verbal comprehension.

Secondary outcome measure: self-perceived impact of cognitive functioning problems

As secondary outcome measure, we assessed the perceived impact of persisting problems with cognition, as assessed by the Patient-Reported Evaluation of Cognitive State (PRECiS) [76, 77], which includes 27 core items asking respondents about the impact of cognition on four conceptual dimensions: everyday life skills, family and life, mood and sense of self.

Statistical analysis

All statistical analyses were performed using SPSS software (version 20, SPSS Inc., Chicago IL, USA). As a criterion for significance, we used a α of .050. With Bonferroni correction the α was .002, as such significant differences with corrected p-values are also mentioned. Normality of data was assessed with the Kolmogorov-Smirnov (KS) test. As some data were not normally distributed, nonparametric tests were used to evaluate the inter-group and intra-group differences. The Wilcoxon signed-rank test (W) was used to analyze the within group changes over time, while the two-tailed Mann-Whitney (MW) test was used to compare the between-group differences from baseline to the end of the study. Demographic differences between groups were measured with the Mann Whitney test (MW). Effect sizes (r) were computed as Z/√N on the pairwise comparisons. The criteria for interpretation of the effect was 0.1 = small, 0.3 = medium, and 0.5 = large.

Results

Sample description

The sample consisted of thirty-two patients with stroke randomly distributed in two groups. The Reh@City v2.0 group comprised fourteen (5 male, 9 female) adult (M = 59.1 years old, SD = 11.8) patients with stroke (11 right hemisphere, 3 left hemisphere; 12 ischemic, 2 hemorrhagic), with an average of 45.9 ± 43.6 months post- stroke and a mean of 8 ± 5.3 years of schooling. The TG group comprised eighteen (11 male, 7 female) adult (M = 65 years old, SD = 6.2) patients with stroke (9 right hemisphere, 6 left hemisphere, 3 not specified; 14 ischemic, 3 hemorrhagic, 1 not specified), with an average of 21.3 ± 12.9 months post-stroke and a mean of 5.5 ± 3.2 years of schooling. The Mann-Whitney test revealed no differences between groups in the demographic characteristics and in all baseline outcome measures (Table 2).

Table 2 Demographic characteristics (presented as Means ± SD’s) of the two groups and differences between groups measured by the Mann-Whitney test (MW)

According to the Kolmogorov-Smirnov (KS) test, data were normally distributed in both groups for age (KSReh@City = .189, p = .190; KSTG = .182, p = .118) and time post-stroke (KSReh@City = .211, p = .091; KSTG = .187, p = .095). Data were not normally distributed for gender (KSReh@City = .407, p = <.001; KSTG = .392, p = <.001), type of stroke (KSReh@City = .510, p = <.001; KSTG = .463, p = <.001), the side of lesion (KSReh@City = .510, p = <.001; KSTG = .463, p = <.001), and the number of years of schooling (KSReh@City = .345, p = < .001; KSTG = .405, p = < .001).

Primary outcome measures

MoCA - general cognitive functioning

We analyzed the global cognitive functioning, as assessed by the MoCA, of the two groups in the pre and post intervention assessments, and follow-up for the Reh@City v2.0 and TG groups (Table 3).

Table 3 MoCA scores (presented as Medians and IQR) pre and post intervention and follow-up highlighted for within-groups significant differences and marked with an asterisk for between-groups significant differences

A Wilcoxon test for within-groups differences revealed that only the Reh@City v2.0 group presented significant statistical improvements between pre and post assessment times in MoCA [Pre: Mdn = 23, IQR = 19.8–26; Post: Mdn = 25, IQR = 23–27.3 (W(14) = 64.00, Z = − 2.777, p = .005, r = .74)]. In the subdomains analysis, we found significant improvements in visuospatial ability and executive functioning [Pre: Mdn = 3.5, IQR = 2.8–4; Post: Mdn = 4, IQR = 3–5 (W(14) = 41.00, Z = − 2.310, p = .021, r = .62)] and attention [Pre: Mdn = 4, IQR = 2.8–5.3; Post: Mdn = 5.5, IQR = 3–6 (W(14) = 28.00, Z = − 2.460, p = .014, r = .66)].

Concerning the TG group, the only significant change was in the MoCA orientation subdomain [Pre: Mdn = 6, IQR = 5–6; Post: Mdn = 6, IQR = 6–6 (W(18) = 15.00, Z = − 2.121, p = .034, r = .57)].

A Mann-Whitney test indicated that the Reh@City v2.0 group improved significantly more than the TG group, in terms of general cognitive functioning, as assessed by the MoCA, from baseline to post-intervention [Reh@City v2.0: Mdn = 2, IQR = 0–3; TG: Mdn = − 1.5, IQR = − 3.25–2 (U = 65.00, Z = − 2.334, p = .020, r = .41)].

TMT A & B - attention

We computed the TMT A and TMT B performance for the two groups, in terms of errors and completion time, pre, post-intervention and follow-up (Table 4). Only the TG group showed a significant improvement in the reduction of time to completion of the TMT A test pre to post-intervention [Pre: Mdn = 84, IQR = 59.5–114.3; Post: Mdn = 72, IQR = 58.8–99.5 (W(18) = 18.00, Z = − 2.588, p = .010, r = .61)].

Table 4 TMT A and B; WMS-III Verbal Paired Associates (VPA); and WAIS-III Digit Symbol Coding (DSC), Symbol Search, Digit Span and Vocabulary scores (presented as Medians and IQR) pre and post intervention and follow-up highlighted for within-groups significant differences

WMS-III verbal paired associates - memory

Table 4 describes the Verbal Paired Associates test performance for the two groups pre, post-intervention and follow-up. In this learning and memory test, we found significant improvements within the Reh@City v2.0 group for the retention [Pre: Mdn = 75, IQR = 0–100; Post: Mdn = 100, IQR = 74.1–100 (W(14) = 36.00, Z = − 2.524, p = .012, r = .67)] and recognition [Pre: Mdn = 24, IQR = 21.8–24; Post: Mdn = 24, IQR = 24–24 (W(14) = 21.00, Z = − 2.214, p = .027, r = 59)] scores post-intervention. In the TG group improvements were only significant for the retention score in both post-intervention [Pre: Mdn = 0, IQR = 0–56.3; Post: Mdn = 82.9, IQR = 26.5–100 (W(18) = 118.00, Z = − 2.602, p = .009, r = .61)] and follow-up [Pre: Mdn = 0, IQR = 0–56.3; FU: Mdn = 82.9, IQR = 37.5–100 (W(18) = 95.00, Z = − 2.776, p = .006, r = .65)].

WAIS-III Digit Symbol Coding, Symbol Search and Digit Span - Executive Functions.

Table 4 describes the executive functioning outcome measures for the two groups pre, post-intervention and follow-up. The Reh@City v2.0 group showed improvements in the Digit Symbol Coding codification task post-intervention [Pre: Mdn = 28.5, IQR = 23.5–36.8; Post: Mdn = 33, IQR = 26.8–47 (W(14) = 87.00, Z = − 2.171, p = .030, r = .58)]. The TG group had significant improvements in the Symbol Search at follow-up [Pre: Mdn = 12, IQR = 7.8–13.5; FU: Mdn = 15, IQR = 9–20.3 (W(18) = 101.00, Z = − 2.340, p = .019, r = .55)].

WAIS-III vocabulary - language

The analysis of the language outcome measure for the two groups pre, post-intervention and follow-up revealed that only the TG group showed improvements in the Vocabulary assessment at follow-up [Pre: Mdn = 19.5, IQR = 13–28.5; FU: Mdn = 24.5, IQR = 16.5–30.3 (W(18) = 166.00, Z = − 3.514, p < .001, r = .83)] (Table 4). This difference was also significant with Bonferroni correction (p = .002).

Secondary outcome measure

Patient-reported evaluation of cognitive state

When analyzing the answers of the two groups pre, post-intervention and follow-up to the PRECiS questionnaire, only the Reh@City v2.0 group revealed a significant self-perceived decrease in the stroke cognitive deficits impact post intervention [Pre: Mdn = 13.5, IQR = 7–23.8; Post: Mdn = 12, IQR = 3.8–21.3 (W(14) = 13.00, Z = −2.041, p = .041, r = .55)] (Table 5).

Table 5 PRECiS score (presented as Medians and IQR) pre and post intervention and follow-up highlighted for within-groups significant differences

Discussion

In the last years there has been significant growth in the evidence for post-stroke cognitive rehabilitation [12, 13], with a number of studies proposing ecologically valid VR-based simulations of ADL’s as the most promising training solutions [26, 27]. Notwithstanding the advantages of VR, there are several areas that require further progress. One is the need to bridge widely accepted paper-and-pencil methodologies with VR-based ADL’s simulations. In the field of cognitive assessment, some studies have compared VR cognitive assessment tools with their paper-and-pencil original versions [78,79,80]. Existing studies about VR simulations of cities showed convergent validity between some measures of the performance in simulated virtual cities and clinical neuropsychological tests of variable strength, which ranges from moderate for attention [37], good to excellent for executive functioning [37], and excellent for general cognitive condition [28]. Poor to moderate correlations have also been reported between navigation in a real and a virtual city [34]. If we look at cognitive training, to the best of our knowledge, there is not much work that we can compare with our study. In terms of VR-based interventions, Gamito and colleagues used a virtual city with everyday life simulations with stroke patients and have found improvements in attention and memory [36]. Related with Task Generator paper-and-pencil training, there are the Guttman Neuropersonal Trainer [19] and the Cogweb [20] that are customizable and have already some evidence about its efficacy in cognitive training.

Another limitation about VR-based cognitive rehabilitation tools is that, an important number of the developed systems have not been field tested [38], have only gone through studies with a small number of participants [29, 36] and/or with healthy control groups [78]. Additionally, none of these VR-based ADL’s simulations are compared with widely used and clinically accepted paper-and-pencil tasks, being always compared with non-equivalent interventions as OT [81]. Our RCT, is the first to implement an adaptive paper-and-pencil training and compare it with a content equivalent VR-based ADL’s simulation, by using the same tasks, personalization and difficulty adaptation framework within a longitudinal clinical intervention.

TG and the Reh@City v2.0 groups differed in the motor demand of the training; while the TG group used a pencil with the healthy arm to solve the tasks, the Reh@City v2.0 group used an adapted handle with the affected arm, performing 2D upper limb reaching movements to interact with the tasks. Assuming the interdependence between the recovery processes [32], we may have provided a more effective rehabilitation to the Reh@City v2.0 group in both cognitive and motor domains; however, motor recovery is not an outcome in this study and, consequently, was not assessed. The Reh@City v2.0 main goal is to be an ecologically valid cognitive training tool, and the use and virtual representation of the paretic arm to solve the ADL’s simulations was implemented to increase ecological validity. As such, the difference in motor demand is intrinsic to the research hypothesis. If we asked the TG group to use the paretic arm, we would be harming the resolution of the paper-and-pencil tasks, thereby, Reh@City v 2.0 and TG interventions could not be fairly compared.

Primary outcome measures

The Reh@City v2.0 group improved in the MoCA general cognitive functioning and in its attention, visuospatial ability and executive functioning subdomains, with large effect sizes. The TG group improved in the MoCA orientation domain, also with a large effect size. In a between groups comparison, the Reh@City v2.0 group had a higher impact in the general cognitive functioning comparatively to the TG group, with a medium effect size. Wong and colleagues (2017) determined that the MoCA score change associated with a change of health in general, in a sample of 175 aneurysmal subarachnoid hemorrhage patients, was of two points [82]. The generalization of this finding to our study is consistent with this between-groups difference, since the Reh@City v2.0 group improves its median score from 23 to 25 points in the post-intervention, while the TG maintains the median score of 21 points. At the follow-up, Reh@City v2.0 improves its median score to 28 points, while the TG has a more discrete improvement to 23 points. Meaning that the Reh@City v2.0 group was the only to present Minimum Clinically Important Differences (MCID) from baseline to end, and both from end to follow-up.

Despite the fact that both Reh@City v2.0 and TG are content equivalent interventions that follow the same personalization guidelines and difficulty progression rules, the superiority of the impact of the VR-based intervention was expected since it reunites a number of promising features that distinguishes it from the paper-and-pencil intervention: 1) it is an ecologically valid ADL’s simulations; 2) stimulus are everyday-life brands and products; 3) it has game elements so participants are rewarded for successful performance in real time and; 4) it provides immediate feedback enabling higher success in accomplishing the tasks. Although with a different primary outcome measure, these results are coherent with a previous Reh@City v1.0 study, where the Reh@City v1.0 group improved in the Addenbrooke Cognitive Examination [83] general cognitive functioning, attention, memory and visuospatial abilities and was superior between groups in general cognitive functioning, attention and fluency (where the control group had a significant decrease) [29]. Reh@City v1.0 was an initial prototype with four commonly frequented places of daily life (Pharmacy, Bank, Supermarket, and Post-office) [28]. The upgrade to Reh@City v2.0 included: an increase of the ecological validity through the improvement in the overall visual realism of the city and existing tasks, and the interaction through the paretic arm using an adapted handle; the implementation of dynamic difficulty adaptation based on the framework we developed for TG tasks [23]; and the increase of the number of cognitive training tasks and locations (Magazine Kiosk, Home, Park, and Fashion Store) [67].

Also, the Reh@City v2.0 intervention group had a significant impact, and with large effect sizes, in verbal memory (as assessed by the retention and recognition from the Verbal Paired Associates test - WMS-III), and processing speed (as assessed by the Digit Symbol Coding task - WAIS-III), which is superior to what we have found in the Reh@City v1.0 study, where we only had improvements in the executive functioning measure (the Picture Arrangement test from the WAIS-III) and the control group improved in the reduction of the number of errors in the TMT B, a processing speed and selective and divided attention measure [29]. This superiority of the Reh@City v2.0 concerning verbal memory and processing speed was expected since it has an increased number of cognitive training tasks and locations in comparison to Reh@City v1.0, that did not even had verbal memory training.

The TG group improved in verbal memory (as assessed by the retention from the Verbal Paired Associates test - WMS-III) and processing speed (as assessed by the TMT A task execution time) subdomains, with large effect sizes. At follow-up, participants who underwent the TG intervention maintained the verbal memory benefits with new improvements in the sustained attention and processing speed (as assessed by the Symbol Search task – WAIS-III) and language (as assessed by the Vocabulary – WAIS-III) domains, also with large effect sizes. These findings may be related to the fact that the TG offers a more domain-specific training and recent evidence supports that attention, language [14], memory, executive functions and visuospatial and perceptual skills [13] training after stroke is effective.

Secondary outcome measure

Besides cognition, we assessed the intervention’s impact in the self-perceived cognitive deficits. Only the Reh@City v2.0 group had a significant reduction in the self-perceived cognitive deficits in different aspects of their everyday life (everyday life skills, family and life, mood and sense of self), measured by the PRECiS questionnaire with a large effect size. This finding is in line with existing literature that states that, comprehensive neuropsychological rehabilitation, is effective to reduce functional disability after stroke [14]. In the Reh@City v1.0 study we assessed the intervention’s impact in the multiple domains of health and life with a different outcome measure - the Stroke Impact Scale 3.0 [84]. In this self-report questionnaire, the VR group improved significantly in the physical domain, memory, emotion, social participation and overall recovery while the control group decreased in the physical domain, only improving in memory, mobility and social participation, which supported the superiority of the intervention with the Reh@City v1.0 in comparison to the control group.

Limitations

Some limitations of our study must be considered when interpreting the results. Concerning the sample, although 42 stroke participants is a small sample it is comparable to previous similar clinical trials [29, 36]. In the perspective of the interventions comparison, Reh@City v2.0 and TG have several intrinsic differences mainly related with the fact that cognitive tasks are presented differently (through VR and paper-and-pencil, respectively). Besides the novelty carried by the use of VR technology in Reh@City v2.0, it requires the subject to use its paretic upper limb while TG does not, as it might be hard or even impossible for a participant to properly use a pencil to solve cognitive tasks with his paretic arm. Although the discrepancies inherent to both interventions do not allow us to know if Reh@City v2.0 impact superiority is related to the use of VR, to the ecological validity of the tasks, or the integration of motor and cognitive training, the main objective of this study was to compare Reh@City v2.0 with a content equivalent intervention that follows the same personalization guidelines and difficulty progression rules (the TG), acknowledging their differences. Also, in the Reh@City v2.0 group, most participants were lost at follow-up. Hence, the comparison between TG and Reh@City v2.0 at this assessment moment should be considered with caution. Furthermore, the intervention was not blind since the same persons performed the assessments and interventions. Also, there might have been learning effects of the cognitive assessment tools, at post-intervention and follow-up assessment times, since only the MoCA had parallel versions for multiple assessments. Yet, even if a learning effect existed, this would apply to both groups, and the comparison would still be valid.

Conclusions

The results of this one-month longitudinal study showed a positive impact of a rehabilitation training with the Reh@City v2.0, an ecologically valid VR ADL’s simulations, in general cognitive functioning, visuospatial ability and executive functioning, attention, verbal memory and processing speed, generalized for other health and life aspects measured by the self-perceived impact of cognitive deficits scale. This generalization did not happen in the TG group that only revealed similar cognitive impact in the orientation, processing speed and verbal memory domains. The TG intervention sustained impact at follow-up, maintaining processing speed and verbal memory improvements and revealing a new one in language. Finally, by comparing interventions between themselves, we have found Reh@City v2.0 to be superior in general cognitive functioning, visuospatial ability and executive functioning. Only the intervention with TG allowed cognitive gains to last over time. However, these results need to be considered with caution given the dropout at follow-up in the Reh@City v2.0 group.

Overall, our results contribute with new evidence about the impact of ecological validity - using personalization and adaptation in VR simulations of ADL’s and paper-and-pencil tasks - in the rehabilitation of cognitive deficits, which can facilitate the adoption of these innovative tools by health professionals in their daily practice. Nevertheless, there is still a need for further research considering other clinical populations, as well as the implementation of a wider variety of cognitive training tasks.

Availability of data and materials

The central tendency and dispersion data, as well as the Mann-Whitney and the Wilcoxon tests, from which the conclusions are drawn, are provided in the article. Raw data is available from the corresponding author on reasonable request.

Abbreviations

ADL’s:

Activities of Daily Living

KS:

Kolmogorov-Smirnov

MoCA:

Montreal Cognitive Assessment

MW:

Mann-Whitney

PRECiS:

Patient-Reported Evaluation of Cognitive State

RCT:

Randomized Controlled Trial

TG:

Task Generator

TMT:

Trail Making Test

VR:

Virtual Reality

W:

Wilcoxon

WAIS:

Wechsler Adult Intelligence Scale

WMS:

Wechsler Memory Scale

References

  1. 1.

    Feigin VL, Norrving B, Mensah GA. Global burden of stroke. Circ Res. 2017;120(3):439–48.

    CAS  PubMed  Article  Google Scholar 

  2. 2.

    Ankolekar S, Renton C, Sare G, Ellender S, Sprigg N, Wardlaw JM, et al. Relationship between poststroke cognition, baseline factors, and functional outcome: data from “efficacy of nitric oxide in stroke” trial. J Stroke Cerebrovasc Dis. 2014;23(7):1821–9.

    PubMed  Article  Google Scholar 

  3. 3.

    Bennett JE, Stevens GA, Mathers CD, Bonita R, Rehm J, Kruk ME, et al. NCD countdown 2030: worldwide trends in non-communicable disease mortality and progress towards sustainable development goal target 3.4. Lancet. 2018;392(10152):1072–88.

    Article  Google Scholar 

  4. 4.

    Dimyan MA, Cohen LG. Neuroplasticity in the context of motor rehabilitation after stroke. Nat Rev Neurol. 2011;7(2):76–85.

    PubMed  PubMed Central  Article  Google Scholar 

  5. 5.

    Ganguly K, Byl NN, Abrams GM. Neurorehabilitation: motor recovery after stroke as an example. Ann Neurol. 2013;74(3):373–81.

    PubMed  Article  Google Scholar 

  6. 6.

    Cumming TB, Marshall RS, Lazar RM. Stroke, cognitive deficits, and rehabilitation: still an incomplete picture. Int J Stroke. 2012;8(1):38–45.

    Article  Google Scholar 

  7. 7.

    Tang EY, Amiesimaka O, Harrison SL, Green E, Price C, Robinson L, et al. Longitudinal effect of stroke on cognition: a systematic review. J Am Heart Assoc. 2018;7(2):e006443.

    PubMed  PubMed Central  Article  Google Scholar 

  8. 8.

    Andrew NE, Kilkenny M, Naylor R, Purvis T, Lalor E, Moloczij N, et al. Understanding long-term unmet needs in Australian survivors of stroke. Int J Stroke. 2014;9(SA100):106–12.

    PubMed  Article  Google Scholar 

  9. 9.

    Pollock A, St George B, Fenton M, Firkins L. Top 10 research priorities relating to life after stroke – consensus from stroke survivors, caregivers, and health professionals. Int J Stroke. 2014;9(3):313–20.

    PubMed  Article  Google Scholar 

  10. 10.

    Bernhardt J, Borschmann KN, Kwakkel G, Burridge JH, Eng JJ, Walker MF, et al. Setting the scene for the second stroke recovery and rehabilitation roundtable. Int J Stroke. 2019. https://doi.org/10.1177/1747493019851287.

  11. 11.

    Sohlberg MM, Mateer CA. Cognitive rehabilitation: an integrative neuropsychological approach. New York: Guilford; 2001.

    Google Scholar 

  12. 12.

    Aminov A, Rogers JM, Middleton S, Caeyenberghs K, Wilson PH. What do randomized controlled trials say about virtual rehabilitation in stroke? A systematic literature review and meta-analysis of upper-limb and cognitive outcomes. J Neuroeng Rehabil. 2018;15(1):29.

    PubMed  PubMed Central  Article  Google Scholar 

  13. 13.

    Rogers JM, Foord R, Stolwyk RJ, Wong D, Wilson PH. General and domain-specific effectiveness of cognitive remediation after stroke: systematic literature review and Meta-analysis. Neuropsychol Rev. 2018;28(3):285–309.

    PubMed  Article  Google Scholar 

  14. 14.

    Cicerone K, Goldin Y, Ganci K, Rosenbaum A, Wethe J, Langenbahn D, et al. Evidence-based cognitive rehabilitation: systematic review of the literature from 2009 through 2014. Arch Phys Med Rehabil. 2019;S0003-9993(19):30194–7.

    Google Scholar 

  15. 15.

    Cicerone KD, Dahlberg C, Kalmar K, Langenbahn DM, Malec JF, Bergquist TF, et al. Evidence-based cognitive rehabilitation: recommendations for clinical practice. Arch Phys Med Rehabil. 2000;81(12):1596–615.

    CAS  PubMed  Article  Google Scholar 

  16. 16.

    Cicerone KD, Dahlberg C, Malec JF, Langenbahn DM, Felicetti T, Kneipp S, et al. Evidence-based cognitive rehabilitation: updated review of the literature from 1998 through 2002. Arch Phys Med Rehabil. 2005;86(8):1681–92.

    PubMed  Article  Google Scholar 

  17. 17.

    van Heugten C, Gregório GW, Wade D. Evidence-based cognitive rehabilitation after acquired brain injury: a systematic review of content of treatment. Neuropsychol Rehabil. 2012;22(5):653–73.

    PubMed  Article  Google Scholar 

  18. 18.

    Parsons TD. Neuropsychological Rehabilitation 3.0: State of the Science. In: Clinical Neuropsychology and Technology: Springer International Publishing; 2016. p. 113–32. Available from: http://link.springer.com/chapter/10.1007/978-3-319-31075-6_7. Cited 2016 Apr 8.

  19. 19.

    Solana J, Cáceres C, García-Molina A, Opisso E, Roig T, Tormos JM, et al. Improving brain injury cognitive rehabilitation by personalized Telerehabilitation services: Guttmann Neuropersonal trainer. IEEE J Biomed Health Inform. 2015;19(1):124–31.

    PubMed  Article  Google Scholar 

  20. 20.

    Tedim Cruz V, Pais J, Ruano L, Mateus C, Colunas M, Alves I, et al. Implementation and Outcomes of a Collaborative Multi-Center Network Aimed at Web-Based Cognitive Training – COGWEB Network. JMIR Ment Health. 2014;1(1) Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4607396/. Cited 2016 May 2.

  21. 21.

    Gracey F, Wilson BA. Theoretical approaches to cognitive rehabilitation. In: Goldstein LH, McNeil JE, editors. Clinical Neuropsychology: A Practical Guide to Assessment and Management for Clinicians: Wiley; 2013. p. 463–6.

  22. 22.

    Wood RL. Towards a model of cognitive rehabilitation. In: Wood RL, Fussey I, editors. Cognitive rehabilitation in perspective. Routledge; 2018. p. 3–26.

  23. 23.

    Faria AL, Pinho MS, Bermúdez i Badia S. Capturing expert knowledge for the personalization of cognitive rehabilitation: study combining computational modeling and a participatory design strategy. JMIR Rehabil Assist Technol. 2018;5(2):e10714.

    PubMed  PubMed Central  Article  Google Scholar 

  24. 24.

    Faria AL & Bermúdez i Badia S. Personalizing paper-and-pencil training for cognitive rehabilitation. Funchal: International Conference on Applied Psychology and Human Behavior proceedings; 2018. p. 1–6.

  25. 25.

    Faria A, Bermúdez i Badia S. Development and evaluation of a web-based cognitive task generator for personalized cognitive training: a proof of concept study with stroke patients. In: REHAB 2015: 3rd Workshop on ICTs for improving patients research techniques. Lisboa: ACM; 2015.

    Google Scholar 

  26. 26.

    Luca RD, Russo M, Naro A, Tomasello P, Leonardi S, Santamaria F, et al. Effects of virtual reality-based training with BTs-nirvana on functional recovery in stroke patients: preliminary considerations. Int J Neurosci. 2018;128(9):791–6.

    PubMed  Article  Google Scholar 

  27. 27.

    Maggio MG, Latella D, Maresca G, Sciarrone F, Manuli A, Naro A, et al. Virtual reality and cognitive rehabilitation in people with stroke: an overview. J Neurosci Nurs. 2019;51(2):101–5.

    PubMed  Article  Google Scholar 

  28. 28.

    Vourvopoulos A, Faria AL, Ponnam K, Bermudez i Badia S. RehabCity: Design and Validation of a Cognitive Assessment and Rehabilitation Tool Through Gamified Simulations of Activities of Daily Living. In: Proceedings of the 11th Conference on Advances in Computer Entertainment Technology. New York: ACM; 2014. 26:1–26:8. https://doi.org/10.1145/2663806.2663852. (ACE ’14). Cited 2016 Dec 2.

  29. 29.

    Faria AL, Andrade A, Soares L, Bermúdez i Badia S. Benefits of virtual reality based cognitive rehabilitation through simulated activities of daily living: a randomized controlled trial with stroke patients. J Neuroeng Rehabil. 2016;13(1):96.

    PubMed  PubMed Central  Article  Google Scholar 

  30. 30.

    Hagovska M, Nagyova I. The transfer of skills from cognitive and physical training to activities of daily living: a randomised controlled study. Eur J Ageing. 2016;14(2):133–42 Levin MF, Weiss PL, Keshner EA. Emergence of virtual reality as a tool for upper limb rehabilitation: incorporation of motor control and motor learning principles. Phys Ther 2015; 95(3):415–425.

    PubMed  PubMed Central  Article  Google Scholar 

  31. 31.

    Kizony R, Katz N, Weiss PL. Virtual reality based intervention in rehabilitation: relationship between motor and cognitive abilities and performance within virtual environments for patients with stroke. In: Proceedings of the 5th international conference on disability, virtual reality and associated technology Oxford, UK; 2004. Kizony R, Levin MF, Hughey L, Perez C, Fung J. Cognitive load and dual-task performance during locomotion poststroke: a feasibility study using a functional virtual environment. Phys Ther 2010;90(2):252–60.

    Google Scholar 

  32. 32.

    Subramanian SK, Chilingaryan G, Levin MF, Sveistrup H. Influence of training environment and cognitive deficits on use of feedback for motor learning in chronic stroke. In: 2015 International Conference on Virtual Rehabilitation (ICVR): IEEE; 2015. p. 38–43. Mullick AA, Subramanian SK, Levin MF. Emerging evidence of the association between cognitive deficits and arm motor recovery after stroke: a meta-analysis. Restor Neurol Neurosci 2015;33(3):389–403.

  33. 33.

    WHO. WHO | Assistive devices/technologies: what WHO is doing: WHO; 2011. Available from: http://www.who.int/disabilities/technology/activities/en/. Cited 2016 May 9.

  34. 34.

    Claessen MH, Visser-Meily JM, Rooij NK, Postma A, Ham IJ. A direct comparison of real-world and virtual navigation performance in chronic stroke patients. J Int Neuropsychol Soc. 2016;22(4):467–77.

    PubMed  Article  Google Scholar 

  35. 35.

    Dores AR, Miranda MJ, Carvalho IP, Mendes L, Barbosa F, Coelho A, et al. Virtual City: Neurocognitive rehabilitation of Acquired Brain Injury. In: 2012 7th Iberian Conference on Information Systems and Technologies (CISTI); 2012. p. 1–4.

    Google Scholar 

  36. 36.

    Gamito P, Oliveira J, Coelho C, Morais D, Lopes P, Pacheco J, et al. Cognitive training on stroke patients via virtual reality-based serious games. Disabil Rehabil. 2015;39(4):385–8.

    PubMed  Article  Google Scholar 

  37. 37.

    Jovanovski, Zakzanis K, Ruttan L, Campbell Z, Erb S, Nussbaum D. Ecologically valid assessment of executive dysfunction using a novel virtual reality task in patients with acquired brain injury. Applied Neuropsychol Adult. 2012;19(3):207–20.

    Article  Google Scholar 

  38. 38.

    Klinger E, Joseph P-A, Le Guiet J-L, Fuchs P, du Lac N, Servant F. AGATHE: A Tool for Personalized Rehabilitation of Cognitive Functions. In: ACM SIGGRAPH 2013 Emerging Technologies. New York: ACM; 2013. p. 1:1. (SIGGRAPH ‘13). Available from: http://doi.acm.org/10.1145/2503368.2503369. Cited 2016 Dec 2.

    Google Scholar 

  39. 39.

    Adams RJ, Lichter MD, Krepkovich ET, Ellington A, White M, Diamond PT. Assessing upper extremity motor function in practice of virtual activities of daily living. IEEE Trans Neural Syst Rehabil Eng. 2015;23(2):287–96.

    PubMed  Article  Google Scholar 

  40. 40.

    Besnard J, Richard P, Banville F, Nolin P, Aubin G, Le Gall D, et al. Virtual reality and neuropsychological assessment: the reliability of a virtual kitchen to assess daily-life activities in victims of traumatic brain injury. Appl Neuropsychol Adult. 2015;23(3):223–35.

    PubMed  Article  Google Scholar 

  41. 41.

    Cao X, Douguet A-S, Fuchs P, Klinger E. Designing an ecological virtual task in the context of executive functions: Preliminary study; 2010. p. 71–7. Available from: https://hal-mines-paristech.archives-ouvertes.fr/hal-00785344. Cited 2016 Dec 13.

    Google Scholar 

  42. 42.

    Christiansen C, Abreu B, Ottenbacher K, Huffman K, Masel B, Culpepper R. Task performance in virtual environments used for cognitive rehabilitation after traumatic brain injury. Arch Phys Med Rehabil. 1998;79(8):888–92.

    CAS  PubMed  Article  Google Scholar 

  43. 43.

    Edmans JA, Gladman JRF, Cobb S, Sunderland A, Pridmore T, Hilton D, et al. Validity of a virtual environment for stroke rehabilitation. Stroke. 2006;37(11):2770–5.

    PubMed  Article  Google Scholar 

  44. 44.

    Zhang L, Abreu BC, Seale GS, Masel B, Christiansen CH, Ottenbacher KJ. A virtual reality environment for evaluation of a daily living skill in brain injury rehabilitation: reliability and validity. Arch Phys Med Rehabil. 2003;84(8):1118–24.

    PubMed  Article  Google Scholar 

  45. 45.

    O’Brien J. Simulating the homes of stroke patients: can virtual environments help to promote engagement in therapy activities? In: 2007 Virtual Rehabilitation; 2007. p. 23–8.

    Google Scholar 

  46. 46.

    Devos H, Akinwuntan AE, Nieuwboer A, Tant M, Truijen S, De Wit L, et al. Comparison of the effect of two driving retraining programs on on-road performance after stroke. Neurorehabil Neural Repair. 2009;23(7):699–705.

    PubMed  Article  Google Scholar 

  47. 47.

    Sorita E, N’Kaoua B, Larrue F, Criquillon J, Simion A, Sauzéon H, et al. Do patients with traumatic brain injury learn a route in the same way in real and virtual environments? Disabil Rehabil. 2013;35(16):1371–9.

    PubMed  Article  Google Scholar 

  48. 48.

    Katz N, Ring H, Naveh Y, Kizony R, Feintuch U, Weiss PL. Interactive virtual environment training for safe street crossing of right hemisphere stroke patients with unilateral spatial neglect. Disabil Rehabil. 2005;27(20):1235–44.

    CAS  PubMed  Article  Google Scholar 

  49. 49.

    Lloyd J, Riley GA, Powell TE. Errorless learning of novel routes through a virtual town in people with acquired brain injury. Neuropsychol Rehabil. 2009;19(1):98–109.

    PubMed  Article  Google Scholar 

  50. 50.

    Navarro M-D, Lloréns R, Noé E, Ferri J, Alcañiz M. Validation of a low-cost virtual reality system for training street-crossing. A comparative study in healthy, neglected and non-neglected stroke individuals. Neuropsychol Rehabil. 2013;23(4):597–618.

    PubMed  Article  Google Scholar 

  51. 51.

    Titov N, Knight RG. A computer-based procedure for assessing functional cognitive skills in patients with neurological injuries: the virtual street. Brain Inj. 2005;19(5):315–22.

    CAS  PubMed  Article  Google Scholar 

  52. 52.

    Josman N, Kizony R, Hof E, Goldenberg K, Weiss PL, Klinger E. Using the virtual action planning-supermarket for evaluating executive functions in people with stroke. J Stroke Cerebrovasc Dis. 2014;23(5):879–87.

    PubMed  Article  Google Scholar 

  53. 53.

    Kang YJ, Ku J, Han K, Kim SI, Yu TW, Lee JH, et al. Development and clinical trial of virtual reality-based cognitive assessment in people with stroke: preliminary study. CyberPsychol Behav. 2008;11(3):329–39.

    PubMed  Article  Google Scholar 

  54. 54.

    Raspelli S, Pallavicini F, Carelli L, Morganti F, Pedroli E, Cipresso P, et al. Validating the Neuro VR-based virtual version of the multiple errands test: preliminary results. Presence Teleop Virt. 2012;21(1):31–42.

    Article  Google Scholar 

  55. 55.

    Sorita E, Joseph PA, N’kaoua B, Ruiz J, Simion A, Mazaux JM, et al. Performance analysis of adults with acquired brain injury making errands in a virtual supermarket. Ann Phys Rehabil Med. 2014;57:e85.

    Article  Google Scholar 

  56. 56.

    Yip BC, Man DW. Virtual reality-based prospective memory training program for people with acquired brain injury. Neurorehabilitation. 2013;32(1):103–15.

    PubMed  Article  Google Scholar 

  57. 57.

    Canty AL, Fleming J, Patterson F, Green HJ, Man D, Shum DHK. Evaluation of a virtual reality prospective memory task for use with individuals with severe traumatic brain injury. Neuropsychol Rehabil. 2014;24(2):238–65.

    PubMed  Article  Google Scholar 

  58. 58.

    Erez N, Weiss PL, Kizony R, Rand D. Comparing performance within a virtual supermarket of children with traumatic brain injury to typically developing children: a pilot study. OTJR Occup Participation Health. 2013;33(4):218–27.

    Article  Google Scholar 

  59. 59.

    Jacoby M, Averbuch S, Sacher Y, Katz N, Weiss PL, Kizony R. Effectiveness of executive functions training within a virtual supermarket for adults with traumatic brain injury: a pilot study. IEEE Trans Neural Syst Rehabil Eng. 2013;21(2):182–90.

    PubMed  Article  Google Scholar 

  60. 60.

    Okahashi S, Seki K, Nagano A, Luo Z, Kojima M, Futaki T. A virtual shopping test for realistic assessment of cognitive function. J Neuroeng Rehabil. 2013;10:59.

    PubMed  PubMed Central  Article  Google Scholar 

  61. 61.

    Rand D, Rukan SB-A, Weiss PL, Katz N. Validation of the virtual MET as an assessment tool for executive functions. Neuropsychol Rehabil. 2009;19(4):583–602.

    PubMed  Article  Google Scholar 

  62. 62.

    Schenkenberg T, Bradford DC, Ajax ET. Line bisection and unilateral visual neglect in patients with neurologic impairment. Neurology. 1980;30(5):509.

    CAS  PubMed  Article  Google Scholar 

  63. 63.

    Nasreddine ZS, Collin I, Chertkow H, Phillips N, Bergman H, Whitehead V. Sensitivity and specificity of the Montreal cognitive assessment (MoCA) for detection of mild cognitive deficits. Can J Neurol Sci. 2003;30(2):30.

    Google Scholar 

  64. 64.

    Freitas S, Simões MR, Alves L, Santana I. Montreal cognitive assessment (MoCA): normative study for the Portuguese population. J Clin Exp Neuropsychol. 2011;33(9):989–96.

    PubMed  Article  Google Scholar 

  65. 65.

    Beck AT, Ward CH, Mendelson M, Mock J, Erbaugh J. An inventory for measuring depression. Arch Gen Psychiatry. 1961;4:561–71.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  66. 66.

    Campos RC, Gonçalves B. The Portuguese version of the Beck depression inventory-II (BDI-II): preliminary psychometric data with two nonclinical samples. Eur J Psychol Assess. 2011;27(4):258–64.

    Article  Google Scholar 

  67. 67.

    Paulino T, Faria AL, Bermúdez i Badia S. Reh@ City v2. 0: a comprehensive virtual reality cognitive training system based on personalized and adaptive simulations of activities of daily living. In: Experiment@ International Conference, Funchal, Portugal; 2019.

    Google Scholar 

  68. 68.

    Cameirão MS, Badia SBI, Oller ED, Verschure PFMJ. Neurorehabilitation using the virtual reality based rehabilitation gaming system: methodology, design, psychometrics, usability and validation. J Neuroeng Rehabil. 2010;7:48.

    PubMed  PubMed Central  Article  Google Scholar 

  69. 69.

    Bermúdez i Badia S. AnTS: analysis and tracking system. 2014. Available from: http://neurorehabilitation.m-iti.org/tools/.

    Google Scholar 

  70. 70.

    Pendlebury ST, Mariz J, Bull L, Mehta Z, Rothwell PM. MoCA, ACE-R, and MMSE versus the National Institute of Neurological Disorders and Stroke–Canadian Stroke network vascular cognitive impairment harmonization standards neuropsychological battery after TIA and Stroke. Stroke. 2012;43(2):464–9.

    PubMed  Article  Google Scholar 

  71. 71.

    Tan HH, Xu J, Teoh HL, Chan BP-L, Seet RCS, Venketasubramanian N, et al. Decline in changing Montreal cognitive assessment (MoCA) scores is associated with post-stroke cognitive decline determined by a formal neuropsychological evaluation. PLoS One. 2017;12(3):e0173291.

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  72. 72.

    Reitan RM. Validity of the trail making test as an Indicator of organic brain damage. Percept Mot Skills. 1958;8(3):271–6.

    Article  Google Scholar 

  73. 73.

    Cavaco S, Gonçalves A, Pinto C, Almeida E, Gomes F, Moreira I, et al. Trail making test: regression-based norms for the Portuguese population. Arch Clin Neuropsychol. 2013;28(2):189–98.

    PubMed  Article  Google Scholar 

  74. 74.

    Wechsler D. Escala de Memória de Wechsler - 3a Edição: Hogrefe Editora; 2008. Available from: https://www.hogrefe.pt/shop/escala-de-memoria-de-wechsler-3-edic-o.html. Cited 2018 Mar 23.

  75. 75.

    Wechsler D. Escala de Inteligência de Wechsler para Adultos - 3a Edição: Hogrefe Editora; 2008. Available from:. https://www.hogrefe.pt/shop/escala-de-inteligencia-de-wechsler-para-adultos-3-edic-o.html. Cited 2018 Mar 23.

  76. 76.

    Patchick E, Vail A, Wood A, Bowen A. PRECiS (patient reported evaluation of cognitive state): psychometric evaluation of a new patient reported outcome measure of the impact of stroke. Clin Rehabil. 2016;30(12):1229–41.

    PubMed  Article  Google Scholar 

  77. 77.

    Faria AL, Alegria J, Pinho MS, Badia SB. Avaliação feita pelo Doente sobre o seu Estado Cognitivo. 2018. Available from: https://www.click2go.umip.com/i/coa/precis_pt.html.

    Google Scholar 

  78. 78.

    Brooke J. SUS-A quick and dirty usability scale. Usability Eval Industry. 1996;189:194 Nir-Hadad SY, Weiss PL, Waizman A, Schwartz N, Kizony R. A virtual shopping task for the assessment of executive functions: validity for people with stroke. Neuropsychol Rehabil 2015;27(5):808–833.

    Google Scholar 

  79. 79.

    Neguţ A, Matu SA, Sava FA, David D. Task difficulty of virtual reality-based assessment tools compared to classical paper-and-pencil or computerized measures: a meta- analytic approach. Comput Hum Behav. 2016;54:414–24.

    Article  Google Scholar 

  80. 80.

    Parsons TD, Barnett MD. Virtual apartment stroop task: comparison with computerized and traditional stroop tasks. J Neurosci Methods. 2018;309:35–40.

    PubMed  Article  Google Scholar 

  81. 81.

    Faria AL, Cameirão MS, Couras JF, Aguiar JRO, Costa D, Martins G, et al. Combined cognitive-motor rehabilitation in virtual reality improves motor outcomes in chronic stroke – a pilot study. Front Psychol. 2018;9. Available from: https://www.frontiersin.org/articles/10.3389/fpsyg.2018.00854/abstract. Cited 2018 May 20.

  82. 82.

    Wong GK, Mak JS, Wong A, Zheng VZ, Poon WS, Abrigo J, Mok VC. Minimum clinically important difference of Montreal cognitive assessment in aneurysmal subarachnoid hemorrhage patients. J Clin Neurosci. 2017;46:41–4.

    PubMed  Article  Google Scholar 

  83. 83.

    Mioshi E, Dawson K, Mitchell J, Arnold R, Hodges JR. The Addenbrooke’s cognitive examination revised (ACE-R): a brief cognitive test battery for dementia screening. Int J Geriatr Psychiatry. 2006;21(11):1078–85.

    PubMed  Article  Google Scholar 

  84. 84.

    Duncan PW, Bode RK, Min Lai S, Perera S. Rasch analysis of a new stroke-specific outcome scale: the stroke impact scale. Arch Phys Med Rehabil. 2003;84(7):950–63.

    PubMed  Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Joana Alegria and Carolina Jorge for the collaboration in data collection and Teresa Paulino for the Reh@City v2.0 software development.

Funding

This study was supported by the European Commission through the RehabNet project (Neuroscience-Based Interactive Systems for Motor Rehabilitation) under grant 303891 RehabNet FP7-PEOPLE-2011-CIG, LARSyS UID/EEA/50009/2019, MACBIOIDI (INTERREG program MAC/1.1.b/098), FCT through the BRaNT project (PTDC/CCI-COM/31046/2017) and Agência Regional para o Desenvolvimento da Investigação, Tecnologia e Inovação.

Author information

Affiliations

Authors

Contributions

ALF, MSP and SBB designed the study. ALF performed the data collection and the data analysis. ALF, MSP and SBB wrote the manuscript for publication. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Ana Lúcia Faria.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the Madeira Health Service - SESARAM Ethical Committee (approval number 13/2016) and all the participants gave their informed consent.

Consent for publication

Consent for publication of individual data has been obtained from all the participants of the study.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Faria, A.L., Pinho, M.S. & Bermúdez i Badia, S. A comparison of two personalization and adaptive cognitive rehabilitation approaches: a randomized controlled trial with chronic stroke patients. J NeuroEngineering Rehabil 17, 78 (2020). https://doi.org/10.1186/s12984-020-00691-5

Download citation

Keywords

  • Cognitive rehabilitation
  • Virtual reality
  • Stroke
  • Ecological validity