Skip to main content

Evaluation of power wheelchair driving performance in simulator compared to driving in real-life situations: the SIMADAPT (simulator ADAPT) project—a pilot study

Abstract

Objective

The objective of this study was to evaluate users’ driving performances with a Power Wheelchair (PWC) driving simulator in comparison to the same driving task in real conditions with a standard power wheelchair.

Methods

Three driving circuits of progressive difficulty levels (C1, C2, C3) that were elaborated to assess the driving performances with PWC in indoor situations, were used in this study. These circuits have been modeled in a 3D Virtual Environment to replicate the three driving task scenarios in Virtual Reality (VR). Users were asked to complete the three circuits with respect to two testing conditions during three successive sessions, i.e. in VR and on a real circuit (R). During each session, users completed the two conditions. Driving performances were evaluated using the number of collisions and time to complete the circuit. In addition, driving ability by Wheelchair Skill Test (WST) and mental load were assessed in both conditions. Cybersickness, user satisfaction and sense of presence were measured in VR. The conditions R and VR were randomized.

Results

Thirty-one participants with neurological disorders and expert wheelchair drivers were included in the study. The driving performances between VR and R conditions were statistically different for the C3 circuit but were not statistically different for the two easiest circuits C1 and C2. The results of the WST was not statistically different in C1, C2 and C3. The mental load was higher in VR than in R condition. The general sense of presence was reported as acceptable (mean value of 4.6 out of 6) for all the participants, and the cybersickness was reported as acceptable (SSQ mean value of 4.25 on the three circuits in VR condition).

Conclusion

Driving performances were statistically different in the most complicated circuit C3 with an increased number of collisions in VR, but were not statistically different for the two easiest circuits C1 and C2 in R and VR conditions. In addition, there were no significant adverse effects such as cybersickness. The results show the value of the simulator for driving training applications. Still, the mental load was higher in VR than in R condition, thus mitigating the potential for use with people with cognitive disorders. Further studies should be conducted to assess the quality of skill transfer for novice drivers from the simulator to the real world.

Trial registration Ethical approval n\(^\circ\)2019-A001306-51 from Comité de Protection des Personnes Sud Mediterranée IV. Trial registered the 19/11/2019 on ClinicalTrials.gov in ID: NCT04171973.

Introduction

Power Wheelchairs (PWC) may be the only solution to maintain mobility and autonomy for some people with physical impairments. Several studies have underlined the benefits of using a PWC as it improves mobility and social participation while reducing the burden on caregivers [1], as well as the importance of moving for people with neurological impairments. For example, a study carried out by the Breizh Cerebral Palsy Network has shown the negative impact of limitations in moving around on the quality of life of people with cerebral palsy [2]. Independence in mobility was also identified as one of the main barriers to maintain societal participation in activities such as employment for people with traumatic brain injury (TBI) [3].

In this context, PWC driving confidence is therefore extremely important to guarantee the autonomy of people who need wheelchairs. One of the main barriers to autonomy through the access to mobility is that not everyone can access PWC driving as it can lead to accidents, in particular for people with visual or cognitive impairments. In fact, wheelchair safety reports show that 25\(\%\) of wheelchair accidents are linked to the use of PWC [4]. Moreover, PWC driving is a difficult task which requires cognitive functions which can be impaired for people with neurological disorders. Indeed, up to 40\(\%\) of people with TBI who use PWC regularly have problems with steering and 5\(\%\) to 9\(\%\) cannot steer at all in a clinical setting [3]. To mitigate the risks of PWC driving difficulties and limit the risk of accidents, the learning-to-drive process is therefore a key phase. A recent review of the literature [5] highlighted the wide variety of tasks required to learn to drive in a wheelchair and the need for standardized performance assessment. The use of Virtual Reality (VR) simulators could meet these needs in a safe environment and by allowing tasks to be reproduced in a reproducible manner as needed to promote training. Researchers have developed and tested simulators dedicated to wheelchair driving training which can help users adapt their driving strategies to different environment [6,7,8,9,10].

VR-based simulators may be a promising tool for wheelchair driving training allowing more immersive driving situation but needs a good sense of presence i.e. a subjective experience of driving in the environment with the PWC of good quality. Indeed, virtual presence is the ability of being in the virtual world. A high level of presence then allows for greater immersion and therefore for the person to be very close to reality, thus ensuring better training [11, 12]. Indeed, a link between the high level of presence (measured by a 29 items scales) and involvement in the scene has been shown in crisis situations such as a VR situation replicating evacuation in the case of a fire or stress with soldiers [13]. Similarly, this link has also been observed in learning situations with students [14]. Moreover, recent work in the educational sciences has focused on this sense of presence and its link to learning outcomes [15]. One difficulty in using VR-based simulator is the risk of cybersickness. Cybersickness can be considered as a subgroup of motion sickness induced by VR [16], that is stimulated by artificial moving images [17]. Motion sickness is caused by a sensory conflict induced by the disparity in motion between two sensory systems that are the visual and vestibular systems, whereas cybersickness does not involve the vestibular system [18]. Cybersickness is also often experienced during simulation or other VR exposure and must be taken into account in the use of a driving simulator. The symptoms can be nausea, headache, general discomfort, or sweating, and can be experienced during and after exposure to a virtual environment [19]. It seems to be negatively related with sense of presence, which underlines the importance of the level of immersion, to ensure a good level of sense of presence and thus a good quality of learning [12, 20]. Finally, driving difficulties can be experienced by patients with cognitive disorders. They are therefore potential candidates for the use of VR-based simulator as it provides configurable virtual training without physical risks. However, another common difficulty in immersive VR simulation training is the induced mental load [21]. VR-based simulators must therefore be designed so they do not increase too much the mental load in order to enable these patients to benefit from this type of devices.

In this context, the Assistive Devices for empowering People with disabilities through robotic Technologies (ADAPT) project is developing new technologies to facilitate the training of PWCs driving. The aim of this project is to allow a maximum number of users to use a PWC in safety in daily life. For this purpose, a VR-based PWC driving simulator aiming to contribute to driving training for patients has been developed [22]. The simulator provides combined visual feedback and vestibular feedback in order to limit motion sickness. The platform is able to simulate acceleration resulting from user command and centrifugal effects.. The SIMADAPT1 study aims to evaluate the reliability of the proposed simulator compared to the performance on real circuit, in addition to the tolerance of use of the simulator, including assessment of sense of presence, cybersickness and mental load. Even though this study does not deal with the use of the simulator for learning to drive, this is the final objective of this device. We proposed to regular PWC users with neurological disorders to test this simulator prototype on a driving task in a 3D virtual environment replicating 3 real standardized driving circuits [23].

The proposed study involved 3 sessions of driving on circuits of increasing difficulty in VR with the simulator and in R with a standard power wheelchair. Our hypotheses were the following:

  • H1 The simulator enables comparable driving performances between VR and real-life conditions;

  • H2 There is no significant differences regarding the driving ability;

  • H3 There is no significant differences regarding the usability between real and virtual environments;

  • H4 There is no significant differences regarding the mental workload between real and virtual environments;

  • H5 The cybersickness is tolerable;

  • H6 The sense of presence is high in VR condition;

  • H7 Both the cybersickness, presence, and mental workload do not have a significant impact on driving performance between real and virtual conditions.

Material and methods

Study design and participants

This study is a prospective, monocentric, randomized and controlled pilot study to investigate the realibility of a PWC simulator. This pilot study was conducted in January 2020 at the rehabilitation center Pôle St Hélier in Rennes, France. The study was approved by the People Protection Committee under the ethical approval n°2019-A001306-51.

We screened potential participants 10 days before the beginning of the trial sessions and validated their eligibility for the following inclusion criteria: (1) being over 18 years old; (2) having freely consented to participate in the study; and (3) using a PWC as the main mode of locomotion for more than three months because of a stabilized medical problem. The exclusion criteria were: (1) having difficulties to understand and follow instructions; (2) presenting motor disorders of the upper limb requiring additional driving technical assistance such as specific driving interfaces (head pointer, mouth control, etc.); (3) being pregnant; and (4) unable to express consent. The investigator presented the objectives to each participant as well as the participation modalities according to the validated protocol and delivered an information flier to each participant. The consent form was signed by voluntary participants after a 10 days thinking period.

Experimental setup

A multisensory PWC driving simulator in Virtual Reality was designed within the European Interreg ADAPT project [22][24]. This simulator consists of a four degrees of freedom (pitch, roll, yaw, and heave) mechanical platform designed to induce vestibular feedback [25]. The movements and accelerations of the driving simulator are perceived by the vestibular system of the user while being consistent with the other senses involved in the simulation. Hence, as the vestibular system is responsible for the sense of balance and provides information about the body position, motion sickness is potentially reduced. The user input retrieved from the joystick allows the user to drive the PWC in the virtual environment. A head mounted device displays the virtual environment in which the user navigates. Virtual circuits were modeled in 3D by Unity engine to create test scenarios in VR as illustrated on Fig. 1.

Fig. 1
figure 1

From real to virtual environment: example of virtual scene as a replication of real world scene

To ensure that the PWC driving simulator dynamics are the same as real PWC dynamics, the simulator embeds a commercial power module taken from a real PWC. On a real PWC, the power module is responsible for controlling the PWC wheel speeds according to the user joystick input, preset acceleration, and speed parameters. In the simulator, wheel speeds coming from the power module are directly fed into the physics engine responsible for the simulation. Both power modules from real and virtual PWC are set up the same way to provide the same PWC behavior in virtual and real conditions. Speed and acceleration measures, as well as trajectory analysis comparison through motion capture, were performed both in virtual and real conditions to ensure that both wheelchairs were moving at the same speed and had the same behavior. In the virtual environment, the participant visualizes themselves as a first-person avatar without showing the arms and representing the legs covered by a blanket as illustrated on Fig. 2.

Fig. 2
figure 2

VR simulation capture from user point of view

The head mounted display used in this trial is a HTC Vive Pro Full kit (certifications: CE 2200, ROHS) that complies with the European R & TTE directives. Standard PWC QUICKIE Salsa M2 from Sunrise Medical (Class I Medical Device CE marked) is used to carry out the trials in R condition. Figure 3 provides an illustration of each experimental setup.

Experimental procedure

Each participant completed 3 sessions over three weeks in a row. These three sessions corresponded to three progressive levels of driving difficulties. In each session, users tested two conditions: real circuits (R condition) or virtual circuits in the simulator (VR condition) as illustrated on Fig. 3.

Fig. 3
figure 3

Experimental setup: a real circuit and b VR-based simulator

Three real circuits intended to assess the driving performances with PWC in indoor situations have been defined according to [23, 26,27,28]. The first circuit (C1) is composed of several basic tasks such as driving forwards (10 m) and backwards (2 m), turning in place and while moving forwards (90°). The second circuit (C2) includes slightly more difficult tasks such as getting through hinged door, ascending and descending a 5\(^\circ\) access ramp, rolling on soft surface (2 m), crossing doorstep and driving through narrow corridors. The third and last circuit (C3) is the most difficult as it includes more difficult maneuvers and tasks such as avoiding a moving obstacle, ascending and descending a 10\(^\circ\) access ramp. In C3, the moving obstacle corresponds to a manual wheelchair passing in front of the participant at a waypoint. In R condition the moving obstacle is pushed by an experimenter whereas it is scripted in VR condition. In both conditions, the moving obstacle is triggered when the wheelchair the patient is using is reaching a specific landmark of the circuit. These different scenarios involve different individual skills as proposed in the Wheelchair Skill Test (WST), which distribution on C1, C2, and C3 is presented in Table 1 [1].

Circuits for the R condition were made of plastic walls, exercise mats, boxes and access ramps. Circuits were replicated to scale in the virtual environment for the VR condition. In addition, vestibular feedback was implemented in the simulator to reproduce the motion and driving behavior of the real PWC used in the R condition.

For each condition, the users completed the circuit twice (T1 and T2 trials). The order between real and virtual conditions was randomized for each session with the Randomizer software (simple randomization). Therefore, each user performed 12 runs, with each run corresponding to a complete lap of the circuit: 2 runs in the R condition and 2 runs in VR condition for each of the 3 sessions.

Table 1 Circuits C1, C2, C3 with respect to main items of the WST [1]

Before each test session, participants were asked to drive around in the PWC during 5 min in both R and VR conditions to familiarize themselves with the setup. Then, circuits were presented to the participants.

Study outcomes and measures

The primary outcome is the number of collisions on the three standardized circuits, in R condition versus VR condition. A collision was defined as a contact between the wheelchair and real or virtual parts of the circuits.Two evaluators monitored the participants as they drove around the circuit, and counted collisions during the trial (on a screen for the virtual circuit). An agreement between the two evaluators was necessary to validate a collision. If there was no consensus, the highest number of collisions was retained.

The secondary outcomes consist of the driving speed which was estimated by the time of completion in R and VR conditions.

To assess driving ability, we used the WST 4.2.3 items corresponding to the different courses in R and VR conditions. Each driving skill evaluated through the WST was scored from 0 to 2 (0: fail, 1: pass with difficulty or assistance, 2: pass) by the two occupational therapists involved in the trial. The three circuits were of increasing difficulty and integrated different items of the WST for each one (Table 1).

The mental load under both conditions was measured by the NASA-Task Load Index (NASA-TLX) [29, 30]. The NASA-TLX is a questionnaire in the form of six dimensions which must be scored from 0 to 100. A high score indicates a growing level of intensity. Three dimensions are related to the demands imposed on the subject (mental, physical and temporal) and the three others are related to the interaction of the subject with the task (effort, frustration and performance). A mean score from the six dimensions is computed, with the total obtainable score between 0 and 100. The usability of the PWC under the conditions was evaluated by the Ease of Use Questionnaire (USE) [31]. This questionnaire includes 30-item survey scored on a seven-point Likert rating scales divided in four dimensions exploring usability. These dimensions are namely usefulness, ease of use, ease of learning and satisfaction. For this questionnaire, users were asked to rate agreement with each statement, ranging from strongly disagree (1) to strongly agree (7).

The sense of presence in the virtual environment was evaluated by the Igroup Presence Questionnaire (IPQ) [32]. The IPQ consists of 13 questions and defines the general sense of presence, involvement, spatial presence, and realism. It is composed of three subscales and one additional general item. The general item assessed the general “sense of being there”. The Spatial Presence sub-scale is related to the sense of being physically inside the virtual environment. The involvement subscale aims to evaluate the attention devoted to the virtual environment. The experienced realism sub-scale evaluates the sense of reality attributed to the virtual environment. Each question takes the form of a 7-points scale. Finally, the Simulator Sickness Quantifying (SSQ) measured the feeling of cybersickness and was completed after each VR circuit. The discomfort and cybersickness during the VR condition were also assessed using the Graybiel score [33], which is a questionnaire quantifying the intensity of different symptoms related to cybersickness.

Data analysis

Data are analyzed using the statistical software R Statistic, version 3.6.1 (2019-07-05). Quantitative analysis of data was expressed as mean and standard deviation and median and inter-quartile range (Md ± IQR). The normality of the data was investigated by the Shapiro test. Data for the 3 circuits in conditions R or VR are not normally distributed. Therefore R and VR conditions were compared using the non-parametric Wilcoxon test. We fitted linear models to study relationship between VR experience parameters (cybersickness, mental load, sense of presence, satisfaction) and driving performances. Post hoc analysis was subsequently conducted to explore any unforeseen correlations or patterns among these variables.

Results

Participants

32 users were screened and 31 included: 15 men and 16 women with a mean age of 49.6 ± 13.1 years old. participants were all expert PWC drivers to comply with the objective of this study to validate efficacy and tolerance of the proposed VR-based simulator with people with neurological disorders not experiencing driving difficulties. Characteristics of the participants are presented in Table 2. One user could not perform circuit C2 and two users could not perform circuit C3: this participant refused to continue the experiment due to high level of fatigue.

Table 2 Participants’ characteristics: gender, age (years old), diagnosis, and duration of PWC use (in years)

Comparison of driving performances between real and virtual environment

Results are presented in Table 3. There was no significant difference for the number of collisions between the two conditions in C1 and C2. However, a significant difference (\(p < 0.01\)) can be observed for the number of collisions in the most difficult circuit C3 regarding the two runs: there were more collisions in the VR condition compared to the R condition. Time of completion was also always longer in the VR condition except for the second run in the two last circuits C2 and C3. Therefore, hypothesis H1 is only partially confirmed.

Driving ability

Concerning the WST score (Table 3), there was no statistically differences between R and VR conditions for the three circuits. H2 is confirmed.

Usability

Usability measured by the USE questionnaire was high in the two conditions, while being significantly higher for the R condition. Moreover, the most important difference appears for the usefulness dimension, with a score always below 5 in the VR condition. H3 is not confirmed.

Table 3 Outcome measures—synthesis

Mental load

The mental load is evaluated through the NASA-TLX. Results are presented in Table 4. It appears that the required mental load is reported to be higher in VR condition when compared to the R condition on all three circuits (C1: \(p < 0.001\); C2: \(p = 0.01\); C3: \(p = 0.03\)). Moreover, the mental load does not increase between the C1 and C3 circuits, despite of their increasing difficulty. H4 is also not confirmed.

Table 4 Change in Mental Load as assessed by NASA Task Load Index (NASA-TLX) among the three incrementally challenging circuits in Real (R) and Virtual (VR) conditions

Cybersickness in VR condition

Cybersickness is evaluated through Graybiel and SSQ scales, which both characterize the cybersickness in VR condition. The related results are presented in Table 5. Regarding the SSQ scores (total possible score up to 48), participants rated their cybersickness to be relatively low (mean SSQ values are equal to 4.25, 3.43 and 4.72 with respect to C1, C2 and C3 circuits). For the Graybiel, the highest possible score is 48 while a score above 16 is corresponding to frank sickness (level 4), and a score under 3 is considered as a slight discomfort (level 1). In our results, the maximum Graybiel score of 2.07 is reached on the circuit C3. Sickness was higher in the most difficult circuit (C3). If there was a significant decrease on the Graybiel score for C2 regarding the score for C1, it is different concerning the SSQ. The Graybiel scores were not statistically different between C1 and C3 (\(p = 0.48\)).Therefore, H5 is not confirmed.

Table 5 Outcome measures related to cybersickness for the three circuits

Sense of presence

As shown in Fig. 4, the Igroup Presence Questionnaire (IPQ) results showed an acceptable general and spatial presence with the virtual environment regarding to [34]. Results were similar for the 3 circuits, with lower scores for experienced realism and involvement. H6 is confirmed.

Fig. 4
figure 4

Graphical representation of IPQ results. The IPQ is composed of three subscales and one additional general item. The general presence item assessed the general “sense of being there”

Relationship between driving performances and VR experience

To study relationships between time of completion and the VR experience measures, a simple linear regression has been fitted to the data (Fig. 5) in order to evaluate the relationship between time of completion and mental load (NASA-TLX workload score), satisfaction of use (USE score), Cybersickness (Graybiel and SSQ scores) and sense of presence (IPQ [35]).

Fig. 5
figure 5

Relationship between mental load and time of completion across three circuits; scatter plots with dashed line: LOESS (LOcally Estimated Scatterplot Smoothing) and confidence intervals; Solid line: Fitted regression line

As the circuits are of increasing complexity, the analysis was carried out for each circuit individually, to take these differences into account. For circuit C1, the interaction between time of completion and these five variables have been found to be statistically not significant (\(R2 = 0.17\), \(p = 0.747\), \(adj.R2 = -0.11\)). Nevertheless, we observe an effect of the NASA-TLX based variable which has been found to be statistically significant and positive (\(\beta = 1.43\), \(95\%\) CI [0.09, 2.77], \(p = 0.037\); \(Std.\beta = 0.72\), \(95\% CI [0.05, 1.39]\)). For C2, the same behavior is observed: there is no significant relationship between time and VR experience variables, except for the NASA-TLX based variable, which appears to be statistically significant and positive (\(\beta = 1.43\), \(95\%\) CI [0.09, 2.77], \(p = 0.037\); \(Std.\beta =0.72\), \(95\%\) CI [0.05, 1.39]). For C3, the linear prediction model is not statistically significant (\(R2 = 0.47\), \(p = 0.079\), adj.\(R2 = 0.26\)), and the NASA-TLX based variable has a statistically significant and positive effect (\(\beta = 1.71\), \(95\%\) CI [0.21, 3.21], \(p = 0.028\); Std.\(\beta =0.63\), \(95\%\) CI [0.08, 1.18]).

We performed the same analysis for the number of collisions in VR condition. For all the circuits, each variable has a statistically non significant effect. Only the effect of NASA-TLX score is statistically significant and positive (\(\beta = 0.08\), \(95\%\) CI \([2.90e-05, 0.16]\), \(t(18) = 2.10\), \(p = 0.050\); Std.\(\beta = 0.57\), 95\(\%\) CI \([2.11e-04, 1.14]\)). Regarding NASA-TLX based variable, H7 is only partially confirmed.

Discussion

Results and discussion

In this study, the comparison between driving performances in VR and R conditions show no statistically difference in easy to moderate difficulty circuits (C1 and C2), both for the number of collisions and the time to completion. On these circuits, only a few collisions were reported in the two conditions which is probably due to the expertise of the participants with respect to the driving of a PWC. The circuit C3 was the most difficult circuit with a statistically significant higher number of collisions in the VR condition. Conversely the time of completion of C3 remained significantly higher in R condition with respect to VR condition for the first round, but not for second round. This could be explained by the mental load, but also by the feeling of safety induced by virtual reality, leading people to take more risks by driving faster. Collisions and time of completion are classically the main assessment criteria during a driving test on standardized circuits [23, 36, 37]. Moreover, the comparison between real and virtual conditions for the task of driving a wheelchair has already been studied, and showed similar results. Indeed, Cooper et al. [38] compared wheelchair driving tasks in virtual and real conditions with two types of joysticks. They demonstrated that there were just few differences in task completions and that the driving in a virtual environment relates to driving in a real environment. Moreover, Archambault et al. [39] also highlighted that the driving performances in a wheelchair simulator were not statistically different between virtual and real environments, although task completion was higher in the simulator for the most difficult task.

Frederiksen et al. [21] also showed that the mental load is increased in VR and hypothesized that this can be due to the fact that VR can consist of more virtual elements (visual stimulation, dynamical virtual objects, etc.) and interactions. This is however not the case in our study as we replicate the real environment strictly in the virtual environment, with no additional objects or visual stimulation. In the other hand, the use of a Head Mounted Display (HMD) device can have an impact on the mental load. Winter et al. [40] also observed that the mental load was significantly higher in immersive condition compared with training without VR. In our experience, we have similar results with an increase in mental load in all the circuits. However, despite an increase in complexity, there are no significant difference in mental load between the 3 circuits. Therefore, it can be assumed that the mental load increase is not sufficient to impair wheelchair driving in circuits C1 and C2. For the most difficult circuit C3, this does not seem to be the case either, despite a higher number of collisions. Nevertheless, we have to take into account that the order of the C1, C2 and C3 was not randomized, there might have been a learning effect for the last round, which could explain the lack of statistical significance. Moreover, performance also seems to be linked to cognitive load, with perhaps a threshold effect. The second hypothesis is that the perception of the risk of collisions in VR is different because there is no possibility of injury in the virtual environment, thus encouraging greater risk-taking by users. We then expect that we could increase the risk perception by enhancing the quality of the virtual environment. In our study, the sense of presence is acceptable and leads to a good immersion in virtual environment, but the realism is reported as rather weak [41]. Gotzelmann et al. [42] observed that adding virtual pedestrians can allow driving in a urban environment conditions, with interactions between the virtual pedestrians and the wheelchair driver in the simulator. At this stage, and for this study, the proposed simulator does not incorporate virtual pedestrians to interact with. Such a feature would eventually increase the realism of our simulator, with an impact on the sense of presence as it adds realness to the virtual content [34]. The sense of presence generally depends also on the type of display, the orientation of the visual field and the visualization of the user avatar. In our study, the avatar is restricted to a blanket covering the legs. This has been done to avoid user-avatar visual similarity mismatch issues which can impact the behavior of participants. We could explore avatar more advanced design to provide more realistic PWC driving in our virtual environment. In addition, we can hypothesize that an improvement of the virtual environment could decrease the mental load. Indeed, Kamaraj et al. showed the importance of the quality of both virtual environment and sense of presence regarding their impact on the mental load [43]. In this study, users reported an acceptable general and spatial presence, but the virtual environment appears insufficient in terms of realism. Improvements are needed in this field for future development.

Contrary to previous studies [19, 44], we observed an acceptable low level of cybersickness, although it could be decreased with further technical improvements. For example, Winter [40] synchronized speed of visual motion in the virtual environment with the physical speed of a treadmill with good results. The technological improvement of such a simulator by adding innovative advanced visual and vestibular perception modalities regarding the PWC motion and dynamics seems to be efficient to reduce motion sickness. Moreover it seems also important to gradually and regularly expose users to immersive VR experiences [45]. In our experimental procedure, each run only lasts a short period of time. The next experimental step will consist of longer immersive and repeated sessions in order to enhance the user virtual experience and to assess learning effects.

Perspectives

The place of the simulators in the rehabilitation process remains to be determined. Indeed, 25\(\%\) of wheelchair accidents are caused by PWC users that represent \(10\%\) of the wheelchair users [4, 46, 47]. In addition, 100 000 wheelchair accidents were recorded in US in 2006, twice as many as in 1991 [48]. This shows that the use of power mobility assistance devices comports risks. In particular, falling and tipping are the most common recorded accidents but with PWC, the most frequent reported accidents are due to direct collisions. Ummat et al. noticed that 7.6\(\%\) of accidents cause injuries to the human environment i.e. people around the wheelchair, and not directly the PWC driver themselves [49, 50]. This shows that the risk is important, and it can be easily understandable that the lack of security sometimes leads users to restrict their use of the wheelchair. This highlights the importance of training regarding PWC driving. Currently, real-life assessment remains the Gold Standard. In the present study users appreciated the simulator, but their satisfaction was greater in real situations than in virtual situations on the USE scale over all domains, even if score remained high in VR conditions. Archambault et al. also noticed that participants were neutral in terms of wanting to continue using the simulator particularly when their performance was good [39]. Hence, simulator may be a promising approach to complement training received in rehabilitation centers, especially in case of learning difficulties. Different scenarios can be imagined to optimize the driving control in total safety. Though, simulators remain an expensive technology which is not accessible to everyone. In the first instance, the ecological approach should be preferred with the possible use of a circuit. Note that our VR-based simulator is currently only at the prototype stage and our study has only involved simulations on driving circuits. The creation of more complex environments is necessary in order to consider simulations of everyday life.

Conclusion

In this paper, we evaluated a power wheelchair driving simulator in virtual reality through a pilot study involving 31 expert power wheelchair users. Driving performance were not statistically different when the same circuit was run with the simulator and in real-life for circuits for C1 et C2. Although results were statistically different for C3 regarding the number of collisions which was higher in VR, there were no significant adverse effects such as cybersickness. Virtual reality may not be totally comparable to a real-life situation, but it is nevertheless a effective tool for putting users in difficulty into safe driving situations. Our study thus demonstrates that the proposed simulator is suitable for driving task scenarios in simulation, and show the value of the simulator for driving training applications. Still, mental load remains high during virtual immersion and raises the question of the use of VR for individuals with cognitive disorders who may be in difficulty in complex situations. If the use of simulator may probably not be proposed to these individuals as a first step in driving training, it clearly can be a useful tool for patients unable to safely drive a PWC in order to help them to acquire the necessary driving skills in safe, adaptable, and repeatable conditions. Further studies should be conducted to assess the quality of skill transfer for novice drivers from the simulator to the real world.

Availability of data and materials

Datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

VR:

Virtual reality

PWC:

Power wheelchair

TBI:

Traumatic brain injury

CP:

Cerebal palsy

WST:

Wheelchair skill test

NASA-TLX:

NASA Task Load questionnaire

USE:

Ease of Use Questionnaire

IPQ:

Igroup Presence Questionnaire

SSQ:

Simulator Sickness Questionnaire

C1:

Circuit 1

C2:

Circuit 2

C3:

Circuit 3

T1:

Trial 1

T2:

Trial 2

References

  1. Kirby RL, Dupuis DJ, MacPhee AH, Coolen AL, Smith C, Best KL, Newton AM, Mountain AD, MacLeod DA, Bonaparte JP. The wheelchair skills test (version 2.4): measurement properties. Arch Phys Med Rehabil. 2004;85(5):794–804.

    Article  PubMed  Google Scholar 

  2. Gallien P, Nicolas B, Durufle A, Colin A, Achille S, Fraudet B. Quality of life of adults with cerebral palsy living in britanny. Ann Phys Rehabil Med. 2015;58:131–2.

    Article  Google Scholar 

  3. Mahajan H, Spaeth DM, Dicianno BE, Collins DM, Boninger ML, Cooper RA. Comparison of virtual wheelchair driving performance of people with traumatic brain injury using an isometric and a conventional joystick. Arch Phys Med Rehabil. 2011;92(8):1298–304.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Kirby RL, Ackroyd-Stolarz SA. Wheelchair safety-adverse reports to the united states food and drug administration. Am J Phys Med Rehabil. 1995;74(4):308–12.

    Article  CAS  PubMed  Google Scholar 

  5. Bigras C, Owonuwa DD, Miller WC, Archambault PS. A scoping review of powered wheelchair driving tasks and performance-based outcomes. Disabil Rehabil Assist Technol. 2020;15(1):76–91.

    Article  PubMed  Google Scholar 

  6. Cromby J, Standen PJ, Brown DJ. The potentials of virtual environments in the education and training of people with learning disabilities. J Intellect Disabil Res. 1996;40(6):489–501.

    Article  PubMed  Google Scholar 

  7. Stott I, Sanders D. The use of virtual reality to train powered wheelchair users and test new wheelchair systems. Int J Rehabil Res. 2000;23(4):321–6.

    Article  CAS  PubMed  Google Scholar 

  8. Harrison C, Grant P, Conway B. Enhancement of a virtual reality wheelchair simulator to include qualitative and quantitative performance metrics. Assist Technol. 2010;22(1):20–31.

    Article  CAS  PubMed  Google Scholar 

  9. Ito T, Shino M, Inoue T, Kamata M. Development of a powered wheelchair driving simulator for research and development use. J Mech Syst Transp Logist. 2009;2(2):90–101.

    Article  Google Scholar 

  10. Archambault PS, Blackburn É, Reid D, Routhier F, Miller WC. Development and user validation of driving tasks for a power wheelchair simulator. Disabil Rehabil. 2017;39(15):1549–56.

    Article  PubMed  Google Scholar 

  11. Mahajan HP, Dicianno BE, Cooper RA, Ding D. Assessment of wheelchair driving performance in a virtual reality-based simulator. J Spinal Cord Med. 2013;36(4):322–32.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Bangay S, Preston L, et al. An investigation into factors influencing immersion in interactive virtual reality environments. In: Riva G, et al., editors. Studies in health technology and informatics. Oxford: IOS Press; 1998. p. 43–51.

    Google Scholar 

  13. Lackey SJ, Salcedo J, Szalma JL, Hancock PA. The stress and workload of virtual reality training: the effects of presence, immersion and flow. Ergonomics. 2016;59(8):1060–72.

    Article  CAS  PubMed  Google Scholar 

  14. Pirker J, Kopf J, Kainz A, Dengel A, Buchbauer B. The potential of virtual reality for computer science education-engaging students through immersive visualizations. In: Pirker J, editor. 2021 IEEE conference on virtual reality and 3D user interfaces abstracts and workshops (VRW). Lisbon: IEEE; 2021. p. 297–302.

    Chapter  Google Scholar 

  15. Ferdinand J, Soller S, Hahn J-U, Parong J, Göllner R. Enhancing the effectiveness of virtual reality in science education through an experimental intervention involving students’ perceived usefulness of virtual reality. Technol Mind Behav. 2023. https://doi.org/10.1037/tmb0000084.supp.

    Article  Google Scholar 

  16. Stanney K, Lawson BD, Rokers B, Dennison M, Fidopiastis C, Stoffregen T, Weech S, Fulvio JM. Identifying causes of and solutions for cybersickness in immersive technology: reformulation of a research and development agenda. Int J Hum Comput Interact. 2020;36(19):1783–803.

    Article  Google Scholar 

  17. Bos J, Van Leeuwen R, Bruintjes T. Motion sickness in motion: from carsickness to cybersickness. Ned Tijdschr Voor Geneeskd. 2018;162:1760–1760.

    Google Scholar 

  18. Mazloumi Gavgani A, Walker FR, Hodgson DM, Nalivaiko E. A comparative study of cybersickness during exposure to virtual reality and“classic’’ motion sickness: are they different? J Appl Physiol. 2018;125(6):1670–80.

    Article  PubMed  Google Scholar 

  19. Duzmanska N, Strojny P, Strojny A. Can simulator sickness be avoided? A review on temporal aspects of simulator sickness. Front Psychol. 2018;9:2132.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Weech S, Kenny S, Barnett-Cowan M. Presence and cybersickness in virtual reality are negatively related: a review. Front Psychol. 2019;10:158.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Frederiksen JG, Sørensen SMD, Konge L, Svendsen MBS, Nobel-Jørgensen M, Bjerrum F, Andersen SAW. Cognitive load and performance in immersive virtual reality versus conventional virtual reality simulation training of laparoscopic surgery: a randomized trial. Surg Endosc. 2020;34:1244–52.

    Article  PubMed  Google Scholar 

  22. Vailland G, Devigne L, Pasteau F, Nouviale F, Fraudet B, Leblong E, Babel M, Gouranton V. Vr based power wheelchair simulator: Usability evaluation through a clinically validated task with regular users. In: Vailland G, editor. 2021 IEEE virtual reality and 3d user interfaces (VR). Lisboa: IEEE; 2021. p. 420–7.

    Chapter  Google Scholar 

  23. Leblong E, Fraudet B, Devigne L, Babel M, Pasteau F, Nicolas B, Gallien P. Swadapt1: assessment of an electric wheelchair-driving robotic module in standardized circuits: a prospective, controlled repeated measure design pilot study. J NeuroEng Rehabil. 2021;18:1–12.

    Article  Google Scholar 

  24. Vailland G, Grzeskowiak F, Devigne L, Gaffary Y, Fraudet B, Leblong E, Nouviale F, Pasteau F, Le Breton R, Guégan S. User-centered design of a multisensory power wheelchair simulator: towards training and rehabilitation applications. In: 2019 IEEE 16th International Conference on Rehabilitation Robotics (ICORR), 2019;pp. 77–82. IEEE

  25. Vailland G, Gaffary Y, Devigne L, Gouranton V, Arnaldi B, Babel M. Vestibular feedback on a virtual reality wheelchair driving simulator: A pilot study. In: Proceedings of the 2020 ACM/IEEE International Conference on Human-robot Interaction, 2020; pp. 171–179

  26. Fraudet B, Leblong E, Piette P, Nicolas B, Babel M, Gouranton V, Devigne L, Vailland G, Gallien P, et al. Evaluation of powered wheelchair driving performance in simulator compared to driving in real-life situations. the simadapt (simulator adapt) project: a pilot study. Res Sq. 2022. https://doi.org/10.21203/rs.3.rs-1711205/v1.

    Article  Google Scholar 

  27. How T-V, Wang RH, Mihailidis A. Evaluation of an intelligent wheelchair system for older adults with cognitive impairments. J Neuroeng Rehabil. 2013;10(1):1–16.

    Article  Google Scholar 

  28. Kirby RL, Miller WC, Routhier F, Demers L, Mihailidis A, Polgar JM, Rushton PW, Titus L, Smith C, McAllister M. Effectiveness of a wheelchair skills training program for powered wheelchair users: a randomized controlled trial. Arch Phys Med Rehabil. 2015;96(11):2017–26.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Hart SG. Nasa-task load index (nasa-tlx); 20 years later. In: Hart SG, editor. Proceedings of the human factors and ergonomics society annual meeting. Los Angeles: Sage publications; 2006.

    Google Scholar 

  30. Xiao Y, Wang Z, Wang M, Lan Y. The appraisal of reliability and validity of subjective workload assessment technique and nasa-task load index. Chin J Ind Hyg Occup Dis. 2005;23(3):178–81.

    Google Scholar 

  31. Gao M, Kortum P, Oswald F. Psychometric evaluation of the use (usefulness, satisfaction, and ease of use) questionnaire for reliability and validity. In: Gao M, editor. Proceedings of the human factors and ergonomics society annual meeting, vol. 62. Los Angeles: Sage Publications; 2018.

    Google Scholar 

  32. Kuis EE, Goossensen A, van Dijke J, Baart AJ. Self-report questionnaire for measuring presence: development and initial validation. Scand J Caring Sci. 2015;29(1):173–82.

    Article  PubMed  Google Scholar 

  33. Miller E 2nd, Graybiel A. A provocative test for grading susceptibility to motion sickness yielding a single numerical score. Acta Otolaryngol Suppl. 1970;274:1–20.

    PubMed  Google Scholar 

  34. Melo M, Gonçalves G, Vasconcelos-Raposo J, Bessa M. How much presence is enough? Qualitative scales for interpreting the igroup presence questionnaire score. IEEE Access. 2023;11:24675.

    Article  Google Scholar 

  35. Jurnet IA, Beciu CC, Maldonado JG. Individual differences in the sense of presence. In: Jurnet IA, editor. Proceedings of presence 2005: the 8th international workshop on presence. London: University College London; 2005. p. 133–42.

    Google Scholar 

  36. Sharma V, Simpson R, LoPresti E, Schmeler M. Evaluation of semiautonomous navigation assistance system for power wheelchairs with blindfolded nondisabled individuals. J Rehabil Res Dev. 2010;47(9):877–90.

    Article  PubMed  Google Scholar 

  37. Boucher P, Atrash A, Kelouwani S, Honoré W, Nguyen H, Villemure J, Routhier F, Cohen P, Demers L, Forget R. Design and validation of an intelligent wheelchair towards a clinically-functional outcome. J Neuroeng Rehabil. 2013;10(1):1–16.

    Article  Google Scholar 

  38. Cooper RA, Spaeth DM, Jones DK, Boninger ML, Fitzgerald SG, Guo S. Comparison of virtual and real electric powered wheelchair driving using a position sensing joystick and an isometric joystick. Med Eng Phys. 2002;24(10):703–8.

    Article  PubMed  Google Scholar 

  39. Archambault PS, Tremblay S, Cachecho S, Routhier F, Boissy P. Driving performance in a power wheelchair simulator. Disabil Rehabil Assist Technol. 2012;7(3):226–33.

    Article  PubMed  Google Scholar 

  40. Winter C, Kern F, Gall D, Latoschik ME, Pauli P, Käthner I. Immersive virtual reality during gait rehabilitation increases walking speed and motivation: a usability evaluation with healthy participants and patients with multiple sclerosis and stroke. J Neuroeng Rehabil. 2021;18(1):68.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Heidicker P, Langbehn E, Steinicke F. Influence of avatar appearance on presence in social VR. In: Heidicker P, editor. 2017 IEEE symposium on 3D user interfaces (3DUI). Los Angeles: IEEE; 2017. p. 233–4.

    Chapter  Google Scholar 

  42. Götzelmann T, Kreimeier J. Towards the inclusion of wheelchair users in smart city planning through virtual reality simulation. In: Götzelmann T, editor. Proceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive Environments. New York: ACM Digital Library; 2020. p. 1–7.

    Google Scholar 

  43. Kamaraj DC, Dicianno BE, Mahajan HP, Buhari AM, Cooper RA. Stability and workload of the virtual reality-based simulator-2. Arch Phys Med Rehabil. 2016;97(7):1085–92.

    Article  PubMed  Google Scholar 

  44. John NW, Pop SR, Day TW, Ritsos PD, Headleand CJ. The implementation and validation of a virtual environment for training powered wheelchair manoeuvres. IEEE Transact Vis Comput Gr. 2017;24(5):1867–78.

    Article  Google Scholar 

  45. Golding JF. Handb Clin Neurol. Motion sickness. 2016;137:371–90.

    CAS  Google Scholar 

  46. Kaye HS, Kang T, LaPlante MP. Mobility device use in the United States, vol. 14. Washington, D.C: National Institute on Disability and Rehabilitation Research, US Department of Education; 2000.

    Google Scholar 

  47. Vignier N, Ravaud J-F, Winance M, Lepoutre F-X, Ville I. Demographics of wheelchair users in France: results of national community-based handicaps-incapacités-dépendance surveys. J Rehabil Med. 2008;40(3):231–9.

    Article  PubMed  Google Scholar 

  48. Xiang H, Chany A, Smith GA. Wheelchair related injuries treated in us emergency departments. Inj Prev. 2006;12(1):8–11.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  49. Ummat S, Kirby RL. Nonfatal wheelchair-related accidents reported to the national electronic injury surveillance system. Am J Phys Med Rehabil. 1994;73(3):163–7.

    Article  CAS  PubMed  Google Scholar 

  50. Kim D-J, Lee H-J, Yang Y. The status of accidents and management for electronic assistive devices among the handicapped. Korean J Health Serv Manag. 2016;10(3):223–34.

    Article  Google Scholar 

Download references

Acknowledgements

We thank following individuals for their expertise and assistance throughout all aspects of our study: Marie Dandois, Estelle Cézé, Estelle Poupelin and Elodie Le Toullec, from LAB Pôle Saint Hélier.

Funding

This work is carried out as part of the INTERREG VA FMA ADAPT project “Assistive Devices for empowering disAbled People through robotic Technologies” (adaptproject.com). The Interreg FCE Programme is a European Territorial Cooperation programme that aims to fund high quality cooperation projects in the Channel border region between France and England. The Programm is funded by the European Regional Development Fund (ERDF).

Author information

Authors and Affiliations

Authors

Contributions

The authors cited have all contribute to this study. They declare no Conflict of interest and consent to this publication.

Corresponding author

Correspondence to Bastien Fraudet.

Ethics declarations

Ethics approval and consent to participate

Only participants who consent to participate have been included in this study. This study received the ethical approval n\(^\circ\)2019-A001306-51 from Comité de Protection des Personnes Sud Mediterranée IV. The trial has been registered the 19/11/2019 on ClinicalTrials.gov in ID: NCT04171973.

Consent for publication

All authors consent for publication

Competing interests

The authors declare that they have no competing interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fraudet, B., Leblong, E., Piette, P. et al. Evaluation of power wheelchair driving performance in simulator compared to driving in real-life situations: the SIMADAPT (simulator ADAPT) project—a pilot study. J NeuroEngineering Rehabil 21, 60 (2024). https://doi.org/10.1186/s12984-024-01354-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12984-024-01354-5

Keywords