Case: Combating VR Sickness in Immersive Installation for Public Spaces, pt. 1/2
November 21, 2017
Red Sun | A multisensory experience chamber with VR, ambient sound and scent
May 11, 2019

Combating VR Sickness
in Immersive Installation for Public Spaces, pt. 2/2

User Experience Case Study

Designed & developed by Mariam Zakarian since 2016.
Research via 88page Master Thesis in Media Technology (Aalborg University).
Exhibited in galleries, museums, conferences and other public and private events in Europe, East Asia and USA.

OVERVIEW
In part 1 of this UX Design case study, we defined VR Sickness and what causes it. I also explained how Amaryllis VR : Ocean is designed specifically for public spaces and must be able to accommodate a large variety of users, and I briefly mentioned the preliminary tests on Version 1.0 and 2.0, which provided me with the necessary data to continue developing the VR experience.
In this article, I will dive deeper into user testing to combat VR Sickness.
If you haven't read Part 1, you can find it here.
Amaryllis VR : Ocean was designed specifically for public spaces:
museums and galleries welcoming a large variety of audiences from different backgrounds.
VERSION 3.0 UX TESTING
At this point I had a functional Proof Of Concept of Amaryllis VR : Ocean. For Version 3.0 I replaced the majority of the placeholder assets with my own 3D models, textures, animations, particle effects etc. and finished the design of the experience itself.
Version 3.0 was tested in November 2016. Qualitative and quantitative data were gathered, of which the quantitative data dealt specifically with the question of VR Sickness.
METHODOLOGY:
For the test I used the well-known Simulation Sickness Questionnaire (linked under Sources at the end of the article) with 16 questions related to all of the symptoms which are typical for VR Sickness. The questions are clustered into 3 main groups: Oculomotor, Disorientation and Nausea.
The entire questionnaire was filled by users without supervision prior to testing the VR installation, as well as after. The pre-test questionnaire also asked about the user's gender, age and occupation, their experience with VR and the HTC Vive. Finally, there was a section where users could self-report any other relevant information they might want to disclose.
Users were tested individually rather than as part of a group and testing times went from early morning to early evening for 6 days.

The questionnaire was coupled with observation of user behavior during testing, with notes about movements, gestures and other expressions, also documented via audio and video recordings. Timestamps were taken for the start and end of each test.
To ensure that every user had the same basic knowledge of room-scale VR, everyone was assisted in and out of the VR HMD, and all users received a short introduction to moving within the boundaries of the VR playspace.

After welcoming users, the test began. The different phases were as follows:
1

Pre-test Questionnaire

2

Brief intro to VR, helping user into VR

3

User tests installation until they want to stop

4

Helping user out of VR

5

Post-test Questionnaire

SETTING:
An actual gallery or museum was not available at the time, so the lobby of Aalborg University in Copenhagen was chosen because of the similarity: A casual area with a moderate amount of traffic with a low to moderate amount of background noise and sensory distractions.
Important distinction: users are not specifically there to experience art and are often on their way somewhere, which means they may be in a hurry.
SETUP:
A 3x3m floor area was marked for the HTC Vive playspace. The hardware was an HTC Vive and Sennheiser HD 600 headphones running on a VR-ready Vision desktop PC. The Vive base stations were mounted on stands at opposite corners of the playspace.
A Canon DSLR camera and a Samsung Galaxy Note 4 phone were used to capture video and still images of users during testing, and audio from interviews.
Top-down view of the VR setup for the test
Setup of the HTC Vive playspace in the SteamVR app
USERS:
The ideal test group would consist of random users of all ages and background who visit art museums and galleries. 42 people were selected from as many different demographic groups as possible, based on their availability on test days. The youngest person was 14 years old, the oldest; 64, and the average age was 31. The largest represented group was made up of university students, but the backgrounds were diverse, ranging from artists to speech therapists and programmers.

Gender Distribution

  • Male
  • Female

Has tried VR before?

  • Never
  • Yes

Has tried HTC Vive before?

  • Never
  • Yes

Uses VR Regularly?

  • No
  • Yes
VERSION 3.0 UX TEST RESULTS
PRE-TEST QUESTIONNAIRE RESULTS:
A few users self-reported discomfort, light hangover, mild pain and illness prior to testing. In all cases, their pre-test scores for VR Sickness symptoms were either mild or nonexistent. Although it would have been preferable to test on users with no symptoms presented in the pre-test questionnaire, none were evaluated to be too ill for trying VR, and no one stopped the installation out of discomfort.

The most reported pre-test symptoms were, in order of frequency: Fatigue, "Fullness of the Head", Eye Strain, General Discomfort, Difficulty Focusing Vision, Difficulty Concentrating, Headache, Blurred Vision and Sweating. These were therefore also expected to be high in the post-test questionnaire results.

Most were related to Oculomotor Disturbances and all symptoms were mild, except for Fatigue, Eye Strain and "Fullness of the Head", which were moderate in under 12% of cases. The majority of the pre-test symptoms could likely be attributed to studying or working hard before the end of the school semester and the holidays.

Pre-test vs Post-test Symptoms Reported (% of Users)

  • Pre-test symptoms
  • Post-test symptoms
POST-TEST QUESTIONNAIRE RESULTS:
In all instances, there was an increase in severity of symptoms except for Fatigue and Headache. Surprisingly, post-test scores for Burping and Difficulty Concentrating decreased.

The most dramatic increases were as follows:

● Eye Strain, Dizziness with Eyes Open and Blurred Vision (+19%)
● Dizziness with Eyes Closed (+14%)
● Difficulty Focusing Vision, Salivation Increasing, Sweating, Nausea and Vertigo (+12%)
All increased symptoms were reported to be mild, except for Eye Strain and "Fullness of the Head", which had a few moderate cases with a 9.5% and 7.6% increase, respectively. No statistically significant severe symptoms were reported.
In all of the combined results from testing Version 3.0,
79% of users did not report any adverse symptoms.

21% of all users reported symptoms.
Of these, the results were mild for 17%,
moderate for 0.3%
and severe for only 0.01% of the entire pool of answers.

User testing sessions on Version 3.0 at Aalborg University in Copenhagen
EVALUATION AND DISCUSSION
PREEXISTING SYMPTOMS:
Among the 42 users in the quantitative VR Sickness test, there was from the start a possible problem that a good portion of the group presented some symptoms already in the pre-test questionnaire. To get more accurate results, it would be a good idea to conduct the test with a larger group of healthy people.

If users had no preexisting symptoms before trying the installation, they also reported no or low symptoms in the post-test questionnaire. However, if they did have symptoms before, these usually increased in severity in the post-test questionnaire, particularly for Eye Strain, Dizziness with Eyes Open and Blurred Vision. It is logical that trying VR would be uncomfortable for an already unwell body.

The questionnaires were filled in on different times and days over a six day period. The time of day and the activities prior to the test would likely be a significant contributing factor to the differences in physical state of the users prior to starting the test.

INTERPRETING RESULTS:
Based on the results, Version 3.0 of the installation caused symptoms which might be related to VR Sickness in 21% of users, but none experienced severe enough symptoms to ask to stop the installation and test.

Oculomotor symptoms were the most widely reported. These show disturbances in visual processing, which often arise from not viewing the VR simulation clearly.

Disorientation and Nausea symptoms had low scores, and at least 1 of the symptoms (Sweating) could also be attributed to the tendency in users to walk around, kneel, reach and use their bodies to explore the playspace of the VR installation and thus might not have anything to do with VR Sickness.
It is difficult to assess how many of the symptoms were affected by Amaryllis VR : Ocean and how many were caused by preexisting or external factors, or the HMD itself. Particularly in the case of Oculomotor disturbances, it is worth noting that fogging of the VR lenses, light reflections from below the HMD, and wrong focus settings are common problems. For comparative data analysis it would be interesting to examine a comprehensive study of the HTC Vive HMD where a variety of different simulations are tested.
SURPRISES:
Difficulty Concentrating, and Burping actually decreased in the post-test questionnaire. In the case of the former, it could be interesting to examine if the slow pace of Amaryllis VR : Ocean has positive effects on people suffering from stress, tension, anxiety or difficulty concentrating - particularly considering the feedback from several users that the installation was calming and that it made them want to meditate.
Alternative use cases, such as using the installation for relaxation or meditation were brought up by several users, and the idea of creating a parallel application specifically for this purpose might be interesting.
ISSUES WITH THE QUESTIONNAIRE:
A number of problems were experienced with the questionnaire itself. Firstly, the questions had to be translated to Danish for users
with poor English language skills, and secondly, the questions in the questionnaire are so brief, that some of them may have been misinterpreted by users. For example, the test does not contain precise explanations of what is meant by the different symptoms, and many could be misinterpreted.
One user thought that “Stomach Awareness” could include menstrual cramps, and it was necessary to add the word “vision” to the “Difficulty Focusing” question, as it was unclear whether the user was being asked about their eyes or their concentration in general.

Importantly, the users who noted symptoms related to Disorientation, such as Vertigo, explained that their symptoms were caused by the rising waves of the Ocean in the installation, giving them a feeling of "floating", but that the induction of Vertigo was NOT a negative symptom for them. The wording of the questionnaire should thus perhaps be more precise in asking of users about negative symptoms rather than something that might contribute to a positive feeling of presence in VR.

Arguably, the most important thing to note is that the questionnaire was developed for the US Navy for military aviators -a population of people less prone to experience the symptoms of VR Sickness. This greatly influences the suggested categorization of symptoms into mild, medium or severe. Symptoms which may seem negligible to aviators might indeed be felt as severe to non-aviators.
SELF-REPORTING:
The reliability and accuracy of the data in questionnaires is not fully accurate, in that people’s accounts of what they feel may not always correspond to what they actually feel. Furthermore, the reaction of the same user may vary every time they use VR, e.g. because of adaptation to the technology and changes in their health and general fitness from day to day.
PRIMING:
Finally, it is important to note the problem of priming. Theoretically, it is possible that having users fill in a questionnaire of symptoms before testing may have resulted in them feeling more aware of their symptoms, which could result in higher reports of symptoms post-test.
In the future, it would be ideal to only select users for the test if they report no symptoms in the pre-test questionnaire. This was not possible due to limited time and availability.
Testing Version 4.0 at Creative Coast Festival 2017
FURTHER DEVELOPMENT AND PREMIERE
The considerations from the qualitative and quantitative tests were used to develop Version 4.0 of the installation, in which all assets were replaced and refined. Animation, timing and visual design was also improved and re-built to further assist users in preventing VR Sickness symptoms -effects such as using visual anchors and subtle cues to orient users in space while in VR proved to be effective in reducing symptoms further. In this version I also finally replaced the placeholder sound with music I composed for the installation.
The premiere of Amaryllis VR : Ocean at Galleri Oxholm, Copenhagen, Denmark
Version 4.0 was stress tested at Creative Coast Festival in mid 2017, where a large volume of users tried the installation and each was asked if they had VR Sickness-related symptoms. No symptoms were reported or observed.

Version 5.0 was the final version of the Amaryllis VR : Ocean art installation, in which only detail work was improved. It premiered at Galleri Oxholm in Copenhagen in end of August 2017 and continued out into the world since.

SOURCES
● Rebenitsch, L., & Owen, C. (2016). Review on cybersickness in applications and visual displays. Virtual Reality, 20(2), 101–125. https://doi.org/10.1007/s10055-016-0285-9

● Uliano, K. C., Lambert, E. Y., Kennedy, R. S., & Sheppard, D. J. (1986). The effects of asynchronous visual delays on simulator flight performance and the development of simulator sickness symptomatology.

● Kennedy, R. S., Lane, N. E., Berbaum, K. S., & Lilienthal, M. G. (1993). Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness. The International Journal of Aviation Psychology, 3(3), 203–220. Retrieved from
http://web.a.ebscohost.com.zorac.aub.aau.dk/ehost/pdfviewer/pdfviewer?sid=d6246959-c41b-4c52-b460-
6d8d8026117c%2540sessionmgr4010&vid=1&hid=4002