RAPTURE VR A VIRTUAL-REALITY HAND-GESTURE PSYCHEDELIC-RHYTHM GAME. - POSTMORTEM

Eliciting Retention of Memories and Provoking Self-Examination via Sensory Emotional Ques and Triggers in a Virtual Reality Context.

The past few months of Development have been extremely challenging but nevertheless rewarding. Once the original idea for Rapture VR was approved from my Statement of Intent, I became extremely excited to have the opportunity to bring my vision to life and experiment with new technologies. I must admit that coming up with an intuitive system around an input method, which is extremely new and experimental, has been a challenge with a non-stop need to playtest and work around various technological limitations. In this document I will aim to outline a summary of the passed development on Rapture VR and give an updated explanation and look at what it has become over the past few months.

1.Rapture VR Project Overview

Rapture VR is a Psychedelic-Rhythm game running natively on the Oculus Quest 2, exclusively for Hand-Tracking enabled devices.

The premise of the artefact is simple: The Player is a Non-Corporeal being which is trying to transcend by fighting the various aspects which cause humans’ suffering. The artefact focuses on Level 1 – Anger, to give an idea of the possibilities of the systems built around the project.

The idea is to craft an experience where the Player is influenced by the game subconsciously during the timeline of the game using various methods derived from research. One prominent aspect to mention here is how Emotion is integral to how humans learn and process memory. VR is the perfect vehicle to explore this, and I believe that Rapture VR can illustrate some of these concepts applied in practice.

The Project is split into 3 separate systems (1 game Architecture) to be able to achieve the desired effect:

1. A seamless and intuitive Hand-Gesture based system for the games interactions, requiring no controllers building upon the idea that interactions should feel like magic. This helps Immerse the player in the world and help them “get lost” in a more “suggestable” flow state.

2. A Visually Captivating and Immersive Level design with Models and Animations based specifically around each level’s Theme and Topic. A Level that Responds Accordingly to the Players interactions Live and shifts depending on what is happening to further emphasise the emotional impact on the player.

3. A Soundtrack Specifically designed around the theme of each level with the Goal of using Audio as a Narrative Device and Illustrate the idea to the players. The Games Music will be Themed around each Aspect of Human Suffering and the Player’s Battle with them, leaving it to the players to fight the Final Battle in their own minds after.

The Boss In-Game, as well as the Level itself respond Visually both to the Music and the Players interactions with the Main Objective to tie together the Emotional Impact on Players. The hope is that this will have a few impacts on the player after the game’s completion:

1. The Players will Remember and Retain their memories of the Music and the “Battle” between the In-Game Character and the Level Boss due to the Emotional cues. They will hopefully carry this over into further thought after they have played the game and taken off the headset. This will hopefully allow them to embody the Underlying message of hope, love, happiness and that they can Heal, put into each Level Song and Theme.

2. They will have a fun experience with a game made to be as fun as it is impactful. The intuitive gesture system and the Boss-Player mechanics of the fight aim to give the player a challenging experience in order to be able to defeat the boss.

3. They will seek to keep playing until defeating each boss, as that is the only way to unlock the next sequential Thematically, Musically, Mechanically and Visually unique level to progress the game.

I believe that Level 1- Anger- represents these ideas quite well and serves as a good overview of how all systems are meant to work together. I believe that the original pitched goals and systems from the Statement of Intent which also match up with ones above have been met to the best of my abilities as I have followed the Statement of Intent as a plan. The Difference is that a lot more Research has been put into Psychology, Emotions and Impact on a Players Perception. I have had to upgrade my development device to Oculus Quest 2 and non-stop playtest with other people to Ensure that the game can “Influence” the player accordingly.

2.Rapture VR Project Development and Evaluation

I must admit that ween I pitched this project, I severely underestimated the amount of R&D as well testing it would take to bring it to life. Although all ideas sounded good on paper, I had never touched a VR headset for development before, built anything for Android or done Music/Modelling.

I joked around with my lecturers that “I Stretched the T-Shaped Learning a bit too far this time” and although it has been painfully true It has taught me a lot of valuable skills from Music Production to 3D Modelling and VR development.

Everything in Rapture VR has been made by me with a bit of guidance from friends from other courses. They have helped me learn the proper tools and techniques of working in their respective fields, however none of them know how to model in VR as I used Gravity Sketch or produce music in Reaper as they used Logic Pro. Every single asset in the game has been made in the past 2 months by me.

With that being said, I believe I can give an accurate Evaluation of the Passed Development by splitting it into my thoughts around the 3 separate systems outlined above as well as a section for the R&D around Emotion and Psychology which informed all decisions:

1. Right after the Statement of Intent got approved, I decided to focus my efforts on a single Level. I wanted this level to Illustrate all aspects of the game which all revolve around Influencing the Player, so they come out of the experience with the desired effect.

I put a lot of Research into Emotion and how it influences Learning and Memory in people. Some Prominent Research on the topic by Elizabeth A. Kensinger from Boston College on the topic shows how much emotions influence us remembering details or memories at all. In her article “Remembering the Details: Effects of Emotion” (Kensinger 2009) she outlines the natural processes which our brains use to process memories and how negative memories are more remembered than others due to further engagement of sensory processes. Another prominent piece of research I based this on is an article by Centre for Intelligent Signal and Imaging Research (CISIR) from the Universiti Teknologi PetronasSeri Iskandar, Malaysia titled “The Influences of Emotion on Learning and Memory” (Tyng 2017)also Illustrates what an impact specific emotional situations have on each specific part of the human brain and how much the cognitive process is influenced when adding emotion in terms of attention, learning, memory etc.

All of this has served as the basis for crafting the Rapture VR prototype with all 3 separate systems being crafted around this concept.

2. Music- The Music in the Game Has been Mixed in Reaper and Written in such a way that It provides 4 separate Musical Parts which are alternating from the Boss voice to the subtle Player voice, The Level and Visuals are made to React to these changes in tone and feeling to further enhance the experience. Each Lyrical part is meant to elicit different Questions in the Player and Give them things to think about. The Music itself is made so each part sounds different to the other and elicits specific emotions catered to the Experience.

In Level 1- Anger the first music part (with its corresponding thematic vocal) is meant to elicit a feeling of uneasiness, stress, Anger and Anxiety. The following second part is meant to bring a tone of hope, light, perseverance and triumph over raw power. After a third response from the Boss voice, the 2 parts can be heard together both Musically and Vocally in the last part Signifying a “Final Battle” where the player has the last chance to do enough damage to beat the boss.

The biggest challenge and hurdle with this aspect of the Game is that the Mechanical Audio-Reactivity as well as the Emotions and perceptions elicited by the Music must be tested throughout development. I have had to bring in friends and peers in periodically to test the various separate Music Sounds as well as the Vocals to ensure they give the proper feeling. This has further had to be tested for accurate Rhythm reactivity in-game.

3. Visuals – The visual aspect of the game has been just as challenging as the Music. The Initial plan for the Models as well as the basic blackout was easy. The real problems came when I had to model the ideas in VR. I chose to work in Gravity Sketch on the Oculus Quest as I am not familiar with traditional tools such as Maya or Blender. I found that while some ideas translated quite well to VR others had to be reworked as they did not work well from the Players VR Perspective. After a lot of tweaking during the past month I was able to model multiple separate pieces which can be seen in-game. The process of Concept-Creation with the models was easy as they would not change during gameplay much and I could animate them freely. From all the tools mentioned in the Statement of Intent, I chose Gravity Sketch as it proved to be the most robust and advanced tool I could find. It was meant for asset creation.

Another problem faced was the tone and feel of the Game. Due to VR being quite new and Oculus Quest being a Standalone Device I had to work though some limitations to give the proper feel of the game right. I started working with the Built-In Render Pipeline as I thought it would work best on the Quest but after further tests and a lot of troubleshooting, I kept facing a lot of problems around implementing Post-Processing and various needed effects. I ended up switching the project to the Universal Render Pipeline in Unity and Migrating all my assets to that.

It served to be a great choice as within a day after that All the changes I need had been implemented and the Game looked as it needed.

4. Mechanics – Originally, I set out to make an intuitive and modern hand-gesture based system but severely overestimated how much the technology had progressed. After I built a system to capture and use various Hand-Gestures using the Hand-Tracking on the Oculus Quest I found that Accuracy of Tracking is not as high quality as I though it was. Since writing this report, Oculus have enabled Experimental 60hz High Refresh Rate Hand Tracking on the Quest 2 and after testing It has dramatically improved the Gesture-Detection.

Despite these technological hurdles I believe I have managed to illustrate a descent solution regarding the proposed game input-system. It can be used to create all sorts of various gestures with the flexibility to add and edit new ones at any time. Although with limited beta hand tracking, I believe that it is accurate enough to Illustrate how the future might leave traditional Controller-Based input for something more intuitive. Although limited, the system already feels intuitive and like magic for those who learn how to use it, just like a real magic system. It shows that a future for VR Hand Tracking with unique input systems is possible.

Rapture VR is a Fast-Paced game which required users to make fast Gestures and Reactions with their Hands and I believe the needed accuracy of input had been achieved.

All the aspects of development above have had to be accompanied by non-stop player tests to ensure that all parts of the game progress’ in the same direction in terms of the Players Perspective. The Example Level 1 – Anger has been tested piece by piece (or minute by minute in terms of the 5 separate minutes that the Song and Level Duration) with a multitude of different people ensuring a fresh look into on each part. Each tester was asked different questions correlated to the Immersive, Perceptual and Emotional Impact each part and the whole game had on them. Their feedback has been recorded and aspects tweaked throughout development with the sole purpose to keep the Deep Emotional Influence affecting the players throughout the Song and Level. By comparing various answers between players on the same and different versions of the game I was able to do small but significant tweaks to achieve the desired effect.

4.Rapture VR- Main Takeaways Overall, I feel like the Challenge of Making this Project has opened my horizons as a Game Developer to new levels. I believe that although I faced a lot of difficulties which required me to troubleshoot and often backtrack on work, I believe I managed to achieve what I set out to do.

I must say that this project has brought me down to reality with the current state of the VR industry in terms of technology and advancement. It has also given me a realistic outlook and perspective on the Skills, Time and Effort required to Craft any type of Game, Immersive or Interactable Experience in a 3D context.

I believe that the skills I gained in Music and Audio Production, Visual Concepting and 3D Modelling, and Unity and VR Mechanics coding will be of huge help on my journey as a developer forward.

I must admit that I also had a lot of fun having to solve the various challenges around limitations of new and emerging technologies.

The main takeaway from this project for third-year for me is a renewed passion for Immersive Technologies and the Innovations of the future. I am very excited to have the opportunity at Falmouth to Develop on new Tech and aim to push the boundaries of what is possible even further.

The skills gained throughout the making of this project and the extremely valuable research and information I learned will hopefully stick with me for the future. For the time being I believe that Rapture VR- Level 1- Anger is an experience on the Oculus Quest 2 I am certainly very proud of.

References:

TYNG. M Chai, Hafeez Amin, Mohamad Saad, Amir Malik. 24 Aug 2017. The Influences of Emotion on Learning and Memory. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5573739/ : Centre for Intelligent Signal and Imaging Research (CISIR), Department of Electrical and Electronic Engineering, Universiti Teknologi Petronas, Seri Iskandar, Malaysia

KENSINGER, Elizabeth A. 2009. Remembering the Details: Effects of Emotion. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2676782/ : Department of Psychology, Boston College

OCULUS. October 2020. Oculus Quest 2. Consumer Market: Oculus.

OCULUS. 2019. Oculus Quest. Consumer Market: Oculus.

COCKOS. 2006. Reaper. Windows, Mac, Linux: Cockos

GRAVITY SKETCH LIMITED. 2017. Gravity Sketch. Steam, Microsoft Windows: Gravity Sketch Limited.

BLENDER FOUNDATION. 1994. Blender. All common Operating Systems: Blender Foundation.

AUTODESK. 1998. Autodesk Maya. All common Operating Systems: Autodesk