Immersive Portraits of COVID: [Pause] will focus on 3D data-visualization art in an immersive 3D display environment portraying COVID as abstract images of the viewer’s face and faces of the frontline worker of the COVID19 pandemic in real-time on a layered 3D projection surface (an abstract form of giant mask). With artistically interpreted data visualization, the exhibition aims at making people pause and reflect on how the COVID19 pandemic has affected our lives, how the world’s health and other frontline workers displayed the best of humanity, and how we cope with daily challenges. The real-time dynamic portraits of the audiences and people projected over a dark background of the COVID data will create an immersive interactive environment using spatial augmented reality (AR). COVID data, such as geographic region, socioeconomic class, vaccination, and different impacts on diverse demographic groups, will be retrieved from publicly available sources to create stories aligned with a positive message. An additional goal is developing an environmental intervention using spatial AR to offer a therapeutic experience for the audience and the built environment’s occupants. This project is a pilot for larger future applications demonstrating the emotional and psychological benefits of AR in everyday settings such as an office, a health care facility, a classroom, or a home.

Immersive technology (virtual reality, AR, mixed reality) is rapidly growing and reshaping storytelling with its unique ability to build empathy and to engage audiences. Specifically, projection mapping base spatial AR has the power to convert a built environment into a memorable exhibition space. The built environment significantly influences mental health. Myriad studies demonstrate the impacts of environmental elements on issues such as stress, depressive symptoms, and behavioral disturbances. About 1 in 4 American adults has a diagnosable mental disorder (hopkinsmedicine.org). The COVID19 pandemic brought many changes to how we live; uncertainty, new daily routines, social isolation, resulting in a significant increase in the number of people reporting symptoms of anxiety or depressive disorder. We propose to convert the built environment into a therapeutic agent using spatial AR with meaningful live data.

The exhibit will contain five main elements: 1. projection surfaces (i.e., multiple screens integrated to represent a built environment), 2. ultra-short-throw 3D projector, 3. Microsoft Azure Kinect for video, sound, and motion tracking, and 4. a mobile workstation to process selected COVID data and videos in real-time using ToughDesigner. The display area will be darkened to best view immersive visuals. A podium will be installed in front of the display area near the viewer station to house the projector and the workstation. The Kinect sensor on the podium will capture the viewer’s face.

Dr. So-Yeon Yoon is an associate professor of Design + Environmental Analysis at Cornell University (yoon.human.cornell.edu) and the director of Design-User Experience-Technology (DUET) Research Lab. Yoon’s research focuses on design for optimal user experience employing emerging technology to understand the human response to the physical environment as well as the virtual environment. She was the president of the Design Communication Association from 2018 to 2021.

Jintae Kim is the Co-Founder and Director of TY Studio (tystudio.kr). Kim is a new media installation artist in Seoul, Korea. Kim’s work has been shown in theaters and exhibition halls. Some of his interactive installation art pieces have been featured in commercial settings.

Anne Seo-Young Lee is a PhD student in Design and Environmental Analysis and researcher of DUET lab. She received her Master’s degree in Information Science and Communication with an emphasis on Human-Computer Interaction and User Experience. Her research interests are in Human-Computer Interaction (HCI), especially, design evaluation using emerging technologies to understand psychological and behavioral responses to different design elements in virtual environments. She is also interested in considering VR/AR/MR as a design research methodology tool.