FEATURES

Print

A virtual first-person perspective of war-torn Gaza UTokyo, Al Jazeera collaborate to create VR educational content allowing users to place themselves within real footage from Gaza Strip

November 19, 2024

Israel’s attacks against Palestinian territories dramatically increased following the Hamas-led assault into Israel on Oct. 7, 2023. The widespread destruction of civilian life and infrastructure, along with the statements and actions of the Israeli government and military, has led many experts to raise the alarm over signs of Israel committing genocide against Palestinians. In the Gaza Strip alone, more than 40,000 people have been killed, mostly children and women, and over 100,000 injured so far. According to the World Health Organization, only 17 of Gaza’s 36 hospitals remain partially functioning and some continue to be under siege (as of Oct. 19, 2024).

Accurate reporting and data gathering on the conflict remains difficult. Journalists and humanitarian experts, including United Nations officials, face severe restrictions, arrests and deadly attacks by Israel while trying to conduct their work. The world has come to rely on footage shared on social media and other platforms by civilians, aid workers and the remaining journalists, who provide distressing testimony of the dire conditions in Gaza and beyond.

On Sept. 18, 2024, the University of Tokyo’s Interfaculty Initiative in Information Studies (III) and Al Jazeera Media Network, a Qatar-based global media company, signed a memorandum of understanding to work together to create educational virtual reality content from real footage of the war-damaged Gaza Strip. We spoke with some of the team involved, including Professor Hidenori Watanave, director of the Media Contents Research Initiative at III and digital archive expert; Koki Shigenoi, visiting researcher at III and a specialist in Middle East and conflict studies; Shohei Komatsu, project researcher at III and doctoral student of the Graduate School of Interdisciplinary Information Studies (GSII), as well as award-winning virtual reality developer; and Arafat Madi Shoukri, a senior researcher at the Al Jazeera Centre for Studies, a think tank under the Al Jazeera Media Network. Below are excerpts from the interview.

Image from interactive map of Gaza area showing level of war damage.
Interactive map of war damage across the Gaza Strip. Created by Watanave Lab based on UNOSAT data 2024. Interactive map: https://wtnvlab.maps.arcgis.com/apps/instant/basic/index.html?appid=20920358dfa748f5b94707b8f8b8bc44
 

A new perspective on a yearlong war

── What is the Gaza virtual reality project and how did it come about?

Watanave: My lab focuses on information design, digital archiving and data visualization. We have created many digital archives covering various wars and disasters, and have often collaborated with local people and newspapers. I was surprised when I heard that Israel had attacked the Gaza Strip, so when Mr. Shigenoi approached me with this idea, I wanted to use the same methods I had been using to document natural disasters (such as the Türkiye-Syria earthquake and Maui wildfires in 2023) to create a digital archive for Gaza. We are now working with Al Jazeera Media Network to develop virtual reality (VR) content that will enable viewers to relive real footage taken in war-damaged Gaza from a first-person perspective, either through a VR headset or on a screen.

Shigenoi: My background is in Middle East and conflict studies. Mr. Komatsu is an old friend, so I was aware of his work with Professor Watanave and the Crisis VR system they had developed a couple of years ago. This award-winning system allows multiple users to experience the same VR metaverse (immersive virtual world) space simultaneously and project their avatars into it. I thought this was a strong tool to help bring attention to the current crisis, so I contacted him and suggested taking this system to the Middle East. During my Ph.D., I worked as a visiting researcher at the Al Jazeera Centre for Studies and Hamad Bin Khalifa University (in Doha, Qatar), so I had contacts with a couple of institutions including local universities. Mr. Komatsu introduced me to Professor Watanave, and following a brief conversation, we contacted Al Jazeera to discuss the idea.

Shoukri: As a senior researcher at the Al Jazeera Centre for Studies, our decision to collaborate with Professor Watanave and his team stems from our shared commitment to harnessing cutting-edge technology to deepen global understanding of critical issues. Professor Watanave’s expertise in immersive media aligns with our mission to present the realities of conflict zones in a manner that is as impactful as it is authentic. This partnership allows us to combine our narrative expertise and in-depth reporting with innovative VR technology, providing viewers an unprecedented perspective on the lived experiences of those affected by the Gaza conflict.

Video footage of destruction of hospital entrance, converted for VR. Al-Shifa Hospital was once the largest medical complex in Gaza, and as a hospital, had special protection under international humanitarian law. Israel accused the Hamas military of commanding operations from a hidden base underneath the complex. However, to date, Israel has not provided proof to justify their destruction of the medical facility and deaths or killings of the patients, staff and civilian refugees within. Interactive content: https://gaza.archiving.jp/sequence/entrance.html © Al Jazeera/ Watanave Lab, 2024
 

── There are a number of ongoing crises globally. Why focus on Gaza?

Shigenoi: It is true that there are many tragic conflicts worldwide. How many people paying attention to Gaza are also aware of the civil war in Sudan? However, our time is limited and I believe that within that limited time, it is more meaningful to contribute to areas where one has expertise. Fortunately, I have both contacts and expertise in the Middle East.

Throughout my academic career, I have worked in the humanities and social sciences, focusing on war and international politics, with the belief that I could contribute meaningfully to politics through my studies and work. But now, truly horrific events are unfolding in Gaza, and I feel powerless to take effective action. Decades, or even a century, from now, I think that historians and politicians will criticize us for failing to stop what is essentially a genocide of the Palestinian people. Even so, I feel that, at the very least, we can document and convey what has happened in Gaza and the experiences of its people through technology. It was out of this sense of helplessness that we undertook this project and Al Jazeera is an invaluable partner in this effort.

Shoukri: The level of destruction inflicted by Israeli forces has been staggering, not only in scale but also in its disregard for international laws that prohibit targeting hospitals and medical facilities. Al-Shifa Hospital, like other medical centers, should have been protected as a neutral zone. However, the devastation there reflects a shocking breach of these “red lines.” Despite the immense challenges and constant threats, the resilience and courage displayed by the people in Gaza — especially the medical staff and volunteers at Al-Shifa who continue to work under such dire conditions — have been both humbling and inspiring. Witnessing their commitment to care for the wounded amidst relentless chaos underscores the importance of sharing these stories in as immersive and compelling a way as possible, to raise global awareness of these ongoing violations.

Real-life scenes of chaos

── The footage for this project includes scenes of people at Al-Shifa Hospital when it was first heavily damaged by Israeli attacks in November 2023. What are the technical and personal challenges, and advantages, of working with these videos?

 
Image of premature babies lying next to each other in the hospital.
Image of video footage of premature babies at Al-Shifa Hospital, converted for VR. Israel’s siege of Gaza’s Al-Shifa Hospital in November 2023 created power outages and unsafe conditions, leading to the reported deaths of eight premature babies, while 31 were evacuated. Tragically, Al-Shifa was not the only hospital where babies in intensive care died due to the invasion. Interactive content: https://gaza.archiving.jp/local/babies01.html © Al Jazeera/ Watanave Lab, 2024
 

Watanave: From a technical perspective, artificial intelligence (AI), which we use to create 3D models for VR application from regular video footage, can only detect information from a continuously moving camera. This limits what footage we can use and the quality of the result. However, an important point is that during war or disaster, we cannot shoot the same exact situation again. So, even if the footage isn’t perfect, we have an important record which we can reproduce again and again as needed.

From a conceptual perspective, the videos we are working with were chosen because they have a strong message. The first video I chose shows premature babies lying side by side on a hospital table. It was my first time to see something like this and I found it very shocking and upsetting. Other videos show people either seeking shelter or walking through debris. I think these children and these people are saying “please deliver our message to the world for us.”

Shigenoi: Given that Al Jazeera’s videos are recorded for television, often capturing a scene within a few seconds, we are currently investigating the application of 3D Gaussian splatting (a technique to create 3D scenes from 2D images) to generate high-quality 3D models. Conducting 3D scanning to document damaged objects in war zones presents significant challenges, as does employing 3D photogrammetry (to convert photos into 3D models), which requires capturing images from multiple angles to combine them into a single 3D model. Consequently, we must rely on whatever brief footage can be captured by journalists on the ground.

Our next challenge, using 3D models of crisis scenes, is to make the VR experience more immersive and help viewers understand that there are real people involved, not just numbers. Perhaps like what was done for the project for Ukraine (in which an avatar was used to serve as a guide), for Gaza we could create an avatar of a real Palestinian boy who could tell his own story and you could talk with him.

Video footage of patients outside in the hospital courtyard, converted for VR. Hundreds of patients and thousands of civilians seeking shelter at Al-Shifa Hospital were simultaneously ordered to evacuate while also trapped by Israeli forces. Journalists and medical staff within the hospital reported harrowing interrogations and summary executions carried out by the Israeli military, along with a blockade of supplies, such as fuel, medicines, food and water, to those inside. Interactive content: https://gaza.archiving.jp/sequence/people.html © Al Jazeera/ Watanave Lab, 2024
 

Komatsu: I think the biggest challenge with VR is that it can easily be mistaken for being purely artificial. It is important to explain that this 3D digital archive is created from real war footage. Additionally, we’ve found that a guide is needed to clearly explain the situation. For our project on Ukraine, I created an avatar for a Ukrainian photographer, so they could provide an explanation in a metaverse space and talk to multiple visitors simultaneously. I hope to create a sort of “dark tourism” metaverse, which can help educate people about war.

The key elements of our VR project are "immersive" and "narrative." Our goal is not to deliver an ordinary static VR experience, but rather to develop a first-of-its-kind immersive experience that pushes technological boundaries. Alongside its technological significance, Al Jazeera's journalistic expertise enhances the educational value of our product. By integrating Al Jazeera’s footage and storytelling with our advanced technology, we aim to create impactful VR educational experiences that contribute meaningfully to society.

Shoukri: While traditional footage is powerful, VR brings an additional dimension that allows viewers to “feel” the environment rather than just observe it. This medium enables users to not only see but also virtually “step into” the conflict, evoking empathy and understanding on a deeper, more visceral level. Our goal is to create an experience that goes beyond passive viewing, encouraging viewers to engage more thoughtfully with the reality on the ground and fostering a stronger connection to the human stories behind the headlines.

VR technology as a catalyst for action

Photo of crisis VR with avatar demo.
Crisis VR development demo with Professor Watanave's avatar. A test demonstration shows Watanave's avatar walking through the VR space converted from Al-Shifa Hospital footage. This feature will allow guides to lead multiple visitors through the scene while providing explanations about the situation. © Al Jazeera/ Watanave Lab/ PocketRD Inc., 2024

── The situation in Gaza is being widely recorded and shared through social and traditional media. What do you hope to achieve by adapting this footage into a 3D archive and educational VR?

Watanave: I have two young children and they watch YouTube every day, but not TV programs or the news. This means they don’t see what is happening in Gaza. But we have seen from our previous work on conflict and disaster, such as an exhibition we held this summer about (the atomic bombing in World War II of) Nagasaki, that new technology like 3D digital archives and VR can attract children’s attention and help them understand the impacts of war. Of course, we must consider what we expose them to and how, but I believe in the potential, long-term impact of our work. I hope it will contribute to a more peaceful world without war in the future.

Shoukri: Our primary audience includes policymakers, international organizations and the broader public, who may have limited exposure to the realities of Gaza. We envision sharing the final VR experience in international forums, museums and educational institutions, where its impact can resonate deeply and spark meaningful conversations. By placing users directly “in” these environments, we aim to inspire a sense of urgency and empathy that can potentially influence policy and humanitarian efforts.

Shigenoi: This year, Nihon Hidankyo (Japan Confederation of A- and H-Bomb Sufferers Organizations) won the Nobel Peace Prize. In Japan, peace education related to the atomic bombings of Hiroshima and Nagasaki has been widely accepted, thanks to the tireless efforts of civil society, educators, the government and organizations like Nihon Hidankyo. Our project is still in its early stages, but we hope it will become a resource for peace education that is used worldwide. We also hope that this VR content and archive can act as a catalyst to spark people’s interest. I believe it is crucial to raise awareness of the issue and how civil society has the power to influence politics. Our aim is to create a trigger for change.

Komatsu: I actually wasn’t interested in the news about Gaza before this project. But, by experiencing the footage through this Crisis VR system that I created, I was able to understand how terrible the situation is. In VR, I can feel the scope of the environment much more than on a flat screen, such as the size of the buildings, the narrow rooms and the extent of the damage. It feels more real and much closer to me, almost like I could touch it. The day after trying it out, I started watching the news about Gaza, as I could clearly imagine the situation. I hope that, like for me, this VR experience will give people a deeper understanding of the feeling of war.

Photo of a group of people trying crisis VR.
Photo of Crisis VR Ukraine project demonstration at TBS Tech Design X innovation space in Tokyo. In this demonstration, multiple users accessed the same metaverse space to experience a virtual re-creation of a real damaged building in Ukraine. © Watanave Lab, 2024

── What are your next steps for this project?

Shoukri: Following this project, we would like to expand the scope of immersive journalism to cover other underreported conflicts and crises, providing a platform for affected communities to have their voices heard globally. Additionally, we are interested in analyzing and publishing data, such as satellite imagery assessments, to highlight the lasting impacts of conflict on infrastructure and communities. This project could also serve as a foundation for further collaborations in educational outreach and policy advocacy, ensuring these narratives reach the stakeholders who can make a difference.

Watanave: An idea our lab is working on is to create a sort of a digital “time machine” by using augmented reality (AR). When you are in front of a building, this would allow you to use your smartphone to see images of how it used to look in the past. This concept could be applied to Gaza, where many buildings have been destroyed. I want to keep people’s memories of the past alive using this technology and keep them in the physical world, where they cannot be erased.

The use of emerging technology is a very important concept for our lab, as is connecting with people and supporting them in realizing their ideas. We want to conduct research which has a clear outcome for society. To do this, we will continue to try to engage different generations with our projects, through newspapers, TV and social media.

Komatsu: In developing the VR system, I will incorporate my expertise in VR and metaverse technology with my work on medical applications for VR. Drawing from my experience in therapeutic VR, I prioritize creating safe experiences that avoid traumatizing viewers, while still effectively conveying the realities of war. Our goal is to make these experiences accessible to people of all generations. Our immersive storytelling is designed to provide psychologically safe VR experiences that can deepen people's understanding of these critical situations.

Shigenoi: As well as being researchers, Mr. Komatsu and I are entrepreneurs, so we consider how we can apply our business experience to our research. The ultimate goal of this project is to provide comprehensive 3D archives and immersive VR content to the education sector. That’s why we have had discussions with educational institutions, museums and libraries in Qatar, as well as Japanese educational institutions and media outlets, from the very beginning. We are very focused on implementing this work beyond our research and on supporting education for peace.

Group photo of Watanave lab members.
Left to right: Researchers Shohei Komatsu and Koki Shigenoi, Professor Hidenori Watanave and research assistant Yuka Sashihara. © The University of Tokyo 2024
 

Hidenori Watanave
Professor, Interfaculty Initiative in Information Studies (III)/Director of Media Contents Research Initiative at III

Graduated with a Bachelor of Engineering degree from the Department of Architecture, Faculty of Science and Technology, at Tokyo University of Science and received a doctorate in engineering from the Graduate School of Systems and Information Engineering at the University of Tsukuba in Japan. Assumed current position in 2018. Creator of Hiroshima Archive, Nagasaki Archive, Digital Archives Series on the Great East Japan Earthquake, The Battle of Okinawa, Ukraine War, among numerous other digital archives. Author of Deta o Tsumui de Shakai ni Tsunagu (“Spinning Data to Connect to Society”) (Kodansha Gendai Shinsho, 2013), and co-author with Anju Niwata (a UTokyo alumna) of AI to Color-ka Shita Shashin de Yomigaeru Senzen, Senso (“Rebooting Memories of Prewar & War Through Colorized Photographs in Collaboration with AI and Human”) (Kobunsha, 2020), among other titles.

Koki Shigenoi
Visiting Researcher, Interfaculty Initiative in Information Studies (III)

Graduated with a bachelor’s degree in economics and a master’s in global studies from Doshisha University in Japan. As visiting researcher at UTokyo III focusing on conflict studies, particularly within the Middle Eastern context, leads collaborative research initiatives with Al Jazeera and partners with both national and international institutions. Currently completing doctoral dissertation, examining rebel governance, legitimacy and third-party interventions. Prior to joining UTokyo, held several research positions, including visiting researcher roles at Al Jazeera Centre for Studies and Hamad Bin Khalifa University in Qatar, and the Konrad Adenauer Foundation in Germany. Diverse background spans entrepreneurship, academia and consultancy.

Shohei Komatsu
Project Researcher, Interfaculty Initiative in Information Studies (III)

Graduated with a Master of Engineering degree from Tokyo University of Science. In addition to project researcher, currently also a doctoral candidate at the Graduate School of Interdisciplinary Information Studies (GSII), leading the development of the Crisis VR system at UTokyo III-GSII. Research focuses on creating psychologically safe virtual experiences that effectively convey social issues to diverse audiences. Besides academic work, works on therapeutic VR applications. As chief product officer at medical technology company BiPSEE Inc. and project lecturer at Kochi University School of Medicine, leads the development of Japan's first VR digital therapy, XRDTx, for depression treatment, working toward pharmaceutical approval. Additionally, contributes to the UTokyo Brand Studio, serves as director of design conference organizer Designship Inc., and also engages with government-led initiatives, such as the Ministry of Economy, Trade and Industry’s White Paper on Design 2024 project.

Portrait photo of Dr Arafat Madi Shoukri, wearing a suit.

Arafat Madi Shoukri
Senior Researcher, Al Jazeera Centre for Studies

Received his master’s and doctoral degrees in Oriental studies from the School of Oriental and African Studies, University of London, in 2008. In 2014, joined Al Jazeera Media Network to head the corporate relations and communications department. Currently a senior researcher at Al Jazeera Centre for Studies. An expert in migration in the Middle East and author of two books about this topic: Refugee Status in Islam: Concepts of Protection in Islamic Tradition and International Law (I.B. Tauris, 2010) and Refuge in the Islamic, International and Arab Legislations (Al Jazeera Centre for Studies, 2018).

Interview date: October 17, 2024
Interview: Nicola Burghall

Related links

Access Map
Close
Kashiwa Campus
Close
Hongo Campus
Close
Komaba Campus
Close