On behalf of the faculty and staff of the University of Tokyo, I would like to extend my heartfelt congratulations to all of you who have reached this commencement ceremony today. Congratulations on your graduation.
Because of the COVID-19 pandemic that has continued since 2020, you may not have had the freedom to spend enough time on campus, studying and doing research together with each other in laboratories and seminar rooms. I respect you all deeply for the efforts you made to overcome those difficult circumstances so that you could be here today. I would also like to extend my congratulations and gratitude to your families and to everyone else who has supported you along the way.
At the forefront of the changes in daily life brought about by the pandemic has been the rapid and widespread adoption of remote technology. Many of you probably started using tools like online meeting software for classes only after 2020. The pandemic forced us to avoid close contact between people in order to prevent infections, and, whether we liked it or not, the use of remote technology expanded at an unprecedented pace and scale.
Remote technology has steadily evolved ever since the invention of the telegraph as typified by Morse code in the 19th century. Today we use it throughout our daily lives in remote controls, mobile phones, and many other ways. Research has also been conducted on its application to robotics, including teleoperation and telepresence. Through various interfaces, such technology enables new means of perception—visual and auditory perception in the case of online meeting tools. It seems to eliminate our distance from things that are far away, and it enlarges things that are too small to see with the naked eye or that cannot be approached directly. I would like to think a bit now about the significance for people today of this technology, which allows us to manipulate distance and size freely.
All of you have probably seen some advantages to online university classes. Even if concerns about COVID-19 were to completely disappear, people will not want to lose the convenience of online tools. We have been able to take part in events that we could not have attended before because of distance or a lack of time. Students studying abroad can now still attend classes back home, faculty can deliver lectures from overseas, and international research exchanges through online conferences and the like have become easier. UTokyo is now inviting faculty at overseas universities to be Global Fellows and give lectures online for us.
Remote technology has certainly overcome some of the difficulties posed by physical distance, but does that mean that the problems of distance have been solved and we no longer need to gather in classrooms? Probably not, as shown by your strong and consistent desire to return to face-to-face classes. When in-person classes resumed after the loneliness of many months of online education, a quiet excitement and sense of liberation could be felt in every classroom. Both students and faculty seem to have felt strongly that sharing the same space and directly sensing each other’s reactions is an irreplaceable experience. Precisely because of that forced isolation, we rediscovered the value of face-to-face interactions.
So what is lacking in our current online tools?
Although they eliminate physical distance, we must not forget that two-dimensional images and audio cannot reproduce the complex and multifaceted sense of distance in the three-dimensional space in which we live. The two-dimensional screen of online tools flattens the distance between people and objects, putting them all at a uniform distance. However, real space is three-dimensional and has depth that cannot be reproduced only by two-dimensional perspective. We move our bodies in real space and interact with people and objects in the same timeframe. We use not only our vision and hearing but also our sense of touch and smell to perceive our environment and the state of the objects in it. As shown by how we whisper to someone nearby but speak loudly to someone else far away based on our perception of the space we are in, we confirm and experience the depth of that space through our physical bodies.
There is a paradox there: While remote technology might seem to shorten distances, in fact it flattens our perceptions and deprives us of real physical experiences. What is lacking in online conversations, in other words, is the depth of physical realism, the three-dimensionality created by interactions in real space.
Remote technology continues to advance quickly, however. Many attempts are now being made to enable three-dimensional spatial perception. Examples include the virtual spaces called metaverses and augmented reality that combines the virtual with the real. It is already possible to participate in conferences through personal avatars that appear on the screen, and participants can enjoy greetings and conversations as if they were in the same location. The use of virtual spaces for education and research has been particularly active in the field of medicine; in fact, the Japanese Society for Medical Virtual Reality was founded more than two decades ago. In educational settings, there are attempts to enable students with disabilities, those recovering from illness, and those who have stopped going to school to “attend” school by operating robot avatars from home, creating experiences closer to real physical sensations than would be possible with two-dimensional online tools.
As you know, last September the University of Tokyo established the Metaverse School of Engineering. Its kickoff ceremony held in a metaverse Yasuda Auditorium was attended by many avatars. The starting point was students’ attempt to visualize a classroom in the Faculty of Engineering Building 1, where the Virtual Reality Educational Research Center, also known as the VR Center, is located. Dozens of people, including myself, have been scanned in 3D using a system with 80 cameras to create avatars. In January 2022, I was able to give a lecture at a symposium of the National Institute of Informatics through my avatar located in an underwater metaverse. Students’ avatars also participated, and we were able to have a dialogue together there.
Virtual spaces are also being developed to assist with vocational training and hiring, incorporating deep insights from many different fields, including information science, medicine, psychology, sociology, the arts, and linguistics. Activities now underway at UTokyo include a project by the University of Tokyo Hospital to study the effects of virtual reality therapy for people who have lost limbs, a project by the Graduate School of Humanities and Sociology utilizing 3D scanning data and high-resolution images from museum and gallery exhibitions for analysis, and a project by the Graduate School of Mathematical Sciences on the visualization of mathematical concepts. During math lectures it can be difficult to visualize a multidimensional shape from a diagram on the blackboard; if a three-dimensional figure appears in front of the student and they can touch or enter it in a metaverse space, the mathematical concepts might become easier to understand and the methods for grasping them may be transformed.
These new ideas go beyond just passively relying on remote technology when you have no other options. Rather, the goal is to take advantage of the metaverse to achieve what has never been possible before.
Research on virtual reality (VR) and augmented reality (AR) is, in a way, an exploration of the nature of reality itself. It challenges us to think about why we perceive something as real and what reality is in the first place.
The immersive and engaging nature of the metaverse likely offers a more profound sense of realism and experience than one-way lectures in a classroom. It may even lead to the discovery of a new kind of reality that would be difficult to experience in the physical world.
I myself have conducted deep-sea expeditions using remotely operated vehicles (ROVs) for my research. Japan used to have an ROV called KAIKO that was capable of diving to the world’s deepest depths of around 10,900 meters. The ROV I frequently used is called HYPER-DOLPHIN and is owned by the Japan Agency for Marine-Earth Science and Technology (JAMSTEC).
ROVs are attached to an umbilical cable that carries power and data, and they are operated remotely from a ship on the sea surface for exploring the deep ocean. In a dimly lit control room, operators look at a wall of displays showing images from the ROV as they operate it and make observations. They feel as if they are immersed in the deep-sea environment themselves. The experience may be similar to reading a novel, where our bodies and imaginations are swept into a world we have never experienced before as we immerse ourselves in the story.
This is why the metaverse can be helpful in supporting employment and social reintegration for those who have difficulty participating in society due to sensory sensitivity or social anxiety. Such people can regain their connection with society by, for example, using an avatar to order and sell products for a metaverse store and to interact with the store’s customers. Through such an interactive space filled with active bodily sensations and dialogue, a new reality can be born.
A unique aspect of human culture is language, and this new technology gives us an opportunity to rethink how we communicate with language. Communication is not merely a one-way transmission of information; rather, it emerges through interactions in which people respond to each other. Nor does communication end after just one response; rather, what truly makes up communication is the accumulation of repeated, overlapping, and interconnected responses. The language that enables this process can be thought of as a virtual reality space that has been created and inherited over countless generations.
From the 19th and into the 20th century, the theorization of language-based communication advanced along with the development of linguistics, semiotics, and information theory. The starting point was the transmission model. In his Course in General Linguistics, Ferdinand de Saussure presented a diagram of a “speaking-circuit” to explain a telephone conversation between two people from an external perspective. Claude Shannon, the founder of information theory, developed the mathematical basis for a similar model of communication, while structural linguist Roman Jakobson proposed a comprehensive model that focused on the factors and functions of communication while also incorporating the transmission model. His approach considered not only the conveyance of the content of a message but also the role of interjections like “Oh!” or “Ah!” and seemingly meaningless expressions like “Um...” when starting a conversation. All of these models were based on the idea of transmission, that is, a message that has been encoded by a speaker being received by the listener and decoded into its original meaning.
But as we know from our daily lives, communication is not a simple one-way transmission or an exchange of messages. During a conversation, we check for our partner’s reactions, use gestures and metaphors, and watch for further reactions as we try to achieve mutual understanding. Human dialogue can thus be thought of as a kind of translation. In fact, Roman Jakobson discussed not only translation within a language, such as added explanations and paraphrasing, but also inter-semiotic or multimodal translation, which involves translating between different symbolic systems, such as images to words.
Furthermore, in research fields related to artificial intelligence and robotics, there have been ongoing efforts for many years to create communication between multiple robots in order to achieve accurate functioning as environmental conditions change and to understand structurally the essence of communication in living organisms, including humans. Many studies have been conducted, including experiments to determine how different robots perceive the same objects, experiments on creating languages through dialogue between robots, and experiments to create communication signals shared among robots in swarm robotics systems, particularly regarding spatial cognition. The models of swarm robotics systems utilize findings about social organisms like ants and bees, with translations made from individual actions to control the collective behavior of the group.
These expanding horizons overlap with the virtual reality I discussed earlier. Translation can be seen as a process of discovering a common ground or a responding undercurrent that connects seemingly separate entities, enabling us to perceive the interconnections among them.
So far, I have discussed the challenges of distance raised by remote technology, the realization of spatial perception and reevaluation of reality through the three-dimensional virtual space of metaverses, and the importance of language grounded in physical entities like our bodies or robots. The metaverse and VR technologies that are now being adopted more and more widely not only reproduce our real-life bodily sensations but also enable us to experience things we haven’t experienced with our bodies before, such as the environments experienced by animals and the bodies of others, all with a sense of reality.
Let’s take a moment to consider what this all means. Humans have the ability to think from the perspective of others without being confined to the narrow framework of our own real-life experiences. The ability to share questions with others and address unknown challenges through dialogue is essential for creating a better future society. To utilize the still immature metaverse and to move it in a positive direction, we need to come up with fresh ideas by engaging in inter-disciplinary dialogue.
During your time here, you have all been exposed to the cutting edge of academic fields. You may have spent considerable time translating that difficult content so that others could share it. You may have also been surprised to find that much cutting-edge research is connected to completely different fields and transcends the boundaries of the humanities and sciences. As you take your first steps into a new world, I sincerely hope that you will soar high and thrive both in Japan and globally and that you will make the most of what you have experienced at the University of Tokyo.
Congratulations once again on your graduation.
FUJII Teruo
President
The University of Tokyo
March 24, 2023
