Permanence in an impermanent world Cyberarchaeology brings the past to life and preserves the present for tomorrow

February 15, 2012

An emerging technology dubbed Mixed Reality, or MR, overlays virtual objects onto the real world to create a mixed reality environment. MR can provide users with an immersive experience that is richer and deeper than that provided by Augmented Reality, a related technology. A group of researchers at the University of Tokyo’s (Todai) Institute of Industrial Science is using MR to digitally resurrect Japan’s past architectural splendors, as well as to preserve cultural artifacts and monuments at home and abroad for the world to view. Chief researcher Professor Katsushi Ikeuchi, a pioneer in MR, is heading these endeavors, collectively called the e-Heritage Projects.

Professor Ikeuchi

Professor Ikeuchi watches on as the first group of visitors prepares to depart on the Asuka tour © Ikeuchi lab

“It is important to preserve today what is irreplaceable tomorrow,” says Ikeuchi. “These e-Heritage projects aim to capture and model in digital form various cultural monuments and artifacts for later viewing.”

Though these models can be displayed on 2D-screens, Ikeuchi says this is not very interesting and will soon be forgotten by audiences. Rather, viewers can have a much richer and more memorable experience when they encounter virtual models of monuments and buildings in three-dimensional Mixed Reality environments.

One such project is the recreation of a section of Japan’s 7th century imperial capital city Asuka, located in Nara Prefecture. Using line drawings of the time, the researchers have digitally reconstructed 3D models of buildings that once stood in a particularly part of the city?an area today made up of paddy fields, though they still retain traces of post-holes and stone platforms of the ancient buildings. Visitors wearing head-mounted displays and riding in an open vehicle see real-world views of the area produced by cameras mounted on the bus. Overlaid on these views are 3D computer-generated models of ancient Asuka structures situated on the sites where the buildings once stood. These models are adjusted to take into account the direction the wearer is facing, and they even display the shadows cast by the people viewing the scene.

“Asuka was the cradle of high technology in Japan,” says Ikeuchi. “The latest technology from abroad was introduced there and used for the construction of the Great Buddha (Daibutsu) located in Nara. It’s fitting, then, the area has been chosen for a high-tech research effort using Japanese technology that may soon be exported around the world for the preservation of cultural artifacts.”

Collecting Buddhas

Ikeuchi returned to the University of Tokyo in 1996 to continue his research after having worked abroad for many years. His research team was already modeling small items in the laboratory. “So we decided we should next try to model something outside the lab on a grand scale, and came up with the idea of the Great Buddha in Kamakura,” says Ikeuchi. The project was successful, and the team went on to model the giant Buddha statues and historical buildings in Nara, Asuka, which accounted for the major giant Buddha statues in Japan.

The team then turned its attention overseas and to Thailand in particular, a country well known for its many Buddhist temples and giant Buddha statues. During their work there, they heard about the ancient Bayon temple in Angkor Thom, near Angkor Wat, in neighboring Cambodia, which is renowned for its 173 carved faces of Buddha.

Except for one brief period, Angkor was the capital of the Khmer Empire, which flourished in that area from the 9th to the 14th centuries, and is considered one of the largest pre-industrial settlements in history. For the people of Cambodia, it is important both as the symbol of their historical roots and of the present Kingdom of Cambodia. Nevertheless, it is in imminent danger of collapse, and international efforts to preserve the Angkor ruins are being lead by the Japanese Government Team for Safeguarding Angkor (JSA).

The huge Bayon temple complex in the center of Angkor Thom covers an area of 160 meters by 140 meters, and stands 45 meters high at its tallest point. More than 50 towers at the site contain faces of the Kannon Buddha carved in relief, as many as four faces on a tower, smiling benevolently on visitors to the temple. The intricate architectural design consists of two corridors and three levels of terraces covered entirely throughout in carvings, creating an incredibly complex structure.

Size is important

For Ikeuchi, the size of the temple ruins, its complexity and beauty and the fact that its central tower is in danger of collapsing, made Bayon the perfect subject for an e-Heritage project.

But gathering the data for constructing a 3D model is hardly a matter of simply taking a series of photographs. In order for the project to work as a true virtual preservation, it has to capture both the form and appearance of the original. This requires sophisticated data acquisition techniques to render the exact shape and color of the object, using such tools as laser range-sensors.

Two types of commercial laser scanning devices are used for this data acquisition. The first type aims a laser beam at a surface and measures the time it takes to reach the target, from which the distance can be calculated and the shape of an object gradually plotted. This kind of device is effective over long distances but has a relatively low accuracy (about 1 cm resolution). The second type of scanner employs a laser together with a camera that is mounted in such a way that both instruments can be used to triangulate the target’s position within millimeter-level accuracy, though only over very short distances. Software is then employed to integrate the numerous scanned data points into the images used to generate a 3D model of the object.

Both types of scanning devices are normally mounted on the ground and need to be fixed in position for several minutes during the scanning process. When large objects are to be scanned, it is necessary to mount the scanner on a scaffold or tower. In the case of the Bayon temple and the other intricately carved buildings that make up the complex, however, such conventional data acquisition methods were impractical. So the team had to develop a variety of new technologies and techniques to surmount the challenges posed by the sheer size and complexity of the ruins.

Up in the air

The team came up with the idea of using an aerial laser scanner by attaching a normally grounded scanning device, along with a GPS system, to a large helium balloon. The resulting data gathered using this method proved highly successful and enabled the team to scan the entire outer surface of the complex. Nevertheless, several hurdles had to be crossed before this was achieved. For instance, the researchers had to find a way to compensate for the movement of the balloon, which caused data distortion during the scanning process. This was solved by referencing the scanner’s range-sensor data with that taken by a video camera attached to the balloon (the PhD subject of team member Atsuhiko Banno) and with a ground-based laser range sensor (the PhD subject of team member Tomohito Masuda).

Before: 3D image generated from original data gathered by the aerial scanner

Before: 3D image generated from original data gathered by the aerial scanner © Ikeuchi lab

After: 3D image from corrected data

After: 3D image from corrected data © Ikeuchi lab

The researchers also had to overcome the difficulty of data scanning in the narrow spaces inside the temple. This was done by engineering a robot that ascended a ladder with the scanner, enabling it to scan the walls (the PhD subject of team member Shintaro Ono). For particularly narrow spaces, a mirror was used to reflect the laser beam up onto the surface of the walls from a scanner on the ground. Consequently, parts of the temple that had never been viewed in modern times were seen for the first time, resulting in an important discovery. In one narrow corridor, the decorated pediments (a type of gable) at roof height were decorated with Buddhist images, the religion of the builders of the Bayon temple. But the 3D model later revealed that several of the pediments had been altered by erasing the central Buddha images and replacing them with Shiva lingam, the phallic generative symbol of the Hindu deity. This was most likely carried out during a revival of the Hindu religion in the region during mid-13th century.

Pediment with the original image of Buddha

Pediment with the original image of Buddha © Ikeuchi lab

Pediment with the altered image of the Shiva lingam

Pediment with the altered image of the Shiva lingam © Ikeuchi lab

Color and image alignment

Another challenge facing the team was accurately recording the colors of the temple’s surfaces. The time taken to scan such a large structure rendered conventional techniques unfeasible because of the variations in lighting that occur throughout the day and because of changing weather conditions. Such varying conditions would have created different sets of color data. So again the team developed new methods to record the surface chromaticity, the characteristics of the surface itself, and the illumination chromaticity, the actual color of the light hitting the surface. (This research became the PhD subject of Rei Kawakami, now a postdoctoral researcher at UC Berkeley.) This generated a chromatic dataset from which the colors of the surface could be computed in any lighting situation, enabling the production of true-color images of the entire complex for any time of day and in any weather conditions?essential criteria when preserving a cultural heritage site such as Bayon.

Original image

Original image © Ikeuchi lab

Image after adjusting for chromaticity and brightness

Image after adjusting for chromaticity and brightness © Ikeuchi lab

With the task of data acquisition accomplished, the next task was to align and integrate the individual images produced from the scanned data points into a unified dataset of the entire complex. But in aligning images, small errors can build up during the process, a problem that would be greatly magnified in processing the Bayon dataset, given the large number of images involved. The team overcame the problem by comparing all of the images at the same time. Because the resulting dataset was terabyte-sized, the researchers also had to find a new way of computing this vast amount of data. Researcher Takeshi Oishi developed a high-speed parallel processing method that was over a thousand times faster than previous methods employed, by using the graphics-processing units of a number of computers linked together in a computer cluster.

Upon generating a 3D model of the temple, an important discovery was made. It has been suggested that the 173 faces of the Buddha located throughout the temple can be divided into three groups, which appear to be Deva (gods), Devi (goddesses) and Asura (devils). The team’s 3D model showed that the 173 faces can indeed be grouped into three clusters, according to similar characteristics, and that the similar faces are located close together. This finding provides support for the theory that a particular master craftsman carved a single image, and other craftsmen working nearby copied that example.

“By creating an actual 3D model, rather than simply collecting 3D data, we have been able to provide evidence of historical events,” says Ikeuchi. “We have named this method cyber archeology.”

Sharing the results

e-Heritage doesn’t end here. The researchers are working on various ways to make the results available to different audiences. The Asuka project described at the beginning of this article is one novel example that Ikeuchi and Oishi are currently working together to perfect. While such a historical reconstruction could easily be shown on a screen in a darkened theatre, the two researchers stress the importance location plays in enhancing the viewing experience.

“If you don’t see the scene in its original location, you can’t say that you have truly ‘experienced’ it, says Ikeuchi. “If you tell a group of elementary school children visiting Asuka that ‘here was such-and-such temple’, they’ll just say, ‘Whatever.’ Then after enjoying their lunch on the grass nearby, they won’t remember a thing when they return home.” A much better way to impress visitors, he says, is to have them tread the same ground and breathe the same air where the monuments once stood, while simultaneously viewing the reconstructed models through MR.

Ikeuchi notes that it is the dream of all engineers to have their research applied in the real world, and he hopes that the MR technology being developed in his lab will be used to preserve irreplaceable cultural artifacts and enable people around the world to enjoy and experience their cultural heritage. To this end he is continuing his research into MR technology to make this dream a reality. Moreover, he, Oishi and former lab member Tetsuya Kakuta have set up a venture company, Asuka Lab, to make this technology available outside the university.

Ikeuchi notes that the process of commercialization creates a whole new set of problems, but adds that these in turn can produce new directions for further research. He also stresses that developing a close relationship with society is vital when it comes to applying research to real-life situations. “Creating a venture company is an important means of bridging the gap that exists between universities and society, and it can also make the achievements of research more broadly available,” says Ikeuchi. “Now a new way of experiencing the past and the present is just around the corner.”


Katsushi Ikeuchi
Professor Katsushi Ikeuchi
Institute of Industrial Science
Associate Professor Takeshi Oishi
Associate Professor Takeshi Oishi
Institute of Industrial Science

Title Image

3D scan of Buddha statues at Bayon © Ikeuchi Lab

3D scan of Buddha statues at Bayon © Ikeuchi Lab

Access Map
Kashiwa Campus
Hongo Campus
Komaba Campus