Artificial intelligence is increasingly pervasive and capable but is also very hungry for power. Researchers, including those at the University of Tokyo, have developed a new approach to building physics-based AI, systems that use the natural behavior of physical systems to process information, that dramatically reduces the amount of computation required for them to run. The approach simplifies both how these systems learn and how they are structured. It was demonstrated using an optical computing system that processes information with light and could help pave the way toward faster and more energy-efficient AI hardware.
It’s hard to ignore the presence of AI. News stories and software constantly make its presence felt, even if you don’t interact with AI systems directly. Behind the scenes, AI systems are becoming larger, more complicated and more resource-intensive. One of the biggest concerns around the growth of AI is the energy required to run these systems. Training AI models in particular can consume vast amounts of energy, driving researchers to search for ways to improve efficiency rather than simply increasing the size and complexity of AI systems. One possible approach is to use specialized physical systems to perform some of the calculations.
Associate Professor Kohei Nakajima from the Laboratory for Intelligent Systems and Informatics at the University of Tokyo and his team including NTT researchers, combined both in their latest set of experiments, and the results are promising. “Today’s AI systems have achieved remarkable capabilities, but they also require enormous computational resources. Instead of continuing to build larger and more complicated models, we wanted to rethink the learning process itself so that physical systems can perform computation much more naturally,” said graduate student Yongbo Zhang. “For this, we turned towards an emerging form of computation that uses light to process information instead of electrical signals. But to operate with this, we also had to change the way AI learning works so it could be implemented on vastly different hardware to traditional learning algorithms.”
Ordinary computer chips perform calculations using electrical signals that run through large numbers of transistors. This consumes energy and produces heat. Light, however, can carry information extremely quickly and can carry multiple signals simultaneously. This combination of speed, parallel processing and potential energy efficiency makes optical computing a promising platform for AI applications. Conventional AI methods can be implemented on optical and other physical systems, but doing so can be technically challenging and may require considerable measurement, control and data storage. The researchers therefore explored whether the learning process could be redesigned to better suit the characteristics of physical systems.
“We developed a new overall approach combining a training method called SS-DFA with a simpler network architecture called the single-map framework,” said Nakajima. “Conventional training requires information about the system to be measured and recorded as it evolves, before the learning process can then backtrack to determine what needs to change to achieve a desired outcome. Our method avoids the need to access most of these intermediate states, using the final error together with information about the system to guide learning. This reduces the amount of measurement, data storage and computation required. We also designed a simpler architecture in which many units follow the general rules, rather than relying on a set of highly complex internal connections. We demonstrated the combined approach through computer simulations and experiments using an optical computing system.”
Surprisingly, the simplified design matched, and in some tests outperformed, more complex neural network architectures, while requiring far less computation. In previous research, the team showed complex patterns of activity naturally emerged inside deep neural networks, including large language models, and suggested these patterns could be used to improve information processing. The new study demonstrated that these principles can be reproduced in an optical-based system. Future work will explore whether the approach can be scaled to larger AI models, applied to more complex tasks involving multiple types of data, and implemented on a wider range of computing platforms. If successful, the framework could reduce the computational demands of AI while taking advantage of optical systems that generate little heat. This could lower the energy needed for both computing and cooling in future AI data centers, while also reducing water consumption due to the lower heat output.
“At present, we have integrated the proposed architecture into several relatively baseline modern neural network architectures, and the results show that it can improve performance while reducing computation,” said Nakajima. “However, we have not yet tested it on today’s largest AI models, particularly language and so-called multimodal models, and this is an important direction for future research. Because our framework is modular and can be inserted as a computational block into existing networks, it provides a possible route towards such scales. And, if our system can be miniaturized and integrated onto a chip, it could process data directly inside phones, cameras or self-driving cars, enabling faster responses with lower energy consumption. Before this becomes practical, however, further work is needed on miniaturization, stability and large-scale manufacturing.”
