Better deep learning: Fewer neurons, more intelligence

A research team from TU Wien, IST Austria, and MIT (USA) has developed a new type of artificial intelligence based on biological models, such as simple nematodes. The new AI model can control a vehicle with a surprisingly small number of artificial neurons. The system has crucial advantages over previous deep learning models: it handles imperfect input data much better, and its simplicity allows for detailed explanation of its functionality. It doesn't have to be viewed as a complex "black box" but can be understood by humans. This deep learning model has now been published in the journal "Nature Machine Intelligence."

Learning from Nature

Similar to living brains, neural networks in computers consist of many individual cells. When a cell is active, it sends a signal to other cells. All the signals that the next cell receives collectively determine whether that cell also becomes active. Exactly how one cell influences the activity of the next is initially unknown—these parameters are adjusted in an automated learning process until the neural network can solve a specific task.

“For years, we’ve been thinking about what we can learn from nature to improve artificial neural networks,” says Prof. Radu Grosu, head of the Cyber-Physical Systems research group at TU Wien. “The nematode C. elegans, for example, manages with an astonishingly small number of nerve cells, yet it exhibits interesting behavioral patterns. This is due to the efficient and harmonious way its nervous system processes information.”

“Nature shows us that there is still much room for improvement in artificial intelligence,” says Prof. Daniela Rus, director of the Computer Science and Artificial Intelligence Laboratory (CSAIL) at MIT. “Therefore, our goal was to drastically reduce complexity and improve the interpretability of the neural network.”

“Inspired by nature, we have developed new mathematical models for neurons and synapses,” says Prof. Thomas Henzinger, president of IST Austria.

“The processing of signals within the individual cells follows different mathematical rules than previous deep learning models,” says Ramin Hasani, a postdoctoral researcher at the Institute for Computer Engineering at TU Wien and CSAIL, MIT. “Furthermore, not every cell is connected to every other cell – this also simplifies the network.”

The task: Autonomous lane keeping

To test the new ideas, the team chose a particularly important test task: lane keeping in autonomous driving. The neural network receives a camera image of the road as input and is supposed to automatically decide whether to steer right or left.

“For tasks like autonomous driving, deep learning models with millions of parameters are often used today,” says Mathias Lechner, a TU Wien alumnus and PhD student at IST Austria. “However, our new approach makes it possible to reduce the size of the network by two orders of magnitude. Our systems manage with 75,000 trainable parameters.”

Alexander Amini, a PhD student at CSAIL, MIT, explains that the new system consists of two parts: The camera input is first processed by a so-called convolutional network, which perceives the visual data only to recognize structural image properties in the pixels. The network decides which parts of the camera image are interesting and important and then passes signals on to the actually crucial part of the network—the control system, which then steers the vehicle.

Both subsystems are initially trained together. Many hours of traffic videos of human drivers in the Boston area were collected and fed into the network, along with information on how the car should be steered in the respective situations—until the system has learned the correct association between image and steering direction and can independently handle new situations.

The neural network's control system (called "neural circuit policy," or NCP), which translates the data from the visual network into a control command, consists of only 19 cells. Mathias Lechner explains: “These NCPs are three orders of magnitude smaller than would be possible with previous state-of-the-art models.”

Causality and Interpretability

The new deep learning model was tested in a real autonomous vehicle. “Our model allows us to examine precisely where the network focuses its attention while driving. It concentrates on very specific areas of the camera image: the roadside and the horizon. This behavior is highly desirable and unique among systems based on artificial intelligence,” says Ramin Hasani. “Furthermore, we saw that the role of each individual cell in every single decision can be identified. We can understand the function of the cells and explain their behavior. This level of interpretability is impossible in larger deep learning models.”

Robustness

“To test how robust our NCPs are compared to previous deep learning models, we artificially degraded the images and analyzed how well the system coped with image noise,” says Mathias Lechner. “While this proved to be an insurmountable problem for other deep learning networks, our system is very resilient to input artifacts. This property is a direct result of the novel model and its architecture.”

“Interpretability and robustness are the two crucial advantages of our new model,” says Ramin Hasani. “But there’s more: Our new methods allow us to reduce training time and make it possible to implement artificial intelligence in relatively simple systems. Our NCPs enable imitative learning across a wide range of applications, from automated work in warehouses to the motion control of robots. The new results open up important new perspectives for the AI ​​community: The fundamentals of data processing in biological nervous systems represent a valuable knowledge resource for generating high-performance, interpretable artificial intelligence—as an alternative to the black-box machine learning systems we have known until now.”

Publication

M. Lechner, R. Hasani, A. Amini, T. Henzinger, D. Rus, R. Grosu. 2020. Neural Circuit Policies Enabling Auditable Autonomy. Nature Machine Intelligence. https://www.nature.com/articles/s42256-020-00237-3