The new camera mimics the involuntary movements of the human eye, creating clearer, more accurate images for robots, smartphones and other image-capture devices.
Computer scientists have invented a camera mechanism that improves the way robots see and react to the world around them. Inspired by how the human eye works, a research team led by the University of Maryland developed an innovative camera system that mimics the small involuntary movements the eye makes to maintain a clear and stable view over time. The team’s prototype and testing of the camera, called the Artificial Microsaccade-Enhanced Event Camera (AMI-EV), is detailed in a paper recently published in the journal Science Robotics.
Advances in event camera technology
“Event cameras are a relatively new technology that are better at tracking moving objects than traditional cameras, but today’s event cameras struggle to capture clear, unblurred images when there’s a lot of movement,” said Botao He, a computer science doctoral student at UMD and lead author on the paper. “This is a big problem because many other technologies, such as robots and self-driving cars, rely on accurate, timely imagery to react correctly to changing environments. So we asked ourselves: how do humans and animals focus their gaze on moving objects?”
Mimicking human eye movements
For He’s team, the answer was microsaccades – small, rapid, involuntary eye movements that occur when we try to focus our gaze. These minute, yet continuous movements enable the human eye to maintain precise focus on objects and their visual textures (such as color, depth, and shadow) for extended periods of time.
“Just as our eyes need small movements to maintain focus, we thought a camera could use a similar principle to capture clear, accurate images without motion blur,” he said.
Technology implementation and testing
The research team successfully replicated microsaccades by inserting a rotating prism inside the AMI-EV to change the direction of the light beam captured by the lens. The continuous rotational movement of the prism simulates the movement that occurs naturally in the human eye, allowing the camera to stabilize the texture of recorded objects in the same way as a human. The research team then developed software to compensate for the movement of the prism inside the AMI-EV and synthesize a stable image from the changing light.
Study co-author Yannis Aloimonos, a professor of computer science at UMD, sees the team’s invention as a major step forward in the field of robot vision.
“Our eyes take pictures of the world around us, then those pictures are sent to the brain, which analyzes the images. It’s through this process that perception happens and we understand the world,” explains Aloimonos, who also directs the Computer Vision Laboratory at the University of Maryland Institute for Advanced Computing (UMIACS). “When you’re dealing with robots, replace the eyes with cameras and the brain with computers. The better the cameras, the better the robots will be able to perceive and react.”
Potential impacts on various industries
The researchers also think their innovation could have a big impact beyond robotics and national defense: Scientists working in industries that rely on accurate image capture and shape detection are constantly looking for ways to improve cameras, and the AMI-EV could be a key solution to many of the problems they face.
“Due to their unique capabilities, the event sensor and AMI-EV are set to play a central role in the field of smart wearables,” said research scientist Cornelia Vermüller, lead author of the paper. “They offer clear advantages over conventional cameras, such as excellent performance in extreme lighting conditions, low latency and low power consumption. These features are ideal for virtual reality applications, for example, which require a seamless experience and fast calculation of head and body movements.”
Enhanced real-time image processing
In early tests, AMI-EV was able to accurately capture and display motion in a variety of situations, including detecting a human pulse and identifying rapidly moving shapes. Researchers also found that AMI-EV could capture motion at tens of thousands of frames per second, outperforming most commercially available cameras, which capture an average of 30 to 1000 frames per second. This smoother, more realistic depiction of motion could be crucial for everything from more immersive augmented reality experiences and improved security surveillance to improving how astronomers capture images in space.
Conclusion and future prospects
“Our new camera system can solve a number of specific problems, such as helping self-driving cars determine if what’s on the road is human and what isn’t,” Aloimonos said, “which in turn enables many applications that many members of the public already use, such as self-driving systems and smartphone cameras. We believe our new camera system paves the way for the emergence of even more advanced and capable systems.”
Reference: “Microsaccade-inspired Event Camera for robotics” by Botao He, Ze Wang, Yuan Zhou, Jingxi Chen, Chahat Deep Singh, Haojia Li, Yuman Gao, Shaojie Shen, Kaiwei Wang, Yanjun Cao, Chao Xu, Yiannis Aloimonos, Fei Gao and Cornelia Fermüller, 29 May 2024, Science Robotics.
DOI: 10.1126/scirobotics.adj8124
In addition to He, Aloimonos and Fermüller, other co-authors from UMD include Jingxi Chen (B.S. ’20, Computer Science; MS. ’22, Computer Science) and Chahat Deep Singh (ME ’18, Robotics; Ph.D. ’23, Computer Science).
This research was supported by the U.S. National Science Foundation (grant number 2020624) and the National Natural Science Foundation of China (grant numbers 62322314 and 62088101). This article does not necessarily reflect the views of these organizations.