Point a traditional 3D sensor at a chrome-plated ball bearing and observe its failure. The laser light intended to bounce back to the detector and reveal the object’s geometry instead reflects at angles determined by the surface’s curvature, most of it traveling in unhelpful directions. The sensor, effectively, perceives almost nothing. The same issue arises with surgical tools coated in saline, the glass rear window of a self-driving car, and the polished aluminum fixtures on a factory inspection line. Specular surfaces (anything reflective, glassy, glossy) have been a persistent challenge for machine vision for many years.
A team at the University of Arizona believes they’ve discovered a workaround, and the solution is strikingly simple. Instead of creating a superior sensor or redesigning the projection patterns that standard 3D imagers depend on, they’ve managed to utilize the walls and furniture surrounding a reflective object as an informal screen. The environment becomes the apparatus. “Essentially everything around you becomes a virtual screen,” explains Florian Willomitzer, who heads the Computational 3D Imaging and Measurement Lab at the university’s Wyant College of Optical Sciences.
The research, released this week in Nature Communications, integrates two measurement techniques that have traditionally been viewed as distinct fields. One technique, known as structured-light triangulation, is the backbone of 3D sensing: project a known pattern onto an object, observe how it deforms from the camera’s perspective, and then apply geometry. It excels on matte surfaces but completely fails on mirrors. The other technique, called deflectometry, measures how a reflective surface alters the image of a large patterned screen placed nearby. Effective on chrome, ineffective on fabric.
Historically, the issue has been that real-world scenes consist of multiple materials. “Think about the interior of a car or a living room,” says Willomitzer. “Such environments feature specular materials, like mirrors, glass, or polished metal finishes, alongside diffuse surfaces, such as walls, fabrics, and furniture.” A surgical site similarly combines elements: glistening wet tissue and dull skin exist within the same field of view. Most 3D sensors, tailored to one type, simply abandon the other.
The Walls Have Eyes
The standard solution for deflectometry’s screen problem has been to create larger screens. For car-body inspection, manufacturers have constructed tunnel-length setups, entire rooms outfitted with calibrated displays, spacious enough for an entire vehicle. Costly, unmovable, restricted to a single purpose. The Arizona method bypasses these issues by recognizing that diffuse surfaces (a white wall, a fabric couch, a matte-painted ceiling) function just like a calibrated screen if their spatial locations can be determined.
Here’s the clever part. A scanning laser traverses the room two times in perpendicular directions. An event camera (more on that shortly) captures wherever the laser makes contact. The team’s algorithms process those signals and apply epipolar geometry (generally, the geometric rule that light from a known source must appear along a predictable line in the camera image) to distinguish direct reflections from indirect ones. Direct signals originate from matte surfaces; indirect signals, which bounce once off a matte wall before impacting a shiny object and reaching the camera, contain information about that shiny surface’s shape. The matte surfaces, whose positions are now determined from the triangulation step, serve the function of the deflectometry screen. No calibration required. No fixed hardware necessary. The screen reconstructs itself based on whatever is present in the room.
“We can utilize a laser scanner to capture everything in the room, including anything inside,” says Aniket Dashpute, the paper’s primary author, now a doctoral student at Rice University. “We subsequently apply our algorithms to segregate the diffuse from the specular surfaces and can ultimately use all measured diffuse scene components as a virtual screen for the deflectometry measurement of the specular components.” A piece of uncalibrated cardboard, as it turns out, serves perfectly well as the screen, and since the system triangulates its position anew for each frame, it can even be moved around a stationary object to enhance coverage without needing to recalibrate anything.
A Camera That Doesn’t Take Pictures
The event camera is perhaps the more unusual element of the duo. Unlike a traditional camera, which captures complete image frames at a constant rate, an event camera only records the moments when individual pixels change in brightness. Each pixel activates asynchronously, generating a timestamped signal the moment it detects a change. The outcome is a stream of sparse events rather than a series of dense frames, reminiscent of how retinal ganglion cells in the eye react to changes rather than to static lighting.
The practical benefit is speed and dynamic range. “The event camera can manage vastly different light levels, from very dim to extremely bright,” states Jiazhang Wang, a postdoctoral researcher on the team.