三维视觉感知 | 环境化身虚拟屏幕：3D 传感器混合反光成像识别

注本文为 “三维视觉感知” 相关合辑。英文引文机翻未校。中文引文略作重排。如有内容异常请看原文。Turning surroundings into a ‘virtual screen’ could help machines see better in 3D将周遭环境转化为“虚拟屏幕”助力设备实现更优质的三维视觉感知by University of Arizona亚利桑那大学 供稿edited by Stephanie Baum, reviewed by Robert Egan斯蒂芬妮·鲍姆 编辑罗伯特·伊根 审核Editors’ notes编者按The GIST内容概要A new 3D imaging approach transforms entire surroundings into a “virtual screen,” enabling accurate measurement of both specular and diffuse surfaces without large physical screens.一种全新三维成像方式可将全域环境转化为“虚拟屏幕”无需大型实体屏幕即可精准检测镜面反射与漫反射两类表面。By combining laser scanning, algorithmic separation of surface types, and neuromorphic event cameras, the method achieves high-resolution, high-speed 3D capture of complex, mixed-reflectance scenes, and is scalable for diverse applications.该技术融合激光扫描、表面类型算法划分与神经形态事件相机能够对复杂混合反射场景完成高分辨率、高速三维采集且可适配各类应用场景拓展使用。Schematic depiction of a 3D scan of a mixed reflectance scene with the novel sensor technology. The laser lines scan the scene. After computational separation of matte and specular scene parts, the 3D shape of the matte parts is evaluated directly, and the specular parts are evaluated via the reflection signal from the matte parts, effectively turning them into a large virtual screen for the specular measurement. Credit: Aniket Dashpute et al.该示意图展示采用新型传感技术对混合反射率场景开展三维扫描的过程。激光束对场景进行扫描通过运算划分漫反射区域与镜面反射区域后直接测算漫反射区域的三维轮廓依托漫反射区域传回的反射信号测算镜面反射区域形态以此将周遭环境转化为大型虚拟屏幕完成镜面反射区域检测。图源阿尼凯特·达什普特 等人Imagine navigating a city street during rush hour—cars and bikes zipping by, pedestrians hustling down a crowded sidewalk, your eyes adjusting to the shop windows’ glare in one moment and a dark underpass the next. Our brain, of course, does all this without us being aware of the complex processes going on in that moment. In real time, our eyes and brain create a three-dimensional, accurate representation of a dynamic scene, constantly calculating distances between objects with myriad shapes, sizes, and surfaces.试想早晚高峰穿行城市街道的场景车辆与非机动车疾驰而过行人拥挤地走在人行道上视线转瞬要适应商铺橱窗的强光随即又要适配昏暗的地下通道。大脑会自主完成一系列复杂运算人体无法感知具体运作流程。人眼与大脑可实时构建动态场景的三维精准影像持续测算不同外形、尺寸、表面材质物体间的间距。Chasing superhuman 3D vision for machines研发超越人类水准的设备三维视觉What humans do subconsciously and effortlessly is a tough nut to crack for machines with 3D-sensing systems—think self-driving cars, for example—tasked with measuring real-world scenes filled with objects that reflect light differently. In a paper published inNature Communications, a research group in the Computational 3D Imaging and Measurement Lab at the University of Arizona now reports clearing a hurdle toward endowing machines with “superhuman 3D vision,” according to the lab’s director, Florian Willomitzer, an associate professor at the U of A Wyant College of Optical Sciences.人类下意识即可轻松完成的视觉判断搭载三维传感系统的设备却难以实现自动驾驶车辆便是典型案例这类设备需要检测光影反射特性各不相同的现实场景。亚利桑那大学怀恩特光学科学学院副教授弗洛里安·维洛米策所在的计算三维成像与测量实验室团队于《自然·通讯》刊发论文团队攻克技术难题距离打造具备超强三维视觉能力的设备更近一步。Rather than simply mimicking the capabilities of human 3D vision, however, his team is developing ways to significantly improve 3D sensors capable of capturing images at higher resolution and faster speed, making the image-capture process impervious to challenging conditions such as highly reflective surfaces.该团队并未单纯复刻人类三维视觉机能而是着手优化三维传感器件提升成像分辨率与拍摄速率让设备在高反光表面等复杂环境下也可稳定完成图像采集工作。“Humans already have a built-in 3D camera system—the stereo vision of our two eyes,” Willomitzer said. “One of our goals is to enable computers and machines to see in 3D better than any human, which is crucial for a multitude of technological challenges, such as reliable navigation of self-driving cars, accurate guidance during robotic surgery, or improved sensing capabilities in industrial inspection and biomedical imaging.”维洛米策表示人体自带一套三维视觉系统也就是双眼立体视觉。团队研究方向包含打造三维视觉能力优于人类的智能设备该项技术可应用于诸多领域保障自动驾驶车辆稳定行驶、辅助机器人手术精准操作同时提升工业检测与生物医学影像的探测水准。Laboratory experiment demonstrating the proof of principle of the new technology. A mixed reflectance scene is scanned with a laser scanner (left). After computational separation of matte and specular scene parts, the 3D shape of the matte parts is evaluated directly, and the specular parts are evaluated via the reflection signal from the matte parts, effectively turning them into a large virtual screen for the specular measurement (right). Credit: Aniket Dashpute et al.实验室实验验证该新技术的原理可行性。左侧设备为激光扫描仪用于扫描混合反射率场景经运算区分漫反射与镜面反射区域后直接解析漫反射区域三维形态借助漫反射区域反射信号测算镜面反射区域轮廓将环境转化为虚拟大屏完成检测效果如右侧画面所示。图源阿尼凯特·达什普特 等人Why mixed-reflective scenes are so hard混合反射场景检测存在技术难点On the way to accomplishing those objectives, 3D imaging has to overcome a stubborn problem: Most state-of-the-art 3D sensors are optimized for imaging either “diffuse” (matte) or “specular” (reflective) surfaces. In contradiction, real-world scenes feature a wide range of surface reflectivities that exist somewhere in between those two extremes. This is where most 3D imagers fail.推进相关技术落地过程中三维成像技术面临棘手难题。目前主流三维传感器仅适配漫反射哑光表面或镜面反光表面单一成像场景但现实场景内物体表面反光程度介于两类特性之间多数三维成像设备无法适配这类场景。“Think of the interior of a car or a living room,” Willomitzer said. “Those environments include specular materials, such as mirrors, glass, or polished metal finishes, alongside diffuse surfaces, such as walls, fabric, and furniture.”维洛米策举例说明汽车内部、居家客厅等空间中既存在镜面、玻璃、抛光金属这类反光材质也包含墙面、布艺、家具等哑光表面。The same is true for robotic surgery applications, as a surgical site typically involves glistening fluids and moist tissues as well as diffuse surfaces such as skin. 3D sensing techniques that can measure all those surfaces equally well are extremely difficult to develop.机器人手术场景也存在同类情况手术区域内既有反光体液、湿润组织也有皮肤这类哑光表层研发可同步精准检测各类表面的三维传感技术存在较大难度。The prototype in the lab. The sensor, consisting of a laser scanner and an event camera (on the right), scans the mixed reflectance scene (on the left). After computational separation of matte and specular scene parts, the 3D shape of the matte parts is evaluated directly, and the specular parts are evaluated via the reflection signal from the matte parts, effectively turning them into a large virtual screen for the specular measurement. Credit: Aniket Dashpute et al.图中为实验室样机设备。右侧设备由激光扫描仪与事件相机组成对左侧混合反射率场景开展扫描。通过运算划分不同反射属性区域后直接测算哑光区域三维结构依托哑光区域反射数据分析反光区域形态将周边环境化作虚拟屏幕完成检测工作。图源阿尼凯特·达什普特 等人The team’s idea is built on an extension of so-called deflectometry—a well-established technique that measures the shape of specular surfaces by observing how a pattern on a screen is deformed upon reflection over the reflective surface. To measure highly complex shapes with deflectometry, however, the screens have to be very large to cover a wide angular range of surface orientations, Willomitzer explained.团队研究思路基于偏折测量法拓展而来该成熟技术依靠观测屏幕图案经反光表面反射后的形变状态测算反光物体外形。维洛米策解释若运用该技术检测结构复杂的物体需要配备大尺寸屏幕以此覆盖物体多角度的表面方位。For applications such as inspecting freshly painted car bodies, this has even led to tunnel-like screen assemblies large enough to accommodate the entire car. Such solutions are expensive, not portable, and tend to be limited to specific tasks.新车漆面检测场景中相关设备需要搭建可容纳整车的隧道式大屏装置。这类设备造价高昂、不便移动仅能适配固定检测工况。Turning whole rooms into virtual screens将整体空间构建为虚拟屏幕The solution that Willomitzer’s team developed is as simple as it is effective: Instead of requiring large screens, the entire surroundings of the specular objects that should be measured are turned into a “virtual screen.”维洛米策团队研发出简便高效的解决方式无需搭建实体大屏把待测反光物体所处的整体环境转化为虚拟屏幕。“We can use a laser scanner to capture everything in the room, with whatever is inside, including objects with specular, glossy, and matte surfaces, as well as matte walls. We then use our algorithms to separate the diffuse from the specular surfaces and can eventually use all measured diffuse scene parts as a virtual screen for the deflectometry measurement of the specular parts,” said the study’s first author, Aniket Dashpute, who started the work with Willomitzer at their previous institution, Northwestern University, and who is now a doctoral student at Rice University.论文第一作者阿尼凯特·达什普特表示借助激光扫描仪采集空间内全部景物数据涵盖反光、亮面、哑光材质物件以及哑光墙体。通过算法区分不同反射属性的表面将测算得到的哑光区域作为虚拟屏幕利用偏折测量法检测反光物体轮廓。该研究最初在西北大学开展如今其为莱斯大学在读博士研究生。“This effectively allows us to repurpose everything inside that room into a giant display—essentially everything around you becomes a virtual screen,” Willomitzer added.维洛米策补充道空间内所有物体均可充当显示载体周边环境整体构成虚拟屏幕。Event cameras push 3D video further事件相机助力三维影像技术升级Rather than relying on a conventional camera, which captures the entire scene frame by frame, the researchers use a so-called neuromorphic event camera, which only captures the important parts of the measurement at very high time resolution. This allows them to capture 3D videos of mixed reflectance scenes with moving objects at high frame rates.常规相机逐帧拍摄完整画面研究团队并未采用该设备而是选用神经形态事件相机。该设备以超高时间分辨率抓取检测关键信息能够高帧率拍摄包含动态物体的混合反射场景三维影像。“The event camera can handle vastly different light levels—from very dim to extremely bright,” said the paper’s second author, Jiazhang Wang, a postdoctoral research associate at the Wyant College of Optical Sciences. “This allows us to measure all object surfaces in a scene with high accuracy, despite their huge variations in surface reflectivity.”论文第二作者、怀恩特光学科学学院博士后王家章称事件相机可适配极暗至极强跨度的光照环境即便场景内物体表面反光程度差异显著依旧可以精准完成整体检测。Currently, the approach has been demonstrated in a tabletop laboratory setting, but Willomitzer said the technology is scalable to whatever the application demands.目前该技术仅在桌面级实验场景完成验证维洛米策表示可依据各类使用场景调整技术应用规模。“Scalability is an important requirement for the wide spectrum of 3D imaging applications,” he said, “from measuring small, shiny blood vessels during surgery to digitizing entire rooms or buildings.”三维成像应用场景跨度较大技术可拓展性具备实用价值既能够检测手术中细小的反光血管也可完成整间房屋、整栋建筑的数字化建模工作。Publication details发表信息Accurate and fast event-based shape measurement of mixed reflectance scenes,Nature Communications(2026). DOI: 10.1038/s41467-026-72254-6基于事件的混合反射场景高精度快速外形检测《自然·通讯》2026年。数字对象标识符10.1038/s41467-026-72254-6把整个房间变成虚拟屏幕新型 3D 传感器同步识别镜面与墙面测量精度达亚毫米级刘雅坤 DeepTech深科技 2026 年 5 月 23 日 10:06 上海传统传感技术应用局限常规3 D 3\mathrm{D}3D传感器可完成墙体类物体的探测作业接触镜面、玻璃、金属类高反光材质时探测功能会出现失效情况。美国亚利桑那大学联合莱斯大学、西北大学组建科研团队研发出全新3 D 3\mathrm{D}3D传感器设备。设备依托事件相机与扫描激光组件将所处空间环境转化为虚拟屏幕单次场景三维重建耗时70 m s 70\ \mathrm{ms}70ms可同步完成漫反射、镜面反射物体形态采集测量数值达到亚毫米区间标准。设备运行无需预先判定物体材质无需配套外置标定设备依托现场环境即可完成多重反光场景的三维建模工作。技术可适配玻璃、金属、塑料、织物共存的复杂场景适用范围涵盖自动驾驶设备、手术机器人、工业质检、生物影像采集等方向后续还可拓展至血管尺寸测算、室内及建筑整体数字化复刻等场景。相关研究成果刊载于Nature Communications期刊。图丨相关论文来源Nature Communications光学三维测量现存难点人眼与脑部可自然完成动态场景三维影像构建持续测算场景内各类物件的尺寸、间距参数。该类成像逻辑难以复刻至光学三维测量技术体系中。镜面、玻璃、抛光金属等物体以镜面反射为主要光学表现形式墙体类物件呈现漫散射光学特征两类物体成像原理存在明显区别。市面主流3 D 3\mathrm{D}3D传感器仅针对单一反射类型优化调试无法同时适配两种成像场景。图丨新型传感器混合反射场景三维扫描示意图来源Nature Communications自然环境内物体表面反射参数处于两类极值区间之间不存在绝对单一的反射形态。医疗手术场景中探测区域包含液态介质、湿润机体组织、皮肤等不同表面形态物件三维数据采集作业具备相应难度。车辆漆面检测工作中传统检测设备需要搭建大型隧道式幕布设施。该类设备制作成本偏高移动调整难度较大可覆盖的作业场景存在约束条件。新型传感系统构建方式此次研发设计参照成熟偏折测量法运作逻辑该方法依托反射面图案形变状态判定镜面外形尺寸。常规测量模式需要搭载大面积屏幕才能覆盖物件多角度表面检测需求。科研团队调整设备搭建思路舍弃大型显示屏幕以场地内现有镜面物件为载体将周边整体环境转化为虚拟显示界面。系统摒弃常规全域帧拍摄相机选用神经形态事件相机。设备具备高时间分辨能力仅采集检测流程中的光学变化信息能够录制动态物件混合反射场景的三维影像资料。图丨新技术原理验证扫描演示来源Nature Communications系统搭配双轴激光扫描装置与事件相机开展全域扫描作业。激光照射漫反射表面后形成散射光线事件相机同步记录光线明暗变动借助三角测算方式生成漫反射物件三维轮廓。系统将已建模完成的漫反射界面划定为虚拟屏幕以此为基准测算镜面类物件参数。莱斯大学博士研究生、论文第一作者 Aniket Dashpute 表示激光扫描可收录空间内全部物件数据涵盖镜面、高光质感、哑光质感各类物体。配套算法可以拆分两类反射形态对应的影像信息拆分后的漫反射区域影像可作为测算镜面物件的虚拟参照界面。光线经由墙体反射后再次穿过镜面最终录入相机的二次反射光路会依托极线几何规则与单次反射光路完成数据划分。结合入射光线与反射光线的几何关联能够推算镜面表面法线参数通过迭代运算程序依据法线数据生成场景深度影像。图丨设备原型机实物展示来源Nature Communications场地内普通纸板、墙体立面、雕塑背光面等漫反射载体均可充当偏折测量的光线来源整套检测流程无需增设辅助硬件设备。设备实测性能数据多组实操测试体现设备具备良好环境适配性。以纯白色平板作为参照样本纯漫反射场景下单次扫描耗时4 m s 4\ \mathrm{ms}4ms运行帧率可达250 H z 250\ \mathrm{Hz}250Hz模型重建偏差数值为210 μ m 210\ \mathrm{\mu m}210μm。雕像、镜面气球、金属配件组合的复合型场景中单次扫描耗时70 m s 70\ \mathrm{ms}70ms设备运行帧率为14 f r a m e / s 14\ \mathrm{frame/s}14frame/s。动态画面里气球旋转轨迹、纸巾抽取动作均可完整还原呈现。定量检测结果显示漫反射球体测量偏差310 μ m 310\ \mathrm{\mu m}310μm镜面球体测量偏差450 μ m 450\ \mathrm{\mu m}450μm。同环境条件下常规事件相机检测方案深度偏差可达6 m m 6\ \mathrm{mm}6mm。英特尔 RealSense、微软 Kinect v2 等商用传感设备处理镜面材质时易出现数据空缺、参数判定偏差等问题。该项研究依托不同反射形态的数据互补特性以低反光物件作为高反光物件的参照载体。事件相机的高速运行属性与广域动态成像能力可补足激光扫描采样密度偏低产生的数据空缺形成全新的三维传感数据采集模式。参考资料1.Dashpute, A., Wang, J., Taylor, J. et al. Accurate and fast event-based shape measurement of mixed reflectance scenes. Nat Commun 17, 4407 (2026). https://doi.org/10.1038/s41467-026-72254-62.https://techxplore.com/news/2026-05-virtual-screen-machines-3d.html运营/排版何晨龙注封面/首图由 AI 辅助生成referenceTurning surroundings into a ‘virtual screen’ could help machines see better in 3Dhttps://techxplore.com/news/2026-05-virtual-screen-machines-3d.html把整个房间变成虚拟屏幕新型 3D 传感器同时看清镜子与墙面精度达亚毫米级https://mp.weixin.qq.com/s/wFDAgP1Bx0wFoh3tIjcNgg