Jaw-dropping new tech builds 3D pictures based on reflection in person's eyes
- 10 months ago
Jaw-dropping new tech can build a 3D picture based on the reflection in a person's eyes.
University of Maryland researchers say the reflective nature of the human eye is an underappreciated source of information about what the world around us looks like.
In a paper currently under review, the team explain how by imaging the eyes of a moving person from different angles they reconstruct a 3D scene beyond the camera's line of sight.
Examples extracted from test eye portraits have included recognisable 3D images of toy aliens and a dog.
The researchers report the task was challenging due to the difficulty of accurately estimating eye poses and the entangled appearance of the eye iris and the scene reflections.
The study website explains: "Our method jointly refines the cornea poses, the radiance field depicting the scene, and the observer's eye iris texture."
Because the cornea geometry is approximately the same across all healthy adults, the pixel size of a person's cornea can be counted in the image and it can be computed exactly where their eyes are.
The study explanation continues: "Using this insight, we train the radiance field on the eye reflections by shooting rays from the camera, and reflecting them off the approximated eye geometry.
"To remove the iris from showing up in the reconstruction, we perform texture decomposition by simultaneously training a 2D texture map that learns the iris texture.
"However, approximating the eye pose just from the image is always very noisy. To address this issue, we perform eye pose optimisation which is critical for performance."
The researchers decided to test their work using the music videos from Miley Cyrus and Lady Gaga to attempt to reconstruct what they were observing.
The study website reports: "We manage to reconstruct the object that appears in Miley's eyes, and we observe an object resembling an upper body from Lady Gaga's eyes.
"However, due to the quality of these videos, the correctness of the reconstruction is unclear."
Applications are being considered, but it could be developed into a futuristic crime investigation tool similar to the famous Blade Runner film scene where a photograph is continually enhanced to offer detail of a scene.
The researchers cannot currently comment on the study because the work is still under review in an academic conference.
University of Maryland researchers say the reflective nature of the human eye is an underappreciated source of information about what the world around us looks like.
In a paper currently under review, the team explain how by imaging the eyes of a moving person from different angles they reconstruct a 3D scene beyond the camera's line of sight.
Examples extracted from test eye portraits have included recognisable 3D images of toy aliens and a dog.
The researchers report the task was challenging due to the difficulty of accurately estimating eye poses and the entangled appearance of the eye iris and the scene reflections.
The study website explains: "Our method jointly refines the cornea poses, the radiance field depicting the scene, and the observer's eye iris texture."
Because the cornea geometry is approximately the same across all healthy adults, the pixel size of a person's cornea can be counted in the image and it can be computed exactly where their eyes are.
The study explanation continues: "Using this insight, we train the radiance field on the eye reflections by shooting rays from the camera, and reflecting them off the approximated eye geometry.
"To remove the iris from showing up in the reconstruction, we perform texture decomposition by simultaneously training a 2D texture map that learns the iris texture.
"However, approximating the eye pose just from the image is always very noisy. To address this issue, we perform eye pose optimisation which is critical for performance."
The researchers decided to test their work using the music videos from Miley Cyrus and Lady Gaga to attempt to reconstruct what they were observing.
The study website reports: "We manage to reconstruct the object that appears in Miley's eyes, and we observe an object resembling an upper body from Lady Gaga's eyes.
"However, due to the quality of these videos, the correctness of the reconstruction is unclear."
Applications are being considered, but it could be developed into a futuristic crime investigation tool similar to the famous Blade Runner film scene where a photograph is continually enhanced to offer detail of a scene.
The researchers cannot currently comment on the study because the work is still under review in an academic conference.