Sign In

Communications of the ACM

ACM TechNews

Getting Audio from Still Images, Silent Videos

View as: Print Mobile App Share:
hen you take a photo on your phone, the vibrations of your voice can create tiny bends in the light that are enough to extract audio.

The idea for Side Eye was inspired by an episode of the sci-fi show “Fringe” that saw the main characters, a team of fringe science investigators working for the FBI, extracting audio from a melted pane of glass.

Credit: Matthew Modoono/Northeastern University

A machine learning tool developed at Northeastern University can obtain audio from still images and muted videos.

Using the Side Eye tool, which leverages image stabilization technology standard in most smartphone cameras, it is possible to determine the gender of someone speaking off camera and the exact words they said.

Northeastern's Kevin Fu explained that the small springs holding a camera lens suspended in liquid experience microscopic vibrations and the light is bent almost imperceptibly when someone speaks near a camera lens.

Taking advantage of the rolling shutter method of photography used by most smartphone cameras, the researchers can extract sonic frequencies from those vibrations.

Side Eye produces muffled audio, but the use of machine learning and training on certain words and audio enables it to extract a substantial amount of information, said Fu.

From Northeastern Global News
View Full Article


Abstracts Copyright © 2023 SmithBucklin, Washington, D.C., USA


No entries found