Researchers reconstructed the sound vibrations of objects on video

Sound is vibrations of a certain frequency, which are distributed in the surrounding space of the source. These waves are close to reach objects and cause them to feel the vibration. A team of researchers at MIT has managed on the basis of these vibrations visible in the video, to partially restore the original sound with some distortion.

In the Abe Davis, Michael Rubinstein, Neal Wadhwa, Gautam Mysore, Fredo Durand and William Freeman used the camera to record video at a frequency of several thousand frames per second, and are common, and are subject to vibrations items like foil packet of crisps, leaves, plants, facet boxes of tissues or water glass. Find similar camera in the home is difficult enough, but another technique showed that the restoration of sound possible with a conventional recording at 60 frames per second.

The quality of the recovered audio allows you to separate words and has a relatively high signal to noise ratio. Recovered record even allow vaguely distinguish human speech or use music recognition service.

In the above video at around 00:45 or project page shows the original sound (the researchers used a well-known to anyone, Interested in the history of recording a song « Mary Had a Little Lamb ») and restore the sound, with high-frequency vibrations in the video is not visible the naked eye - the vibrations reach less than a hundredth of the pixel.

Then, in the video at around 1:50 demonstrates the original sound, the recorded cell phone microphone, and restore the sound of human speech. The camera is located at some distance from the sound waves vibrating on a bag of chips, and between it and the objects are glass, which increases the difficulty of the task. The researchers again used the first recorded on a phonograph Thomas Edison songs.

At around 2:35 it shows that music recognition service can "learn" reconstructed record, in particular, has been recognized by the song Under Pressure group Queen.

The above results were obtained from the camera shooting frequency of thousands of frames per second. However, it was also shown that common household artifacts shooting video cameras (in particular, rolling shutter ) can sometimes be used to produce a sound with a frequency much higher than the frame rate of the original video.

The results of a modified technique can be seen at around 3:35, the researchers were able to restore the rate more than five times the frame rate of the video. We used all the same MIDI-file with the melody of children's songs.

More information and audio available on the project page . A group of researchers promises to publish the code for the project in the near future.

Source: habrahabr.ru/post/232245/