Sound effect synthesis is finally coming back into vogue and with recent increases in development as well as processing power; we are seeing great leaps in quality. In that regard I have asked Charles Verron, a developer of PANO to write this article explaining what they have been up to with their “foley synthesizer.”
There has been excellent work in recent years on sample-based and physical synthesis techniques, to simulate Foley-like sound effects. For example, the book “Designing Sound” by Andy Farnell [1] offers lots of precious information and code examples. Researchers around the globe have also achieved impressive levels of realism [2,3,4,5]. So in theory, we now have all we need to produce sounds that are interactive and realistic. In terms of professional products, however, there are only rare solutions. This article explores one of them, called PANO, for Procedural Ambient Noise Orchestra, designed by French company Noise Makers.
Philosophy
The main goal of PANO is to offer Foley sounds that are interactive and spatialized, to be easily synchronized with a movie, a game or any interactive application. It is a new tool in the sound designer palette, for creating liquid sounds, fire, wind, and all sorts of multichannel textures. During its creation, state of the art research on procedural audio has been very inspirational [1,2,3,4,5]. Our approach was to combine synthesis algorithms, real samples, and spatialization in a unique engine, which became the heart of PANO.
An important aspect during the conception was to provide a plugin as simple and intuitive as possible, yet giving access to complex sound and spatial controls. Hence it was decided to separate PANO into two parts: PANO Player, a plugin (VST, AU, AAX) with few powerful and expressive controls, and PANO Composer, a standalone with many lower-level controls, to create fully customized presets.
The Player
The best thing is probably to start by watching this presentation video.
[vimeo]https://vimeo.com/103082117[/vimeo]
Let’s have a look at PANO Player GUI, illustrated on Fig 1. It shows six controls: X, Y, Angle, Width, Tune and Gain. All can be assigned to MIDI controllers via built-in MIDI learn.
The left panel is for selecting presets, which currently include liquids, bubbles, river, flame, bonfire, fireball, wind, breeze and tornado sounds. That’s a good start, useful for many situations. It is possible to go beyond that, by creating user presets with PANO Composer. But we’ll get back to that later.
Tune controls the global pitch of the synthesizer, if we can say pitch for pitch-less environmental sounds… It will for example transform a sound from a high-pitch whistling wind into a low- frequency noise from large turbines.
The Spatialisation pad section (where you can see the loudspeaker setup) allows simultaneous control of sound position and width around the listener. It supports several output formats, binaural stereo (with front/back effects) and multichannel surround (5.1, 7.1) ideal to create immersive ambient sounds.
The X/Y pad on the right is also an innovative part of PANO. It was designed to explore sounds as intuitively as possible, in two dimensions, thanks to images. Images inform about the 2-dimensional sound texture living behind the pad. One is provided for each preset and users can also load their own images. In factory presets, the mapping is done so that brighter regions contain sounds with more intensity. For example on Fig 1, the extreme down-right corner of the pad plays strong flames and crackling sounds, while the upper-left corner plays small flames and soft hissing noises. This approach provides a comprehensive Foley instrument with continuous control. All the art is to design a preset and an image that fit well together.
The Engine
Behind its simple interface, PANO contains a powerful engine that combines synthesis algorithms, real samples, and spatialisation. Here are the main components:
- 2 granular generators
- A 32-subband additive noise generator
- 4 subtractive noise generators
- 4 wavfiles loopers
- Integrated multichannel and binaural spatialisation
The granular generators contain a bunch of pre-defined grains, and let users control the flow and turbulences. Grains were defined with liquid, fire and wind sounds in mind. They include water drops, bubbles, cracks and rustling leaves. In the future, factory grains should be completed by user wavfiles for extending possibilities.
The 32-subband additive noise generator allows sculpting a noise by shaping its spectral envelope. It is useful to create wide-band background noises, like ocean waves.
The subtractive noise generators are designed to produce narrow-band noises with dynamic variations, like whistling wind.
Finally, 4 loopers are available for loading custom wavfiles as loops, with pitch and volume assignable across the pad. It will help for designing organic textures that would be difficult to reproduce only by synthesis.
To explore the possibilities of PANO’s synthesis engine, a standalone application is provided: PANO Composer. Its goal is to expose lower-level parameters, let users experiment with them, program their own presets, and export the result to PANO Player.
The Composer
PANO composer is the application used to create all PANO presets. It was actually built in Max, at the beginning of PANO experiments two years ago. By that time, the synthesis parameters were not fixed, so Max offered a convenient way to prototype a GUI that was constantly changing. The engine of the Composer, however, is programmed in C++.
In PANO Composer, the X/Y pad is programmable. The task is to assign nine snapshots of PANO synthesis and loop parameters, to nine regions of the pad. No programming language is involved. Assignation is achieved simply by clicking the pad or using keyboard shortcuts. At run-time, the engine interpolates the nine snapshots to deliver a continuous control over the full X/Y pad. That’s the key point to convert dozens of parameters to a simple image-based control in PANO.
Thirteen factory presets are included in the Composer. They are a good starting point to create your own. They allow users to understand and sort of “reverse engineer” the components behind PANO’s liquid, fire and wind sounds.
Future work
What about using PANO directly in a video game ? That would be great indeed ! Typically, the X/Y pad could be linked directly to the movements of a player, or to physical game events, for all sorts of real-time interactions. That would probably be an excellent solution for synchronized physical synthesis in games.
PANO engine is written in C++, so technically it could be integrated into a game engine quite easily. The issue to tackle is the CPU and RAM limitations inherent to some game platforms, which require highly-optimized synthesis algorithms. Noise Makers will investigate these aspects in the near future. For specific needs, the company already offers custom synthesis and spatialisation libraries, available on demand.
[vimeo]https://vimeo.com/121990093[/vimeo]
Conclusion
PANO is a singular synthesizer for designing Foley effects, based on synthesis, real samples, and spatialisation. It is a new way to expand your sound libraries. The two pads aim to offer control of complex sounds, with an intuitive image-based approach. They allow to simulate realistic and immersive environmental sounds in seconds. Liquid, fire and wind are included in PANO factory presets. They can be extended with PANO Composer to produce all kind of spatialized sound textures. We hope that it will bring new creative possibilities, for designing sounds in movies, games and other interactive applications. A full-featured demo is available at www.noisemakers.fr.
Bibliography
[1] A. Farnell, Designing sound, MIT Press, 2010.
[2] S. S. An, D. L. James and S. Marschner, Motion-driven Concatenative Synthesis of Cloth Sounds, ACM Transactions on Graphics (SIGGRAPH 2012), August 2012.
[3] D. Schwarz. State of the art in sound texture synthesis. In Proceedings of the
COST-G6 Conference on Digital Audio Effects (DAFx), Paris, France, September 2011.
[4] J. N. Chadwick and D. L. James, Animating Fire with Sound, ACM Transactions on Graphics (SIGGRAPH 2011), 30(4), August 2011.
[5] Charles Verron, George Drettakis, Procedural audio modeling for particle-based environmental effects. 133rd AES Convention, October 2012.
Incredibly exciting and interesting tool. I will certainly be keeping an eye on this. What about room tones- can it do that?
I would however advise some re-wording of this article though- as none of the elements you offer are typically classified as foley. Fire, wind, water, bubbles- these are all atmosphere / sound effect layers. Foley is typically by deginition- footsteps and clothing movement performed by a foley artist. Perhaps at a stretch you could classify these as Foley effects. But even so, it is not typically the foley tracks where one would expect to pull down the wind. And the fact that they are not being performed by foley artists- but rather being synthesized- all the more reason why they should live in the sound effects / atmosphere tracks- not foley.
Thank you for sharing. I love seeing the different things that come out!