UNLEASH THE KRAKEN

In Star Wars: The Force Unleashed we were working with two physics middleware packages: Havok Physics, and Pixelux’s Digital Molecular Matter (DMM). In addition to the simulation data that each provided, we also needed to manage the relationship between both. While Havok has become a popular choice for runtime physics simulations, the use of DMM spoke to the core of materials and provided each object physical properties enabling – in addition to collision’s – physically modeled dynamic fractures and bending. In some ways tackling the sound for both systems was a monumental undertaking, but there was enough overlap to make the process more pleasure than pain.

Before Jumping into the fray, I just wanted to take a moment to echo a couple of things that were touched on in the companion this article; specifically, that collaboration and iteration are the cornerstones of a quality production when it comes to systems design. Collaboration, because the stakeholders involved usually include people across all disciplines; from programmers to sound designers, modelers to texture artists, build engineer’s to game designers. Iteration, because the initial vision is always a approximation at best and until things get moving, it’s difficult to know what the eventual shape things will take.

While simultaneously reigning in and letting loose the flow of creativity ebbing and flowing across the development team, there is nothing more important than the support of your colleges. Leveraging the specialties of different people helps to bring new idea’s to situations in need of a solution. Your greatest asset as a team member is to recognize and respect the uniqueness of your co-workers and stay open to the constantly shifting requirements of the game. Good listening and better communication will improve the productivity of meetings, and reinforce the fundamental desire of everyone – to craft the best player experience possible.

DIGITAL MOLECULAR MAGIC

Starting with Digital Molecular Matter, Audio Lead David Collins worked closely with Pixelux to identify the core components that could be utilized in bringing sound to the simulations. Prototypes were created offline in pre-production driving toward the best way to score the sounds of the dynamic physically modeled objects being created by the art team. With a list of over 300 types of DMM materials, we chose to abstract a group of about 30 that would cover all of the sound types and object sizes. These DMM Sound Materials were added as a “Sound Material” property to the meta data for each DMM Material type. This was the first step in defining the sound an object would make when calculations regarding collisions, fractures, and bending where concerned.

Behind each of these Sound Materials were a set of parameters for speed (fast/slow), size (small/medium/large), and quantity (one/few/many) – in addition to specifications of sound interaction between surface types – that enabled us to specify different thresholds for each and scale sample content across the different values. The content itself – abstracted Sound Cues (or Events) – were defined for use by the DMM Sound System using “Sound Buckets” which essentially specified the sound content that would be used for a given parameter’s action when triggered.

In this way we were able to appropriately employ the sound of different sized collisions and fractures based on the number and type of actions requested by the system. Behind the Sound Cue referenced in the Bucket for each sound type we had the usual control over file, pitch, and volume randomization in addition to 3D propagation min/max distances and priority – which became crucial to reigning in the number of instances of a Sound Cue during a given request from the system.

We also had bending information to deal with; specifically for metal, wood, cables, and organic vines. When the system determined that bending of a DMM object had begun, it would start a loop that would continue as long as a minimum threshold of force was being applied to the object. While looping, the system also played several single element bend “sweeteners” when spikes in the amount of bending occurred. The best example of this can be heard when wrestling one of the giant doors between area’s in a level.

For an addition summary of the DMM audio system, check out Jesse Harlin’s fantastic overview in Game Developer Magazine from September 2008.

BRING IT TO THE TABLE

We adopted a different approach to handle data coming from the Havok side of the physics simulation – where we had a greater level of detail between objects that were throw-able and caused impacts, across the different environmental material types.

One of the often used audio techniques of Physics integration in the current and previous generations is the look-up table or matrix that is used to define material actions and their interactions. Using a spreadsheet format as the starting point for the system, surface materials can be arranged along the top and far left side of the sheet. At the point where a row and column intersect the Source Material to Destination Material sound interaction can be specified, usually as an audio file or an abstracted reference to a group of files with additional properties for randomizing pitch and volume values – what we were calling a Sound Cue.

We took this methodology one step further by enabling the additional layering of Sound Cues for the Source and Destination objects. This allowed us to not only specify a Sound Cue for the specific interaction between materials, but also a default sound for the inherent object or material type. In this way, a single collision between a metal barrel and the dirt of a forest floor could incur the following impacts: 1. Metal Generic (Source Layer) 2. Dirt Generic (Destination Layer) 3. Metal Barrel on Dirt Explicit (Source + Desination Layer)

Let’s take a step back and look at how each of those things are handled within the lookup table.

MATERIAL 1 & MATERIAL 2

“Material 1” (Column A) is used to define the material type of the actor being used. (ex. A metal object would be tagged with the “metal” material) The material name is defined at the top of each material section. The size of the material selection can be adjusted using the modifier adjustment in the top left corner cell (A1). “Material 2” (Row1) is used to define any other material types used within the game environment.

SOURCE LAYER

The “Source_Layer” (Column B) is used to define a set of sound content that will play every time an object – with “Material 1” defined as it’s material – impacts a surface with any “Material 2” in the game. The “Source_Layer” has a multifunction ability: If there is an entry in the first row of a material type (ex. phy_imp_dirt) then all levels of impact will register as the same “size” and “weight”; otherwise. If the first entry in a row is left blank, you can then slot 3 sounds that will react to the size and weight of an impact as specified in the Threshold tab (sm/md/lg).

DESTINATION LAYER

The “Dest_Layer” (Row 2) is used to define a set of sound content that will play every time an object whose material is defined in the “Material 2” (Row 1) is impacted by an actor with any “Material 1” (Column 1) in the game. The “Dest_Layer” has a multifunction ability: If there is an entry in the first row of the “Dest_Layer” then all levels of impact will register as the same “size” and “weight”, If the first entry in a row is left blank, you can then slot 3 sounds that will react to the size and weight of an impact as specified in the Threshold tab. (sm, md, lg)

SOURCE + DESTINATION LAYER

The source_layer + dest_layer provides a look-up table where a sound is played specifically between a “material 1” and “material 2” impact. In the following example, when an actor with a material of dirt impacts a concrete surface the phy_imp_dirt_concrete content will play.

COLLISION’S COMBINED

In this example we are playing a combination of the 3 sounds when an actor with a material of dirt impacts a concrete surface.

MODIFIER AND EXPORT

The modifier defines the number of rows between each material as a way to prepare the values to be exported into game ready data. The export button is used to convert the spreadsheet to an efficient XML file that will be used by the game engine at runtime.

THRESHOLD

Threshold is used to define the “size” and “weight” values that are used to transition between the 3 slots defined for an object with sm, md, lg.

BODYFALL

We were able to extend the use of our matrix system to incorporate our player and non-player character (NPC) bodyfall collision’s which were handled using a combination of Havok Physics and Natural Motion’s Euphoria.

INTO THE FUTURE

While there has been some laboratory work done in the area of Synthesizing Contact Sounds Between Textured Objects by the GAMMA research group at the University of North Carolina at Chapel Hill, this approach has yet to cross over to games at runtime. In place of true synthesis, the industry is currently invested in a sample playback methodology which requires a multitude of discreet sound files that are used as a representation of a given visual. Whereas once upon a time the game industry was embroiled in the hardcore synthesis detailed at length in Karen Collin’s excellent “Games Sound”, the change to sample playback has caused the synthetic muscle of game audio to atrophy. On the horizon is mounting a recombination of the power and flexibility of synthesis and procedural audio techniques, and the fidelity of linear sound content. Beginning in 2008 with the release of Sound Seed Impact and their Sound Seed Air suite of tools, Audiokinetic is leading the charge in audio middleware towards a return to synthesis that aims to add creative options that leverage the increased CPU and reduces the dependency on predetermined sound content stored in RAM.

With everyone in game audio engaged in battle for the resources needed to achieve an exponential level of quality in the current generation, we need all of the creative tools and tricks at our disposal to accomplish this goal. I’m a fan of anything that expands upon the growing possibilities of interactive audio in a way that puts control in the hands of people who are actively looking to push the boundaries of what is possible. Where it goes from here is up to the people making choices about how we move forward as an industry and where the focus continues to be.

Until next time!

Art © Aaron Armstrong

SYMPHONY OF DESTRUCTION

Physics, the simple pleasure of “matter and its motion through spacetime”.

In games we’ve reached the point where the granularity of our physics simulations are inching closer and closer towards a virtual model of reality. As we move away from the key-frame animated models of objects breaking, and the content swap of yesteryear, towards full scale visual destruction throughout our virtual worlds, we continue to increase the dynamic ability of objects to break, bend, and collide in relation to our experiences of the physical world around us.

“It is just inherently fun break things, and the bigger the thing is the more fun it is to break. It can be a stress relief or just give a feeling of power and control. We worked extremely hard to create a virtual sand box for the player to create and destroy as they see fit, we just hope it gives them the same pure joy they had as a small child kicking over a tower of blocks. “ Eric Arnold, Senior Developer at Volition (CBS news)

Because of the joy involved in seeing the reaction of objects when force is applied, there is developing the potential to derive great satisfaction from the realism of these simulations in games. Piggybacking on the work of people doing the hard thinking about how to manage the visual response and “feel” of this technology, audio has the ability to use information from these systems in order to attempt a similarly pleasing sound accompaniment to the visual display of physical interaction. Call it orchestrating the symphony of destruction if you will, there’s nothing finer than the sound emanating from the wreckage of a virtual building.

Hooking into these systems is no small task, in part due to the tremendous amount of data being constantly output; object to object collisions, velocity, mass, material type all being calculated at runtime to a level of detail necessary to display an unfolding level of realism on-screen. Sorting and sifting through this data becomes one of the main focuses early on in production in order to gain an understanding of how sound can be designed and played back in reaction and relation to these variables.

CONSTRUCTING CONTENT

When it comes to creation of these sounds – in which the smallest impact to the largest fracture must be represented with enough variance to discourage repeatability within a short amount of time – the asset count for any given material type can easily spiral out of control.

Some considerations when defining the assets include scaling based on: number of collisions, object size, object weight (mass), the speed at which an object may be traveling (velocity), and material type. It’s with these parameters that we can begin to build an abstract audio system in order to switch and transition between the different sampled content sets and gain the ability to apply parametric changes to the content based on information coming from the simulation. These changes could include: applying a volume or pitch reduction based on the declining value for velocity, or changing between samples based on the number of collisions at a given time.

“As all of this is going on we also play audio and video cues to let the player know which areas are getting close to breaking. Beyond making the world more believable they serve as a warning system that the structure is unstable and could collapse on the player’s head if they aren’t careful and hang around too long. This small addition took the system from a neat tech demo to pulling the player in to the game world and generating very real chills as they flee from a creaking, groaning building while tendrils of dust and debris rain down around them. “- Eric Arnold (Eurogamer)

RED FACTION: GUERRILLA – CASE STUDY

In an expose’ on the physics of Red Faction: Guerrilla, Senior Sound Designer Kate Nelson lays out the fundamental systems design and decisions that went into orchestrating the sounds of destruction:

“’Players love to blow stuff up’ was a popular phrase echoing in Volition’s halls throughout Red Faction: Guerilla’s development. GeoMod 2.0 technology made real-time destruction an exciting focus of the game design team. For that reason, one of the primary goals for the audio team was to ensure that when players ‘blew stuff up’ they experienced satisfying and immersive audio feedback. Overall, designing the destruction audio system was a rewarding challenge.”

THE ROAD TO RUINATION

Everything to do with audio for the destruction system is piggybacked on top of the underlying physics simulations being run by engineering. Because there are so many stakeholders involved with bringing this to life on screen and through the speakers, it’s usually an exercise of collaboration to the highest degree in order to find commonality between disciplines, and pave the way for elegant design.

“Once art/programming/design had a good prototype of what behaviors were expected, our audio team spent a good amount of time picking the destruction system apart and narrowing it down to the basic core components that affected audio:

– Every building and object in the game was made up of different materials, and those materials each referenced a core “destruction” material

• All things artists made out of metal, regardless of texture, could be thought of as “steel”
• All things artists made that were rocks and concrete could be combined into “concrete”

– Each material was engineered by artists and programmers to break in a specific way:

• Concrete would break into smaller pieces of concrete which would range in size but had consistent shape
• Steel would break into smaller pieces of steel which would range in size and vary in shape (sheets, poles, and solid chunks)

– Each material and shape combination would react differently in the environment when tossed around:

• Concrete blocks impacted the ground, rolled down hills, slid across flat surfaces, etc.
• Steel sheets impacted the ground, slid across flat surfaces, but would not roll, etc.

Leveraging the the core commonalities in order to appropriately simplify things from a sound perspective led to a greater understanding of the content needs and allowed a greater focus and level of detail on a few types of materials. When every game has to temper the fidelity of it’s systems – be it ambient, footsteps, or physics – with the availability of CPU resources and RAM, choosing where to allocate resources can be a constantly shifting management game that relies on constant gardening to make sure every sound type get’s what it needs to survive and be heard.

“From these observations and with keeping in mind that we had to keep our asset count down, we were able to extract a basic idea of the kind of audio assets we would need to create and trigger:

[Material type] + [shape] + [size] + [destruction event]

Examples:

• Steel + Sheet + Large + Impact
• Concrete + Solid + Small + Roll

“Once we identified the kind of sounds we wanted to produce with the destruction engine, we found the best way to be able to communicate our vision was by capturing footage of the game and adding sound design that illustrated the kind of support we wanted. This video was key in selling our concept to the project producer and explaining to our programmers what we needed to hear and why – nothing says “this is why it’s fun” better than witnessing the moment itself.”

FUNDAMENTAL FACTORS

Working closely with Eric Arnold and senior audio programmer Steve DeFrisco, the audio team (including Kate Nelson, Jake Kaufman, Raison Varner, and Dan Wentz) drove towards a system that was able to identify the four destruction factors they had identified for content.

– Material type (destruction sound material) was added to objects, environments and materials by artists or members of the sound team.
– Shape was determined after destruction – in real-time – by code that measured each material piece after destruction.
– Size was determined at the moment the destruction event triggered, based on the mass and velocity of the object:

• A X kg piece of concrete traveling at Y velocity impacted with Z energy value
• Audio designers specified which Z energy value triggered a large, medium, or small sound event for each destruction material.

– Destruction Events were calculated in real-time determined by the action of the material piece:

• Upon colliding with something, an impact would trigger
• if the piece continued to move in a horizontal fashion and was turning on its axis, a roll would trigger

The combination of systems and content design forms a mighty Voltron of destructive power and sound mayhem which can be heard resonating through every destructible element in Red Faction: Guerrilla. Once everything was up and running with sounds being triggered appropriately, the complicated task of applying sound voice limiting and additional sound finesse helped to reign in the multitude of sounds escaping from every impact.

Examples of other key features:

• Volume attenuation was applied based on the velocity of destruction objects.
• Playback of destruction events was controlled in order to not overwhelm the player.
• An additional material layer could be specified if needed to differentiate between surfaces that were struck.

Additional thoughts:

• Adjustments to Mars gravity during development made tweaking size energy values interesting!
• It was necessary to strike a balance between having enough destruction assets to provide necessary playback variety, and having so many assets that we blew our memory allotment.
• It took time to determine how many destruction sounds should be allowed to play at once to provide the necessary destruction feedback for the player without swamping the overall soundscape.

It can be said, that with the escalating demands across departments in game development, the single greatest asset on the side of quality continues to be the ability to quickly develop and implement systems that can then be iterated upon as we all work towards simulating models that meet and exceed expectations. The development of any complex system absolutely depends on this iteration in order to dial in the important aspects which will help to sell the desired effects.

UNTIL NEXT TIME

Tune in for Part Two when we look at the systems behind the physics in Star Wars: The Force Unleashed and look towards the future beyond our current reliance on sample playback technology.

Art © Aaron Armstrong