Interface and Dimensionality in Music Spaces
Continuing from "Played Meaning" -- How are values generated in play when we're not given explicit goals to pursue? One way: engagement with material presence and its variability. Sound, image, touch, interface, feedback-- IMMANENCE -- chaotic movement of emerging and collapsing possibilities.
Boundaries, whether goals or materials/actuality are-- constrained possibilities.
Practical applications of this idea to music design
--seeking fluid spaces--
--seeking fluid spaces--
with new soundtracks for:
Super Mario Galaxy
Assassin's Creed
&
&
Psychonauts
//
//
1 - Super Mario Galaxy Ballet
For every change of state [process, event] in a game, there should be a corresponding change of state in its soundtrack.
Interactive mickeymousing -- first guideline for a musical-cybernetic realism, systemic representation of the idea that every change of state in the perceived world is itself a movement of energy, vibration. Vibrations, material variability-- light, sound, touch, & movements of consciousness, moods, ways of being.
If a soundtrack adheres to this dogma, it can be considered realistic.
Searching for grounds of this new realism-- basins of attraction, affinities and repulsions in our played experience of the game space, leaving behind Dogma 1:
If a soundtrack adheres to this dogma, it can be considered realistic.
Searching for grounds of this new realism-- basins of attraction, affinities and repulsions in our played experience of the game space, leaving behind Dogma 1:
3 - Musical Dimensionality
A realism of movement-as-creativity-- vibrational play in fluid possibility spaces. Games-- play in possibility spaces. Games best represent creative reality by becoming fluid--using continuous variables (i.e. range 0 - 100.) as opposed to constants (i.e. 23) or booleans (1 or 0)---and by becoming perceptibly vibrational, with sound and light and touch feedback.
The structures of sound feedback, music, with their relationships and meanings emerging in play, seem to have a curious relationship to the play of fluid systems more generally.
The way the experience of music dissolves the discrete relationships between sound objects and events is a useful mental model for designing (and playing in) shifting possibility spaces.
Games as objects in motion >> Music as objects in motion >>
The way the experience of music dissolves the discrete relationships between sound objects and events is a useful mental model for designing (and playing in) shifting possibility spaces.
Music objects: note, mp3, chord, A-section, B-section, voice, guitar, fermata, melody, reverb, granulizer, [cos~], etc. These objects are not static-- rather, they're defined by variables, little worlds of objects making up the bigger objects.
The melody went up, but it just as well could have gone down. The chord was a C major, but it just well could have been a c minor. The mp3 was played at normal speed but it could have been 50% speed. The Eb could have been an F. The magic circle is elastic, pulled and twisted when its boundaries are engaged. All of the objects themselves are defined by variable qualities-- possibilities. As such these objects might better be understood as spaces. Music is the organization of these shifting spaces.
Like in a phase space, which maps relationships between variables in a system (above-- 2 and 3 dimensional phase spaces), representing a musical space will require 1 dimension per variable.
Western music notation is a 2-dimensional plane-object (page) with further embedded dimensions at lower levels (> staff > symbols).
The staff object-- horizontal dimension represents time, and vertical dimension represents the present moment, pitch information:
Western music notation is a 2-dimensional plane-object (page) with further embedded dimensions at lower levels (> staff > symbols).
The staff object-- horizontal dimension represents time, and vertical dimension represents the present moment, pitch information:
Object-types placed on the staff are themselves variables or spaces. A fermata means hold, but doesn't say how long-- it's an open space. An eighth note is a selection from the class of note-heads and -tails (whole note, quarter note, eighth note, etc.), which determines the duration of the note. The class of clef (treble, bass, alto, etc.) determines how the staff is mapped to actual pitch-values. We can describe any musical object as a space and/or a position in a space-- an object / a component of an object. The categories are not rigid-- determining the members of a class is a compositional choice.
As flat visual information, the language of musical symbols also exists in a 2 dimensional space, but at a lower level than the staff (which, again, is one level lower than the page). The symbols are the objects which play on the staves, which are the objects which play on the page. The vertical and horizantal dimensions of symbols don't measure anything in particular, they just offer a space for the symbolic images to be constructed in.
This is an impossible project on a 2-d cartersian-XY plane, so we need to look for different representations of dimensional movements and relationships.
Western music notation's stacked dimensionality (page > staff > symbols) is a great solution given the constraint of the page as a static space. Dimension 3 embedded in dimension 2 embedded in dimension 1.
And its potential for layering musical information is very powerful, allowing for the emergence of new complex objects built from symbol and staff units. Layering staff-planes vertically (a higher level of 2-dimensionality, containing the first), Western notation allows for simultaneous play by multiple voices (players), each its own contained space of other spaces in motion, and thus emerges the fluidity of texture, shifting relationships between what objects are present and what they're doing:
There's a lot going on there, including many composite-objects that have emerged from a lower level-- like harmony (emerges from layered pitch) and textural qualities, like which players are playing in unison with one another (emerges from layered staves). Some emergent objects are more intuitively represented by the notation than others. For instance, in the above image, you can see from the shaped paths of symbols on staves that the fourth staff from the top and the second staff from the bottom are playing in unison (rhythmically, at least)-- texture is seen to be communicated fairly intuitively. Harmony, on the other hand, we have little chance of decoding from this distance-- and even once we zoom in, the system of #s and bs doesn't represent very well the fluid centers of gravity/tonality which give the vibrational presence of the sound its meaning.
The game of classical music theory/analysis is an attempt at systematic decodings of musical spaces like these. Creating systems of harmonic, motivic, structural (etc) objects and their interrelationships. There's likely a lot to learn here, though all too often analyses are primarily concerned with pitch/harmonic information and discrete-structural chunks, the fixedness of the objects, avoiding the more difficult but also more interesting task of finding ways that fluidity emerges from shifting relationships between objects playing in dimensionalities across all hierarchical levels.
from "The Rite of Spring" by Igor Stravinsky
There's a lot going on there, including many composite-objects that have emerged from a lower level-- like harmony (emerges from layered pitch) and textural qualities, like which players are playing in unison with one another (emerges from layered staves). Some emergent objects are more intuitively represented by the notation than others. For instance, in the above image, you can see from the shaped paths of symbols on staves that the fourth staff from the top and the second staff from the bottom are playing in unison (rhythmically, at least)-- texture is seen to be communicated fairly intuitively. Harmony, on the other hand, we have little chance of decoding from this distance-- and even once we zoom in, the system of #s and bs doesn't represent very well the fluid centers of gravity/tonality which give the vibrational presence of the sound its meaning.
The game of classical music theory/analysis is an attempt at systematic decodings of musical spaces like these. Creating systems of harmonic, motivic, structural (etc) objects and their interrelationships. There's likely a lot to learn here, though all too often analyses are primarily concerned with pitch/harmonic information and discrete-structural chunks, the fixedness of the objects, avoiding the more difficult but also more interesting task of finding ways that fluidity emerges from shifting relationships between objects playing in dimensionalities across all hierarchical levels.
How we describe musical possibility spaces is an implied project in musical instrument design, music theories, notations-- all musical interfaces, which could be analyzed in terms of their stacked dimensions, as one way of identifying nodes of connectedness and paths of movement.
Graphic scores offer new ways of navigating the 2-dimensional plane (page) of musical information, often by being less systematically precise (but more visually evocative), so that object relationships can emerge in play with more fluidity, an important precedent of games/fluid visual systems -as-movement.
from "Treatise" by Cornelius Cardew
Videogames are musical interfaces. And, being dynamic, they're of particular interest, re: fluidity. What was was possible at one moment isn't at the next-- possibilities shift. The information, and space, we're given at one moment is gone the next. Like pages and staves and symbols coming and going, morphing and transforming, in games dimensional-crossings open and close in real time, a new detail pulling us down into a lower level, another pulling us across dimensions, another shooting us back up to the higher-level object made of those component objects, and up and up and down and across until the dimensions dissolve into pure movement, pure nowness-- the game itself in the world, our being-in-the-game in the world.
4 - Psychonauts Walkthrough
Psychonauts + new score >> mental model of motion-creativity-- walls of a psychedelic (psychê-dêlos / ψυχή-δηλος meaning mind-manifesting) space, coloring ways-of-being with the shifting moods of sense-possibility.
Screen as a visual interface for musical play, much like a Western score, but these possibilities slide, they are not fixed. Movement between dimensionalities here is equated to alternation between different modes of play, engagement of different variables.
So-- pseudo-algorithms, a verbal/written score, the kind of discretization of fluidity that will be needed to implement a music space like this in code. How can music be designed (fixed) fluidly? :
//
Screen as a visual interface for musical play, much like a Western score, but these possibilities slide, they are not fixed. Movement between dimensionalities here is equated to alternation between different modes of play, engagement of different variables.
So-- pseudo-algorithms, a verbal/written score, the kind of discretization of fluidity that will be needed to implement a music space like this in code. How can music be designed (fixed) fluidly? :
//
(:00) - Grinding on a handrail. Automated linear movement in one of two directions, left or right. Loop's playback direction switches based on left (reversed) or right (forward). Switching direction plays melodic event sounds, scale intervals 5, 1.
(:08) - Camera cuts to new perspective, in same room as the avatar. Cut is accompanied by arpeggiated sound event, introduction of fluid room motif. Changing directions also triggers changes from the room's dimension of the sound space.
(:14) - Sliding down ramped handrail. Handrail loop material is pitch-shifted up a whole step, +2 semi-tones. Bell-shaking event plays along with slide.
(:15) - Landing event, dimension of room motif, new key having been introduced by sliding down the handrail. Handrail loop and bell-shakes fade out.
(:16) - Footsteps are accompanied by diatonic scale tones in the new key. Room motif accompanies loosely.
(:21) - "eyes opening", arpeggio up.
(:23) - running on platform, motor rhythms accompanying run-tempo.
(:29) - hop off platform, no sound "effect", instead introduction of a new harmonic rhythm, which stays around for 2 seconds while the avatar is obstructed from the camera by the wood of the platform.
(:32) - flute ornamentations enter, accompanying flight of butterflies on the screen.
(:37) - bright tinkly sound fades in, accompanying shiny object past the fence.
(:39) - jumping stops motor rhythm. jumps cycle through an array of chords. Extra triangle ding accompanies sphere-effect jump.
(:42) - volume swells briefly while spinning over fence. then we hear a mixture of walk and jump motifs for a few more seconds
(:44) - falling, ilinx
//
How might this have sounded if we played differently / went someplace else? Playing with the touch of the controller, do some musical interactions seem more compelling than others? Which of the game's mechanics weren't scored & how might they have been scored?
What would verbal/pseudo-code event-scores for the two preceding videos (Mario Galaxy Ballet, Assassin's Creed) look like?
//
How might this have sounded if we played differently / went someplace else? Playing with the touch of the controller, do some musical interactions seem more compelling than others? Which of the game's mechanics weren't scored & how might they have been scored?
What would verbal/pseudo-code event-scores for the two preceding videos (Mario Galaxy Ballet, Assassin's Creed) look like?
5 - Practice
The practice of scoring game mechanics with music that aims for a 1:1 relationship with them is still not very common. My hope is that the above examples begin to demonstrate that games-as-music are entirely possible, and not even particularly difficult to explore conceptually, when built on the foundations of the above rule of corresponding state-changes (music-game), which of course can, and ought, to be broken (but considered a rule nonetheless).
The rest of the rules we follow or push are supplied by a game's mechanics themselves.
We need to open ourselves to games' existing played time-structures. A common approach to making music games is to introduce a system of quantized timing (even subdivision of flow into time-units, often measured as a sixteenth-note pulse), so that the music grooves automatically, plays with a steady beat. But most games don't play like this. Nor does most music. Our systems of musical representation (i.e. the 16-step sequencer) suggest that music is quantized, but this is only the case in some virtual spaces. In played-actuality, music is smooth, even when it tends toward even subdivision. Don't be afraid to turn off quantization-- open up to response structures, emergent rhythms, these are games' best friends.
If a game being scored has even an average interaction density, the compositional process of designing its soundtrack according to the music-game rule of corresponding change will involve creating and manipulating many many music objects. Most composers are used to composing full pieces made of smaller units-- such processes will need to be redirected here. The smaller pieces, or modules, are what's needed content-wise, the "full" piece being the space of possible movement itself, the music design, which might resemble a more detailed version of the Psychonauts walkthrough (i.e. the rules for John Zorn's "Cobra").
Many objects seem to want to stay fixed (like recorded music), but they will need to move if they're to be plaeyd. Find tactics for penetrating into objects and dissolving them into their components and their potential for played variability. Build notes, motifs, textures, loops, processes. Build events from these, sequences, liquid stories. New game object? New story. Composition will need to happen so frequently and quickly, it would be good to develop an improvisational relationship to it, being able to compose without stressing about doing anything well, just playing in whatever the most efficient and fun way might be. Not a full orchestra, unless there's a well thought-out strategy developed to dissolve its unity. Want an orchestra? Whatever MIDI sounds should do you just fine, or use samples-- modules can easily be cut and shaped from existing recordings, and there's loads of music in the public domain filled with amazing sampleable bits. Don't let "it's too much work" be an excuse to not fill out music designs, just be lazier about it, automate the work, play it always. The content doesn't matter too much anyway-- it's how it moves.
To practice designing music in this way, more videos like the above could be made using the same technique-- music design without even having to touch code, like scores without a performance. At every change of state, we face interesting compositional decisions to play out, possibilities to navigate, movement in so many undiscovered dimensions. We could make games like this forever. Yes, You Can Make Games, but don't think you have to touch programming to do it-- just keep moving.
The rest of the rules we follow or push are supplied by a game's mechanics themselves.
We need to open ourselves to games' existing played time-structures. A common approach to making music games is to introduce a system of quantized timing (even subdivision of flow into time-units, often measured as a sixteenth-note pulse), so that the music grooves automatically, plays with a steady beat. But most games don't play like this. Nor does most music. Our systems of musical representation (i.e. the 16-step sequencer) suggest that music is quantized, but this is only the case in some virtual spaces. In played-actuality, music is smooth, even when it tends toward even subdivision. Don't be afraid to turn off quantization-- open up to response structures, emergent rhythms, these are games' best friends.
from "De Kooning" by Morton Feldman--
timing system based on response rather than subdivision
If a game being scored has even an average interaction density, the compositional process of designing its soundtrack according to the music-game rule of corresponding change will involve creating and manipulating many many music objects. Most composers are used to composing full pieces made of smaller units-- such processes will need to be redirected here. The smaller pieces, or modules, are what's needed content-wise, the "full" piece being the space of possible movement itself, the music design, which might resemble a more detailed version of the Psychonauts walkthrough (i.e. the rules for John Zorn's "Cobra").
Many objects seem to want to stay fixed (like recorded music), but they will need to move if they're to be plaeyd. Find tactics for penetrating into objects and dissolving them into their components and their potential for played variability. Build notes, motifs, textures, loops, processes. Build events from these, sequences, liquid stories. New game object? New story. Composition will need to happen so frequently and quickly, it would be good to develop an improvisational relationship to it, being able to compose without stressing about doing anything well, just playing in whatever the most efficient and fun way might be. Not a full orchestra, unless there's a well thought-out strategy developed to dissolve its unity. Want an orchestra? Whatever MIDI sounds should do you just fine, or use samples-- modules can easily be cut and shaped from existing recordings, and there's loads of music in the public domain filled with amazing sampleable bits. Don't let "it's too much work" be an excuse to not fill out music designs, just be lazier about it, automate the work, play it always. The content doesn't matter too much anyway-- it's how it moves.
To practice designing music in this way, more videos like the above could be made using the same technique-- music design without even having to touch code, like scores without a performance. At every change of state, we face interesting compositional decisions to play out, possibilities to navigate, movement in so many undiscovered dimensions. We could make games like this forever. Yes, You Can Make Games, but don't think you have to touch programming to do it-- just keep moving.
6 - Dissolving Composition, Instrument, & Notation in Play
When a soundtrack is subject to the variability of a game's mechanics, the music itself becomes a mechanic, an instrument-- music to be played. And the fluid space that houses this mechanic becomes a space for the play of instruments, a composition-- again, music to be played.
Spaces hold other spaces which in turn hold others, and so the hierarchical relationship between instrument and composition which says that the former is a component object used in the latter can be dissolved, reversed. Now a composition can be a component object of an instrument. The screen is the "page" displaying the fluid notations that allow for interfacing with the space-- notations which themselves are instruments built of compositions forming instruments, opening new notational dimensional-movements, and so on. This is a music space.
The space of all games-as-music is no less infinite than the space of all music itself-- in play, the two spaces are one.
>>>>>>
**************************
some good books for
******************
>> continuing fluid music/playspace research:
"Infinite Music" by Adam Harper is full of great stuff about music as object & space. The practical applications of the object-oriented music theory it describes are endless. Harper's blog post "Musical Radicalism Beyond the Sonic" is related, and it's about "music without sound," which could just as well be a description of the played time-structures of games-without-soundtracks.
"Chaos" by James Gleick is a great introduction to chaos which a lot of people have read, but if you haven't-- it's a window into lots of inspiring ideas about self-organization, great models for understanding emergent playspaces. In music spaces, the organization of objects requires some amount self-organization around the player as a chaotic agent-- played emergence. Some of the concepts explored in this book, like attractors (the swirly things which i used phase-space images of throughout) and state-transitions (i.e. solid -> liquid -> gas) seem like they might be useful structural units for shaping shifting gravities and tendencies of fluid possibilities that define a space. The book also introduces some ideas from topology, which is a geometry(ish) of elastic spaces in n-dimensions (like music) and thus likely applicable to all of this-- new models of dimensionality. Also part of the "Gleick's Chaos" brand-- this software, which supposedly works in dosbox, but I couldn't figure out how.
And-- Gilles Deleuze-as-playground: start anywhere and swim, movement in the spaces of motion-image/time-image, plane of immanence, nomadology, smooth vs. striated time, the actual and the virtual, difference & repetition-- concept-modules in an alternate theory of play-as-movement that values the always-now of forever-change over controlled behavior governed by false belief in the fixedness of objects. -- As chaotic agents in the played systems we're describing, the need for an ethic of creativity and movement emerges, and there's a project here to discover one in play--
About this-- from the preface to Deleuze & Felix Guattari's Anti-Oedipus, Michel Foucault writes:
"[one is led] to believe it is all fun and games, when something essential is taking place, something of extreme seriousness: the tracking down of all varieties of fascism, from the enormous ones that surround and crush us to the petty ones that constitute the tyrannical bitterness of our everyday lives."
others?
>
