Last year I was experimenting quite a bit with Flash’s computeSpectrum functionality.   With computeSpectrum, I could take a snapshot of a playing sound and get volumes for various frequencies of the snapshot so I can visualize the sound.

Now, I wanted to take it a step further and visualize the sound before it’s even played.  Of course doing this with computeSpectrum is impossible because the sound has to be played first.

Luckily, in Flash Player 10, Adobe introduced sound.extract();  This can take the sound object and turn it into a byteArray.  I came across an example by Thibault Imbert and of course the example worked very well to visualize a sound spectrum from a byte array - though I still wasn’t sure what each byte represented in the Flash Sound object.  And even though, the spectrum looked OK,  a comment from Schell explained that this is a false waveform and to look at his blog post.

The blog post provides some great working code - but there was quite a bit I didn’t understand yet.

So, I attacked the problem from another angle today.  I thought - OK, I don’t really understand the bytes that are coming OUT of the sound object, but maybe if I play with shoving bytes INTO a sound object to create dynamic sounds, I can get a better handle on tings.

The real struggle came because I used Christopher Martin-Sperry’s code to create a sound object from the raw byte array provided by an MP3 file.  I used this before really looking into how all this stuff works.  The code works fantastically - but put my brain on the wrong track.

I was on the wrong track because MP3’s and Flash’s sound Object work a wee bit differently from each other.  In an MP3 file, there is a header to give some info about the file.  Also, the MP3 file is broken into “frames”.  Each frame has a header to give information about the frame (bitrate and other things).

This doesn’t map so well to Flash’s sound object.  First of all, the bitrate of a Flash sound object looks like it doesn’t vary.  It seems to be 44.1 kilobytes per second all the time.  And it’s always in stereo.  It looks like the source of the audio will always get upconverted or downconverted to this 44.1KBps stereo format.

Not only that, but there’s no concept of frames.  I thought I’d need to know how to read a frame header, or at least know how long each frame is to know what type of data I’m getting.

But that doesn’t appear to be the case.  The byteArray extracted from a sound object appears to be an alternating left/right channel stream.

How does the stream relate to time? I felt a little dumb when I figured this one out - because it’s a little obvious.  It generated a few tones before having this little gem of an epiphany.

Well, if the Flash sound object is 44.1KBps, that’s 44,100 bytes per second.  So when reading the stream, we read 44,100 bytes before a second is up. Right?

Well, not so right it would seem.  You have to consider both channels.  You’d read each channel 44,100 times to make up a second, making one second 88,200….something.

What is that something?  Well my computer sciece teaching has failed me, but I can speculate.  Adobe provides a morse code generation sample on their developer network.

In it, there is a comment when they are writing the samples to the sound stream that they are writing 8192 samples to the stream.  However, the length of the byte array they are actually writing is 8192 x 8, explaining that each sample is two 4-byte floating point numbers.

So perhaps when reading floats with ByteArray.readFloat, you’re actually reading both left and right floats from the same byte.  If I’m right, then you’re still using 44,100 bytes per second, but each byte represents two floating point numbers.

However, it works out, I’ve noticed that when creating six 1 second tones, by writing two floats at a time, 44,100 times,  it clocks in at 6 seconds.  So infer whatever you wish from that - I know I have.

My next step is to figure out how to pull frequency data at a particular point in time.

Just writing a short post to tout theWB.com which went live today after a makeover.  My new team and I at Digitalsmiths produced the video players and the clip search tool.  After just cancelling cable, I can’t argue with even better video sites putting out great content.  I’m always a fan of Buffy and Angel which they have plenty of.  Everyone should also definitely check out the short WB shows “Joni and Susanna” and “Children’s Hospital” starring Rob Cordry.  Someday soon I’ll definitely check out “Babylon 5″, but I don’t know if I have enough geek cred yet.

http://www.theWB.com

This is the last part of my recent musings.  Over the last two posts I’ve argued that digital storage has completely changed the way we listen to music, unfortunately in some ways for the worse.  We’re less inclined to replay an album in whole, and more inclined to put it all on shuffle.  A friend of mine wondered how well Pink Floyd’s Dark Side of the Moon would work today given our habits.

I wondered how we can increase replayability of music and get people to appreciate a song or collection of songs more and thought we can use games to achieve this.  After all 30 year olds like myself grew up on video games with soundtracks that repeat every time you play a level.  Why not use casual games to increase your band’s audience - to get people to listen to your music over and over again and get the tune stuck in their heads?

Of course, its easy to slap a soundtrack on an online game - but how can we make the game and soundtrack work together for a great experience?  How can we create musical escapism in games, just like MTV or musical theater?  What is the gaming equivalent of breaking out into dance, or fast camera cuts timed to the rhythm?

As you can tell, I’ve been thinking a lot about this lately, and I’ve recently read This is Your Brain on Music by Dr. Daniel J Levitin to better understand how our brains perceive music and how the various flavors of music appeal to us in different ways.

The most obvious way to tie a game to music is tempo or rhythm.  There are many rhythm based games on the market today - most noteably “Rock Band”.  Music has many more attributes, obviously, like the tone, contour, timbre, loudness, meter, theme, key,  harmony and melody….not to mention that rhythm and tempo are two different things.

Relating any of these aspects to games can be challenging given where we are with mainstream realtime audio-processing technology.  But even putting technology aside, how can we relate these to a piece of music in a systematic way?

Let’s start with tempo.  Tempo is the overall speed of the music.  It’s fairly constant for long periods of time - but can be an excellent baseline metric for establishing a musical pulse and to possibly tie it up with a visual pulse in your game.  When people tap their toes, dance, or bop their heads - its generally to this tempo, or the beat of the song.  So tying visual elements to this beat is probably the most effective tool I can think of.

And this is good, because a beat can be measured in pop music by picking the range of frequencies that the bass drum is on and listening to the volume of this frequency over time.  So tying visual elements to a beat is a very real tool that can be done automatically in music game creation.

Tempo only goes so far though.  It’s a very predictable musical trait.  Levitin argues that our musical tastes can somewhat be centered around complexity and predictability.  Children, for example, enjoy very simple and predictable music.  As we get older, and the more we listen to music, the more boring this simple and predictable music becomes.  So we listen to more complex music - but constantly strive for the right amount of predictability and the right amount of complexity.  As we listen to more and the level of complexity of our favorite music will probably go up to.  This is one of the reasons that the music of other cultures can be distasteful, or classical music, or jazz can be distasteful for people with pop only listening habits.  The more unfamiliar we are with music, the less predictable it is.  And if its not predictable at all - as another culture’s music can be to us - then it can really be unpleasant.

So, going beyond the beat to capture other nuances of the musical piece should be very important in manipulating visuals or gameplay elements.  ONLY syncing a predictable visual beat with a complex soundtrack would be such a shame because you lose all that which makes the music meaningful - and you can lose the connection between the two.  Alternately - having lots of movement that has nothing to do with the flow of the piece but loosely tie to frequency at a particular time, can create a similar disconnect because you’re picking up unpredictable parts of the piece to tie them together visually with the music.

So what types of musical nuances can we pick up and use?

Rhythm is another form of timing like tempo but it’s how notes are grouped together into phrases.  Guitar Hero and Rock Band use rhythm effectively.  I believe Harmonix actually does this by transcribing the notes themselves for each instrument, and grouping them together to have you play out a phrase at a time.   They’re not picking out random notes from the guitarists score and having you play them - no, they’re taking  the most meaningful notes  that make sense to put together the phrase with the limited amount of notes you can play.

Tone and frequency for example are aspects that may be quite hard to integrate effectively.  Considering an overall musical piece, many instruments are playing at different pitches and frequencies all at once.  The bass guitar has a very low frequency, while a flute can have a very high frequency - even though both can be playing the same note or different notes.  Taken out of context, different frequencies/tones in music don’t make us tap our toes.

Pitch, harmony, contour, key, and melody - taken together can be an entirely seperate but equal way to tie visuals to your music.  Unfortunately - it can be very difficult (if not impossible) with mainstream game technologies like Flash to take this into account in an automatic way.  I hope I’m wrong - and I hope somebody PROVES I’m wrong and lets me in on the secret, but consider this….

Using Fast Fourier Transform methods (FFT), I can grab the volume of any frequency at any time.  Can this provide me with what note is playing?  Well, maybe….but only if a single instrument is playing.  Unfortunately many instruments are playing at many different frequencies, and many can bleed into (if not use) another’s frequency.  I know some smart computer scientists have developed pitch detection - but it hasn’t made it’s way into any code libraries I know of yet (though I should look beyond Flash to see what I can find).

Also, on a technical level, using MIDI files, if the music is transcribed correctly (or if it originated on a sequencer or a computer in the first place), we would have access to all the notes seperated out into the different instrumental tracks.

So what if all this technical stuff presented no barrier?  How could we use it?

One of the most interesting parts Dr. Levitins books for me was Appendix B: Chords and Harmony.  Music, if you look at it in a very dull light, is all about patterns and manipulating people’s ears by breaking in and out of  the predictiability of those patterns.

Talking about pop music, there is a verse, a chorus, and a bridge.  A verse is normally a melody and tempo played over a few times.  At the end of the verse, we get into a similar thing with a chorus, which is generally shorter.  Going from chorus to verse breaks one pattern, but picks up another musical pattern which uses slightly different chords with a slightly different tempo.  So predictability is broken, but not very much.

Another way patterns are established is with long standing musical tradition.  The blues is somewhat defined by going from I Major to IV Major to I Major and then IV or V major, and then back to I Major - this happens to also be the basis for rock music as well.  Now, this is another established pattern - and when we break from this pattern, we’re introducing unpredictability.

The most intersting pattern Levitin mentions is when a chord is either resolved or unresolved.  In Western music, our ears have been trained to consider a tritone or an augmented 4th, the most unresolved interval we could possibly hear.  In fact, as Levitin recounts, this interval was banned in the Catholic church and named “Diabolus in musica” citing this interval as the work of the devil.  On the other hand, a simple major chord is considered resolved.  So when if we play a chord containing a tritone, we expect, and almost demand that it be resolved by something like a major chord.

Why do I bring these up?  Well, this type of musical behavior can start to paint a more visual or motion based picture.  When music breaks expectations either by changing tempo or with different chords, or even moving the notes up or down an octave - likewise our gameplay elements should match this level of broken predictability.

Likewise, when a chord is left unresolved - so too should our visuals on screen.  You know something is going to happen, but you don’t know what.

To spell it out more clearly, we can establish a visual and animated baseline in our games by listening to the beat of the music, and timing elements to this.  Assuming a pop song, our baseline is a song verse.

Deviating in minor ways while the verse is played, maybe be changing the tempo, going up an octave, or different instrumentation, will produce a minor change in the visuals or animations.

Going from verse to chorus however is a less subtle change, and often implies changing the tempo, chords, or otherwise quite drastically.  In this, the gameplay needs to change in the same fashion.

Hitting a chord that needs to be resolve creates musical suspense - and so too should it create visual or gameplay suspense.  Something on screen about to fall, or teetering from side to side.

Going back to technical implementation, however, assuming we had access to the entire score of notes in a musical piece, can we feel out these suspensful moments or feel out when a song is predictable and when it becomes unpredictable?

I believe the answer is yes, but it would take quite a bit of work to look for augmented fourths, or other unresolved intervals, or to run pattern matching algorithms on our melodies.

A more pragmatic approach might run a FFT analysis to find beats, and possiby even do some light pattern matching to seek out changes in tempo.  Meanwhile, do some manual markup of subtle changes in verse, and then manually mark where verse goes to chorus, chorus to verse, and where the bridge is.  We can also manually mark where we’re creating suspense, and then where we resolve that suspense.

In this fashion, we could create a sort of music markup language to map a timeline of all these events and use this to create a musical gameplay experience.

I guess my work is cut out for me!

Last Thursday my Hauppauge Dual Hybrid HDTV Tuner card came for my computer.  Until then I’d been using another dual tuner card to record and watch cable TV through Windows Media Center.  It was cool and all, but in that time we’ve gotten Netflix, and found out about Hulu.com, and I got a job working at bringing video and video search to the web from major studios and content producters.  So there’s a ton of digital media out there on the web (stuff you might actually want to watch…..stuff that you might normall have to have cable TV for).

Netflix On Demand is damn cool - I was watching HBO series and new movies through a nice Windows Media Center interface called MyNetflix.

Not only that - but the digital content from online looked a lot better than my cable signal did (I still have a normal non-HDTV by the way).  I think that they compress the content digitally before it comes through the cable wires - and then of course Windows Media Center compresses it again in a different way….and thats no good when you compress the same digital video twice under two different compression standards.

So - thus begins my experiment:  Bringing HDTV to my computer even if my TV doesn’t really support it.

Saturday morning, I installed the card.  I was sorta hoping that it would just magically pick up HDTV from the air without plugging in any antennas or anything.  Well that was a big fat no.

Then I went to Best Buy - I got an multidirectional amplified indoor antenna.  Basically, a big flat sqaure, probably with some wires inside, and you plug it into the wall.  So - I hooked that up….in my finished basement…where we watch TV.  OK, a big fat no.

“This is discouraging” I thought - but thought I’d keep trying, since I was in the basement after all.  I grabbed a roll of coax wire and ran the indoor antenna outside to my raised porch.  I tried again.  This time it was more promising.  I scanned through the channels, it said there were a few digital channels available - so I started flipping through.

It looked good but not great.  It was a little snowy, some bad pictures, definitely not looking like what I expected out of HD.   Oh but what’s this………there’s a WRAL-DT?  Umm….hey a whole bunch of DT channels in the low 1000’s.  So apparently, I wasn’t looking at HD, Windows Media Center just threw in the Analog channels into my guide for good measure at the beginning of my guide.  OK, lets flip to the HD stuff…….daaaaaaaaaamn.  It looked awesome.  Very crisp and clean on my computer monitor, I hadn’t even hooked it back into my TV yet.

So, I set out to see what channels I could get…….

ABC, check….NBC….check, CW….check, PBS….check, FOX….check, CBS…….where’s CBS?

I went to my local CBS affiliates website, and found that I’m about 15 miles outside of their range using an indoor antenna.

Sunday morning, I set out to the hardware store to see what else I could buy in antennas.  I sorta wanted an outdoor one, but the thought of me going up on the roof with my non-existent ladder and installing something - somewhere, somehow on my house seemed hazardous to my health.

I got to Lowes, saw they had rabbit ears and outdoor antennas.  Thats it.  But I picked up a 4 foot long outdoor antenna box and looked at the packaging - “Can be used in your attic”.  Hey!  Awesome, I hadn’t thought of that!  I can put this huge freaking thing in my attic and run a wire down 2 stories to my basement….somehow.

So I bought it at $60 - and a 100 foot roll of coax cabling.

I got it home, brought it into my attic, started unfurling it - and I won’t lie - its very unweildly.  Picture me, in an attic with no floor, standing on beams trying to shift around a 5 foot wide, 4 foot long antenna.  I dropped it in place eventually - didn’t bother bracketing it to anything, and connected the coax cable, and searched for a way to get it out of the attic.

And look….a vent, with a screen over it thats slightly unhinged at the bottom.  Perfect place to drop cable out.  So the cable was unreeled, and unreeled, and unreeled.  I brought it in around the window entrance for the cats into the basement and over to the TV.

At long last, I have all the channels I need!  CBS included, so now it’s just a matter of returning the indoor antenna to Best Buy and setting things back up on Windows Media Center.   Media Center tells me I have 21 channels - most don’t come in, but as I look up the call letters on the internet, I find that those are pretty far away like in Wilmington, Winston Salem, Greensboro, etc.

I settle on 8 channels - WUNC (PBS), WRAL (CBS), WTVD (ABC), WNCN (NBC), WRDC (CW), WUVC (Univision), and WRAZ (Fox).

Now the tricky part….for the short time I’m actually paying for cable still, I want to make use of it, and Windows Media Center (as far as I know) won’t let you have both over the air and cable channels together set up at the same time.  So, I’m installing Beyond TV as I write this (a different DVR software solution), and I plan to use that to record cable, and Media Center to record over the air.  But that’s just till we cancel cable.  I mean - I’ll record Battlestar as long as I can until I have to go to SciFi.com (or someone elses house) to watch it.

But at last count - the Daily Show and the Colbert Report are free at Comedy Central.com.  Monk and Psych are free at Hulu.com, my wife watches Ghost Hunters and both of us Battlestar which is free at Scifi.com.  I just can’t think of any other cable shows we watch really.  And if miss something, we’ll wait and get it on Netflix.  And hopefully I’ll get an invite for Boxee TV on Windows (please please please) since I filled out the request form on their website last week.

So now, I’m just waiting for Beyond TV to install, and then go hit a superbowl party even though I hate football.

In my part one of this post, I took a look at how digital video and audio recorders and portable devices have changed the way I consume and perceive media.  What I’m most interested in is music - and I discussed that because of the ability to hold all of my music library in one place and the ability to get music from many sources free on demand, I’ve started listening to music on a very superficial level, and don’t give albums concurrent repeated listens.

If I have this problem - others probably do to, and may even be perfectly happy to be oblivious to this.

This is also a problem for musicians.  If people give music only superficial listens, and are just as happy to move onto the next song by a different artist, why would a musician have fans that bother coming to shows, buy t-shirts, or buy follow-up works by that musician?  Isn’t it easier just to turn on an internet stream and listen on shuffle than to seek out and buy an album or see who’s playing in your town on Friday night?  It’s as though being a fan is becoming a lot less fanatical.

My concern over this is that music will be written to appreciate on a superficial level - will only make you nod your head to a beat.  Of course it’s already been happening in pop music for years.  Musicians will only get their “radio friendly” hit song played wide-spread.  But many times, this radio-friendly song will be the gateway drug that leads you into the album, and get you to repeatedly listen to all songs by the artist.  Other songs on the album could provide deeper enjoyment.

Whether this problem is new and exasperated by our digital devices, or an old problem but just becoming apparent to me as I change my listening habits, I have to wonder what we can do to change these habits and produce ways to make people want to listen over and over again and come to a deeper appreciation for the music and the artist.  Listening to music over and over again can even make things more memorable.  How many times have listening to a song for the first time in years bring you back to old times when you were listening to that song constantly.  It can be very nostalgic.

I’ve been thinking a lot lately about music and video games (as you can infer from my blog posts).  I think that games like Dance Dance Revolution, Guitar Hero, Rock Band, etc do go quite a way in invoking repeated listening habits in music.  Pressing buttons in time to a beat, however, is just one direction to take this - I would call this a music creation game (even though you aren’t actually creating the music, there’s the illusion that you are).  I’m quite interested in exploring  a game where music is creating the game play environment that you’re in.

Another way to put this is I’d like to explore what games can be to music as what music videos are to music.  The best music videos, in my mind, produce another world where people break into singing and dance.  A mailbox will spring to life on the street and dance with Bjork.  Christopher Walken will dance his way down an escalator and fly through the air in grand choreographed moves.  Weezer will give a show to screaming members of the Happy Days cast.  You get my escapist point (and my Spike Jonz fixation).  I’m not a musical theater fan, though Gilbert and Sullivan have a lot of experience with this too, and they go a little farther back than MTV.  From what I understand, even the ancient Greeks had musical theater.

Bringing it back to games - a lot of great work has been put into cinematic gaming soundtracks, but to my knowledge, they are just soundtracks and don’t really change what happens in the game.  A change in tempo doesn’t make more bad guys come out to hurt you, a minor chord struck doesn’t usually signify coming doom.  I shouldn’t say never.  I’m not an avid gamer - I just hadn’t ever heard of this attention to music detail.  In fact - it’s quite the opposite - more bad guys coming out will change the tempo of the music - and coming doom, will change  the music to a minor key.  Game composers will typically write tiny segments that can be switched depending on what happens in the gameplay.

So, how can we produce a truly musical game?  One that gives a level of escapism straight from Broadway and MTV, gives music a forum to be listened to again and again, and is fun?  One where the music isn’t changed for it’s gameplay, but where gameplay revolves around the music?

(Yah, I’ll answer that question next, or at least try to)

Tivo and iPods (or more generally a digital audio players and a digital video recorders)  have changed the way we consume media.  It’s a pretty obvious statement I know - but hear me out.   I want to talk about music, but let me first say something about tv/film.

I was a late comer to both Battlestar Galactica and 24.   I didn’t start watching either one until a couple seasons in.  I recorded a bunch of 24’s with my DVR, and bought a few seasons of Battlestar Galactica on DVD.  Either way, I wound up watching entire seasons of a show in a week or two in contrast to the six or nine months it usually takes to watch a season of a show.

Watching habits were changed from when the network demanded it to whenever I wanted to see it.  In this, you could almost argue that the story itself behind the show has changed.    There’s no more downtime between episodes to think and ponder, and be on the edge of your seat about what happens next.

In fact, my DVR, Netflix, and DVD collection have changed my attention span on shows when I do have to wait to access them.  When I was watching my 24 marathon, I was really into it.  Then, when I had to wait a week to see what happened, I was less into it.  And now, I’m not even sure how much I care about 24 now that its been a whole year.

This is stark contrast to how we used to watch TV.  Either we were chained to our sets when our favorite shows were on, or our favorite shows just happened to be those that we had the specific nights free for.

The point is that having constant, on demand access to TV and film can ultimately change the story as we percieve it.  Watching TV can become more like reading a book.  You can rewatch/reread, read/watch when you want, and you don’t have to read/watch the whole chapter/episode in a sitting, or you can finish the entire thing in one sitting.  This is especially true as we can watch video on our mobile devices now.

Music can be a little more complex.  In addition to being short form, where consumption of a song can take only 3 or 5 minutes, there’s not usually a cliff-hanger or a compelling reason to listen to the next song on the same album for most people.

This became apparent when your single disc CD player became a 5 disc CD changer back in the 90’s.  I found it was easier to put my 5 discs on shuffle to get a little variety in each music listening session.

My car was a major place that I’d listen to music, and it was especially great to have my music on long rides.  My car, however, had just a single disc CD player.  That means when I put a CD in, it would stay in.  And because I was driving it, was a bit complicated to remove and change it unless there was a break in traffic.  I also only had a limited selection to choose from, as I’d usually only take the last 5 or 6 CD’s I bought with me on a normal car trip.

This means when I bought new music, I’d get very familiar with it.  I’d listen to it over and over again.

I suspect that most people’s music habits had the same limiting factors.  Either you listened to what the technology at the time allowed you to hear, or what the radio allowed you to hear - and each of these things led to the same thing, you’d listen to a song over and over again.

Of course with digital audio players with lots of storage this has all changed.  I have a Microsoft Zune with 120 Gigabytes of storage capacity.  This means that I can put my wife’s music collection and my music collection (probably 500-600 CD’s) on my Zune in its entirety 3 or 4 times over.  Many music players have at least 1 Gigabyte of memory allowing 10-15 albums at a time.

Suddenly, it becomes more convenient just to put every song you own on shuffle than to find the last albums you bought to listen over and over again - and given people’s tendency towards variety, a big shuffled list sounds more appealing when it comes time to listen to music.

So, listening habits have changed - and so has how we percieve music as a result.

Have you ever listened to an album or song, and didn’t really like it at first, but maybe after listening to it for the 5th time or the 10th time it’s one of your favorites?  It used to happen to me all the time.  But it really doesn’t happen to me much anymore.  Music listening has become sort of superficial to me.  I’d buy some new music, listen to it once, and then it appears on shuffle sometimes as I’m buying newer music.

It’s somewhat of a personal goal to give my new music a better listen from now one, despite the fact that my entire collection fits in the palm of my hand.

So as things are getting long, I’m going to continue things in a second post where I think about how people can listen to music on a less superficial level despite the technology we have.

It’s been a busy past month just getting a new job and a new car - I felt like I didn’t have time for anything.  Well, it just the Friday after Thanksgiving today, and I feel like I have all the time in the world this coming weekend, so I’m starting to get excited about my latest pet project.

I’ve been busy early this fall with writing “reactive music games”.  If you read my blog, which you probably don’t, you’d know that I couldn’t shut up about music games all summer.  I even did a little research into the different types.  The reactive game, is where music affects the things that happens in your game environment - but you don’t have to play along to (like Guitar Hero or Rockband).

There’s actually no popular music games I can think of that ARE reactive.  But I like to think of reactive games really…. like an interactive music video.  I even watched a bunch of Spike Jonz videos before I started sketching.  I have 10-15 games to do in the short term, and most are designed to be pretty easy to develop and generic - ESPECIALLY since this summer I’ve been working on min-framework in Flash AS3 that listens to beats in music.

The first game, I’ve done is basically an alpha, and if the music doesn’t load, just hit refresh (sometimes my security sandbox seems to go a little wacky).  I’ve codenamed it “Popfly” in my sketches, cause the balls kinda pop and fly out with the beats.

I’ve tried to pay special attention to things that I’ve noticed I didn’t like in other games.  Specifically, trying to tie visual events to music better.  Balls pop out, and when they reach their peak, its not very tied to music because it’s a split second after the beat.  This is why I put a subtle glow burst at the bottom when the balls do pop out.  I also gave the side gutters some ambient glow - just to have something additional on-screen that’s tied to the music in the game.  I also made some bigger balls pop up when the volume gets high - and it nicely adds a little crazy mosh effect when the music goes crazy.

I can’t wait to get through some more games, and see where this takes me, and hopefully way down the line, do a nice in-depth game with some characters.

But for now - here’s the PopFly game

Or should I say….BIZKIDZ.COM!  The last site I worked on at 360KID finally went live (and out of beta even)!  Its been a long time coming, I think we started a year and a couple months ago?  I’ve since moved on and taken a new position at a video search company called Digitalsmiths here in RTP.  The reasons are numerous, though 360KID is still an awesome company, but I just had to move on to new challenges and other people to learn from.

Anyway, where was I?  So BIZKIDZ.COM is a site for kids to go and set up a storefront, and sell actual product from real online retailers.  Lots of retailers in fact, at least like 400 of em last I checked.  The idea was to create a virtual lemonade stand of sorts.  As a child with an account, you’d populate your store with products and decorations, and send links to family and friends.  If they buy anything through the store, the account holder gets commission!

It’s pretty cool - I ended up doing most of the front end.  We got some help from a great contractor - Clint Little, and J2 Interactive did our server side architecture.  What was great was that I got to touch every part of the project - I did 99% of the Flex (sorry Clint, but those initial designs we had you work on, I had to do them over when the client changed their mind, oh well….great job though!), and maybe 5% of the server side stuff, and most of the HTML.  And of course our talented art team at 360KID did some wonderful designs and illustrations.

I got a lot of Flex training on this, how to make a Flex application NOT look like a Flex application - all in all I think it was a job well done - and if I know BIZKIDZ, they’ll be continuing developing their site with 360KID to make it even better than we imagined at the beginning.

I’ve been playing a bunch with physics engines for the last few weeks.  Physics engines basically serve to give your Flash Sprites some mass and body.  Your physics world can update your graphics so they can bounce off each other, bounce off walls, fall to the forces of gravity, and succumb to other forces all in a semi-realistic manner.

My exploration first took me from Papervision3D to WOWEngine.  Papervision3D is, of course, the popular realtime 3D engine for Flash.  It doesn’t have anything to do with phyics - but WOWEngine seems to be the hot dicussion for bringing physics into your 3D world.

WOW works like this….

Imagine a 3D world that you never see.  Yep, thats it.  That’s what the WOW does.  It’s the 3D world that never gets visualized in any way shape or form.  Of course, the 3D world in question has complete information about all objects in the world and how they interact with their surroundings and forces surrounding them.

It’s actually a nice little setup - you get to keep your Papervision3D or regular 2D world separate from your physics world.  So you have 2 different worlds - each operating independently of each other.  The way you tie them together is with a single handler ENTER_FRAME handler where you cycle through all the physics bodies and update your display with the proper x, y, z, or rotational properties.  The 2 worlds make it easy because you could use any 3D engine, 2D engine, or whatever engine you want and glue them together - there’s no dependencies, no skinning, no nothing - so it’s pretty handy, and a quick concept to grasp.  2 worlds - gotcha!

The problem is that the WOWEngine isn’t that great of a physics engine (yet!).  Another unfortunate problem is that its the only Actionscript Physics engine that handles 3D.  So if you use 3D, you’re kind of stuck using this or writing your own.

WOWEngine is based off of another engine called APE (Actionscript Physics Engine).  APE is a 2D physics engine that seems to actually be pretty good.  The problem with WOW, is that it has such potential, and works pretty well, but hasn’t implemented the richer features of APE yet.  The author definitely plans to, it’s on the roadmap, but I’m impatient need it now!

The killer feature that makes a great physics engine in my opinion is Rigid Body Dynamics - which APE has and WOW lacks.  To explain rigid body dynamics a little bit, let me tell you a little bit about my initial experimentation.

My big “hello world” application was basically making a few balls drop from the sky and hit the ground.  I initially tested in 2D, and just ignored the Z axis.  So we’re talking normal Flash Sprites here.  So the balls hit the ground, bounced off of each other, all pretty cool - and what you’d expect from a physics engine.

Next I tried the same thing with squares.  Still cool, but not quite right.  That’s because I still had to treat each square as a generic ball particle.  With no Rigid Body Dynamics - I couldn’t build a little fortress with my cubes!  That’s because without being able to define a polygon or even a cube, whatever graphics you have…..just sort of act like balls and slide off each other.

So that’s the my big problem.  Someday WOW will be awesome, but not yet for my purposes.

Next I checked out Foam.  Foam is a 2D Physics engine for Actionscript 3.  Foam is actually pretty sweet!  I didn’t spend long on it, though.  The reason is that after I experimented with WOW, I didn’t really like how Foam worked.  It seemed to handle Physics well - but it seemed to integrate the graphics and physics world too much for me.  I think I preferred them seperate - especially in a situation where maybe I want to use a different graphics engine.  I didn’t get far into Foam, but it seemed more like that Foam would take over your stage and you’d skin the physical bodies.  Before I went into this physics thing, it’s how I expected things to work - but I just couldn’t dig in after playing with WOW.

So again I moved on…

Finally I hit Box2DAS3.  This is the one I settled on - it has it’s problems, but if you can overcome it, then I do think it’s the best.  Box2DAS3  is a Actionscript 3 port of Box 2D.  Box 2D is an opensource physics package for C++.  In fact, Box2D has evolved to become Bullet - a 3D version of the physics engine.  Hopefully we’ll see Bullet 3D someday soon.

Box2DAS3 supports all the great features a good physics engine should….well OK OK, I’m not an expert - it supports all the great features I *think* a good physics engine should.  It also follows my beloved 2 worlds philosophy that I liked from WOW/APE.  For me it’s perfect….

….well almost perfect.  I think it has one flaw.  The flaw is that it’s a C++ port, and not really specific to Flash.  If I went back in time and warned myself of this, I would’ve been pretty dismissive and said “So what?  Code is code!”.

Well first of all it seems like classes have the bare minimum of comments - so you just kind of get the picture what everything does.  And it’s really not clear what each class or package does - unless you know….um…how to write a physics engine.  So it’s a little bit of a catch-22.  Like a simple body is name b2Body.  To create a body you need to create a shape definition (b2ShapeDef), convert that to a shape, and then convert the shape to a body, and then add the body to the world.  And it seems there’s only one way to add a body to a world, and that’s through a reference to the world, even though when you create a body from scratch, there’s a world parameter - but it doesn’t seem to work.  Well, I’m rambling.  Needless to say it’s confusing for a newbie - and on top of that there are things like manifolds and AABB - all things I don’t understand but there to get in the way and hamper learning.

But, if you study the examples they give you - you start to get a feel for how things work.  And once I did that - I stumbled a little, especially on design patterns - but got things working pretty well.  My design pattern quandry was that the framework was very incompatible with an inheritance based design - it was really going against the grain when I tried to use inheritance in my design.  What you really need to realize, is that a Factory type of design pattern works much better.

I ended up creating a base class like you would in Papervision3D.  The base class extends the stage and creates a common physics world setup as part of that stage.  Your main class would inherit this “physical stage” and contain the main stuff you want to program (hopefully keeping references to b2bodies and b2worlds to minumum, just cause it’s hard to understand!).

On the side - I have a physical object factory static class.  Here’s I’d call createPhysicalBall or something like that, and poof, it’d create me a ball….passing back seperate references for the graphics and the phyiscal body.

So that’s how I roll with the physics.  It adds some great realism to the boring old games.

So it would appear that visual ambiance in music games can be pretty important.  Take most rhythm games…

Background colors change, flashes appear, etc, all in time to the music in games like Guitar Hero and Rockband.  In PaRappa, characters dance and jerk their bodies in time.  What you end up getting is more music video like.

Rhythm games, those games where you have to tap your fingers, feet, or guitar to the beat as part of gameplay can feel more music video like because you end up “playing” the music to some effect.

Reactive games on the other hand, may have to rely on visual ambiance more.  Reactive games, are more often than not, played like any normal game.  The environment in the game may react to music, but you don’t.  So if you’re not required to react to the music, the music can start to take a back seat to the game play, and it starts feeling less choreographed.

This is why visual ambiance can become more important, and probably should be in your face more in a reactive game.  It would seem that when you don’t control the choreography of the game, other musical visuals need to step up to help out and make the game FEEL musical.

I used the example before of AudioSurf (published by Valve).  I crticized it for not tying the visual to the music as well as it could have.  I can’t criticize it for attempting ambiance to tie the music into the game better, because it does an adequate job of this with colors and background visuals.  However, the environmental reactions to the beats still seem too out of touch with the music, and the ambiance is all it has going for it to make it feel choreographed.

So, the closer you can tie in your reactive environmentals the better.  But like I said, even that might not be enough, as your game starts focusing on the goals of the game - which is why visual ambiance becomes a VERY important silent partner.