Demo here, view source for code

(view source for code, and who knows what bugs there are)

I’m getting into unfamiliar territory these days with exploring sound in Flash.

Flash 10 gives you the ability to take a sound object, and extract the entire thing into the raw data….a byte array.  How do you interpret this data? How can you process it into something meaningful?  Well I don’t know.

What I do know is that to get a firmer grasp on how I can utilize this, I need to understand what the sound object is at a basic level.  Probably the worst way to understand it is to take an entire song (mp3) file and let my eyes glaze over at the stream of numbers coming from the sound.extract feature.

No - the best way, I thought, would be to compose my own notes and chords, and work up from there.  To do this, I needed to understand what a note is, and what a sound is at a basic level.

My first thought was to picture sound as the nifty looking sound visualization that comes with music players these days…you know, those dancing bars:

Turns out that this is the worst way to imagine sound for this purpose.  My first question was….”OK, how do I get the value of the low frequency?”, “how about something in the midrange?”, “what about the highend?”.

Yes you can imagine sound this way, but only after processing your sound data with a Fourier Transform.  If you know how to do this, stop reading this right now, cause you’re way smarter than I am at this point in my sound exploration.

The BEST way to imagine sound is to picture a sine wave:

If you look at how tall the wave is, that is how loud the sound is, or the amplitude.   How close the peaks and valleys are in this picture is the frequency.

Frequency, at this very basic level can’t be thought of as high pitched, low pitched, etc - it’s only how far apart these peaks and valleys are.  Now think of this as a side-scrolling wave that goes on forever.  If you were to scroll at a constant speed, each peak would hit that vertical center line at a constant rate.  This rate would be the frequency.  If it goes faster, the frequency is higher - slower, the frequency is lower.  And, of course, this directly relates to how you hear it.  The more peaks and valleys at a given time, the higher the tone you hear.

Now think about how this relates to our Flash sound object’s byte array.  We could actually draw a sine wave with numbers.  This is nothing new to programmers.  Folks use trigonometry all the time.  But, I personally, never thought to use it for sound.

Try this AS3 code:

for (var i:int=0; i < 100; i++ ) {
var sample:Number;
sample = Math.sin(i * 2 * Math.PI) * 50;
}

This will basically plot a picture like the one above - a sinewave.

If we made this into a sound, it almost, but not quite be a tone.  However, if we pop over to google, we can actually look up the frequency, of say…..a middle C note, or a middle A.

http://www.phy.mtu.edu/~suits/notefreqs.html

There’s one more piece to this puzzle though, and that’s sampling rate.  A digital audio file could have all the sinewaves, peaks and valleys in the world, but if your audio playback is super slow, it’s gonna sound like garbage.  That’s why we tell our audio software to read our sound at a certain speed.

If you’ve heard the term 44.1 kbps when folks talk about sample rate - you can look back at our little for loop that drew the sine wave, and realize that you need more than the 100 points of data as we drew, you need 44.1k, or 44,100 PER SECOND.

So let’s rewrite that code:

for (var i:int=0; i < durationinseconds * 44100; i++ ) {
var sample:Number;
sample = Math.sin((i) * 2*Math.PI/44100 * 440) * volume;
}

OK!  So now, we’re creating a one second middle A (4th octave).  Our sample rate in Flash is always going to be 44100 if coming from a flash sound object (though that’s not true for any audio file).  We learned from our handy/note frequency chart when looking on google that 440hz is a middle A.

As a side note, lets say we want a different octave.  To go lower, half the frequency to get 220.  To go lower, half that.  Higher?  Double it.

Let’s think about the voice now.  The nice sine wave will give a nice, round tone.  For a dirtier tone that sounds like it has sharp edges….well our sine wave needs to have sharp edges - which is actually a square wave.  Our nice round peaks and valleys would be just straight corners.

To change the tone in the code, try this:

sample = Math.sin((i) * 44100 * 2 * Math.PI * frequency) > 0 ? volume : -volume;

As for more voices, well, I haven’t tried it, but a real world instrument would have a hard strike and then some falloff.  So our nice sine wave would be less round at the start of each peak, but comes back down way slower.

Here’s my final code (and keep in mind that I’m writing the sample 2 times, one for the left channel and one for the right):

returnBytes:ByteArray = new ByteArray();
for (var i:int=0; i < _duration * 44100; i++ ) {
var sample:Number;
if (_voice==VOICE_SQUARE) {
sample = Math.sin((i) * 44100 * 2 * Math.PI * this.frequency) > 0 ? _amplitude : -_amplitude;
} else {
sample = Math.sin((i) * 44100 * 2 * Math.PI * this.frequency) * _amplitude;
}
returnBytes.writeFloat(sample);
returnBytes.writeFloat(sample);
}

Now, what do we do with that byte array?  Well, as of Flash 10, our sound object, has a sample data event.  When this event is called, when it needs new bytes, it’ll call out to your custom sample data method.

What I do - and what you may choose to do, or not choose to do, is make my whole byte array first, and then read sequential amounts of data into the byte array each time it’s called:

soundBytes.position = 0;
dynamicSound = new Sound();
dynamicSound.addEventListener(SampleDataEvent.SAMPLE_DATA, addSoundBytesToSound, false, 0, true);
soundchannel = dynamicSound.play();

private function addSoundBytesToSound(event:SampleDataEvent):void
{
var bytes:ByteArray = new ByteArray();
soundBytes.readBytes(bytes, 0, Math.min(soundBytes.bytesAvailable, 8 * 8192));
event.data.writeBytes(bytes, 0, bytes.length);
}

To explain, about the 8 * 8192….

8192 is the maximum amount of samples you can use for each sample data event.  However….each sample is a left and a right 4 byte float.  So that’s 8…..time 8192.

There’s tons of cool stuff you can do with this.  If you don’t believe me, look up Andre Michelle - he’s THE MAN when it comes to this stuff.

It’s been a busy past month just getting a new job and a new car - I felt like I didn’t have time for anything.  Well, it just the Friday after Thanksgiving today, and I feel like I have all the time in the world this coming weekend, so I’m starting to get excited about my latest pet project.

I’ve been busy early this fall with writing “reactive music games”.  If you read my blog, which you probably don’t, you’d know that I couldn’t shut up about music games all summer.  I even did a little research into the different types.  The reactive game, is where music affects the things that happens in your game environment - but you don’t have to play along to (like Guitar Hero or Rockband).

There’s actually no popular music games I can think of that ARE reactive.  But I like to think of reactive games really…. like an interactive music video.  I even watched a bunch of Spike Jonz videos before I started sketching.  I have 10-15 games to do in the short term, and most are designed to be pretty easy to develop and generic - ESPECIALLY since this summer I’ve been working on min-framework in Flash AS3 that listens to beats in music.

The first game, I’ve done is basically an alpha, and if the music doesn’t load, just hit refresh (sometimes my security sandbox seems to go a little wacky).  I’ve codenamed it “Popfly” in my sketches, cause the balls kinda pop and fly out with the beats.

I’ve tried to pay special attention to things that I’ve noticed I didn’t like in other games.  Specifically, trying to tie visual events to music better.  Balls pop out, and when they reach their peak, its not very tied to music because it’s a split second after the beat.  This is why I put a subtle glow burst at the bottom when the balls do pop out.  I also gave the side gutters some ambient glow - just to have something additional on-screen that’s tied to the music in the game.  I also made some bigger balls pop up when the volume gets high - and it nicely adds a little crazy mosh effect when the music goes crazy.

I can’t wait to get through some more games, and see where this takes me, and hopefully way down the line, do a nice in-depth game with some characters.

But for now - here’s the PopFly game

I’ve been playing a bunch with physics engines for the last few weeks.  Physics engines basically serve to give your Flash Sprites some mass and body.  Your physics world can update your graphics so they can bounce off each other, bounce off walls, fall to the forces of gravity, and succumb to other forces all in a semi-realistic manner.

My exploration first took me from Papervision3D to WOWEngine.  Papervision3D is, of course, the popular realtime 3D engine for Flash.  It doesn’t have anything to do with phyics - but WOWEngine seems to be the hot dicussion for bringing physics into your 3D world.

WOW works like this….

Imagine a 3D world that you never see.  Yep, thats it.  That’s what the WOW does.  It’s the 3D world that never gets visualized in any way shape or form.  Of course, the 3D world in question has complete information about all objects in the world and how they interact with their surroundings and forces surrounding them.

It’s actually a nice little setup - you get to keep your Papervision3D or regular 2D world separate from your physics world.  So you have 2 different worlds - each operating independently of each other.  The way you tie them together is with a single handler ENTER_FRAME handler where you cycle through all the physics bodies and update your display with the proper x, y, z, or rotational properties.  The 2 worlds make it easy because you could use any 3D engine, 2D engine, or whatever engine you want and glue them together - there’s no dependencies, no skinning, no nothing - so it’s pretty handy, and a quick concept to grasp.  2 worlds - gotcha!

The problem is that the WOWEngine isn’t that great of a physics engine (yet!).  Another unfortunate problem is that its the only Actionscript Physics engine that handles 3D.  So if you use 3D, you’re kind of stuck using this or writing your own.

WOWEngine is based off of another engine called APE (Actionscript Physics Engine).  APE is a 2D physics engine that seems to actually be pretty good.  The problem with WOW, is that it has such potential, and works pretty well, but hasn’t implemented the richer features of APE yet.  The author definitely plans to, it’s on the roadmap, but I’m impatient need it now!

The killer feature that makes a great physics engine in my opinion is Rigid Body Dynamics - which APE has and WOW lacks.  To explain rigid body dynamics a little bit, let me tell you a little bit about my initial experimentation.

My big “hello world” application was basically making a few balls drop from the sky and hit the ground.  I initially tested in 2D, and just ignored the Z axis.  So we’re talking normal Flash Sprites here.  So the balls hit the ground, bounced off of each other, all pretty cool - and what you’d expect from a physics engine.

Next I tried the same thing with squares.  Still cool, but not quite right.  That’s because I still had to treat each square as a generic ball particle.  With no Rigid Body Dynamics - I couldn’t build a little fortress with my cubes!  That’s because without being able to define a polygon or even a cube, whatever graphics you have…..just sort of act like balls and slide off each other.

So that’s the my big problem.  Someday WOW will be awesome, but not yet for my purposes.

Next I checked out Foam.  Foam is a 2D Physics engine for Actionscript 3.  Foam is actually pretty sweet!  I didn’t spend long on it, though.  The reason is that after I experimented with WOW, I didn’t really like how Foam worked.  It seemed to handle Physics well - but it seemed to integrate the graphics and physics world too much for me.  I think I preferred them seperate - especially in a situation where maybe I want to use a different graphics engine.  I didn’t get far into Foam, but it seemed more like that Foam would take over your stage and you’d skin the physical bodies.  Before I went into this physics thing, it’s how I expected things to work - but I just couldn’t dig in after playing with WOW.

So again I moved on…

Finally I hit Box2DAS3.  This is the one I settled on - it has it’s problems, but if you can overcome it, then I do think it’s the best.  Box2DAS3  is a Actionscript 3 port of Box 2D.  Box 2D is an opensource physics package for C++.  In fact, Box2D has evolved to become Bullet - a 3D version of the physics engine.  Hopefully we’ll see Bullet 3D someday soon.

Box2DAS3 supports all the great features a good physics engine should….well OK OK, I’m not an expert - it supports all the great features I *think* a good physics engine should.  It also follows my beloved 2 worlds philosophy that I liked from WOW/APE.  For me it’s perfect….

….well almost perfect.  I think it has one flaw.  The flaw is that it’s a C++ port, and not really specific to Flash.  If I went back in time and warned myself of this, I would’ve been pretty dismissive and said “So what?  Code is code!”.

Well first of all it seems like classes have the bare minimum of comments - so you just kind of get the picture what everything does.  And it’s really not clear what each class or package does - unless you know….um…how to write a physics engine.  So it’s a little bit of a catch-22.  Like a simple body is name b2Body.  To create a body you need to create a shape definition (b2ShapeDef), convert that to a shape, and then convert the shape to a body, and then add the body to the world.  And it seems there’s only one way to add a body to a world, and that’s through a reference to the world, even though when you create a body from scratch, there’s a world parameter - but it doesn’t seem to work.  Well, I’m rambling.  Needless to say it’s confusing for a newbie - and on top of that there are things like manifolds and AABB - all things I don’t understand but there to get in the way and hamper learning.

But, if you study the examples they give you - you start to get a feel for how things work.  And once I did that - I stumbled a little, especially on design patterns - but got things working pretty well.  My design pattern quandry was that the framework was very incompatible with an inheritance based design - it was really going against the grain when I tried to use inheritance in my design.  What you really need to realize, is that a Factory type of design pattern works much better.

I ended up creating a base class like you would in Papervision3D.  The base class extends the stage and creates a common physics world setup as part of that stage.  Your main class would inherit this “physical stage” and contain the main stuff you want to program (hopefully keeping references to b2bodies and b2worlds to minumum, just cause it’s hard to understand!).

On the side - I have a physical object factory static class.  Here’s I’d call createPhysicalBall or something like that, and poof, it’d create me a ball….passing back seperate references for the graphics and the phyiscal body.

So that’s how I roll with the physics.  It adds some great realism to the boring old games.

The 2008 Codie Awards have been announced.  Little did I know that a project I played a major part in was at least nominated for a couple Codies, we actually WON the award for “Best Instructional Solution for Students at Home”.  Pretty nifty, and its just the latest in a series of awards for the Pokemon Learning League.

So far the awards have been the Codie, a 2007 Outstanding Products Award from iParenting, a 2007 Best Educational Software Award, and we were a finalist for Distinguished Achievement from the Association of Educational Publishers.

I won’t go into the “making of” in the least, cause you know, it’s safe to assume that most clients don’t appreciate any ups and downs of working with them made available in a public forum.  But…..pretty nifty anyway right?

The project is:
http://www.pokemonlearningleague.com/

And of course, yes I played a good part in it, but there were plenty more people involved at the company I work for 360KID.

We just launched a new application at 360KID!  I wish I could say I did all of it, or even most of it, but I did step in to help a lot during the final stages - just so I can say I took all the credit from the project’s real programmer Clint Little.  I have a few Flex projects under my belt now, and they all look fantastic compared to most of the ones I’ve seen out there.  This one is no exception thanks to the great animation and illustration my company does.  I look forward to launching more and more Flex applications that have a fun look and feel.  We’re so close to launching our other projects, I’d love to show them since I was the main developer - but this one is another one to be proud of regardless of who did it.

This game called Budget Hero was developed for American Public Media, and allows you to play cards to adjust the federal budget and see if you can balance it.   Also allowing you to earn badges, you can see if you’re also doing your part to maintain the military, health care, etc.  It’s up to you to make sure the stuff you think is important doesn’t wither away and STILL maintain the budget.

Click to Play Budget Hero

I ‘ve been working with Flex 3 for a little while now, and started my first commercial Flex project in December. Luckily I got the CS3 Master Suite…whoo! So since it’s more fun to draw in Illustrator, that’s what I’ve been doing. In fact, on this latest project I’ve been doing, I’ve been having our artists deliver Illustrator files, then I just export SWF asset libraries. I take these files and embed them right into Flex! It’s pretty nifty, and I’ve been loving the workflow.

Well, the project is going full steam, and I’ve got a bug on my plate. I have some scrolling lists full of graphics. These lists are styled with a background image using 9 Slice Scaling. At first glance it’s fine, but I’ve been noticing some weird redraw issues in the project. It’s pretty subtle, but the bounding box of these scrolling images will discolor the background image. More specifically, if I have a colored square layered over another colored square as my Illustrator artwork, the bug will take away parts of the square in the forefront revealing whats behind it.

I tried so many different export options, and nothing worked! Finally I copied and pasted this same art into Flash CS3 to produce the same SWF asset library, and guess what? No problems.

There must be some buggy SWF producing code in Illustrator CS3. I hope I’m doing something wrong, but I can’t imagine what. But I guess I’ll use Flash when stuff breaks on me, oh well…..the Illustrator high was fun while it lasted.

To see what I mean, check out this Flex Application. The Flash list on the left scrolls fine, but check out the Illustrator list. If you scroll and look at the bottom, you can see the redraw issue.