Changes

So a lot has happened since the last blog. I stumbled onto to some opportunities, one professional, and one personal, and then stumbled right back out of both of them. While they were both quite alluring, and I tried my hardest to make the best of them, I have to admit I felt some relief with they both fell through. The older I get, the more wary I am of big changes.

But they were both interesting and challenging, and the professional one got me pouring over my CLR(S), my operating systems book, and topcoder.com. (That last one should be a good hint ;) ). It was nice to go back and dive into the basics. To overview graph algorithms. To read Ingo Molnar's paper on everything a programmer should know about memory. I even flipped through a few pages of Knuth. And in the end, got tripped up on a pointer accounting question. It was a fine time, and forced me to focus on what I wanted to do next. All the algorithms work made me think about where my limits are, and what really gets me excited.

So, I've started a new phase in the project, with a little different emphasis. The inevitable "dogfood" phase of a software project, where you just get sick of it, and say I want something that I can use! Or more aptly, I want something I could show off to someone, if anyone was even conceivably interested in my project! So I put aside all that stuff about dynamically generating function calls at runtime (since I do have something for making remote requests that currently works, unlike the front end of the project). And decided to focus on two other things:

-Autoconf'ing the source tree, in order to, if nothing else, just be the same as everybody else. You never really want to make building your program a "unique" experience.

-Focusing more effort on the front end, getting more features in the renderer, and tying together all the separate lsystem, physics, and graphics components

I'm very happy to say the autoconf'ing part of the goals are done. Yes, I'm happy that it's all vanilla now. But, no, I didn't have oodles of fun learning all about M4. Still, I like the results, and now that I'm done, they are what matter.

Now the fun part - where I get to see stuff crawling around on the screen (finally), and breeding, and interacting, and... - I'm happy to say is not done, but does have some neat features working. This is always one of the best parts of software, you finally got some features that'll put a smile on your face because they actually work! And you still have a future to be excited about, and features to look forward too.

So about the obligatory screenshots..

The first one (at the top of the article) has no physics enabled, mainly for the sake of generating a pretty screenshot. It shows three trees, all generated from two files. The first containing rules that outline an l-system, it's growth rules, mutation rates, etc:

maxSize 100
minSize 10
options L,R,S
system Y
#mutation rules key
#1)the entry is swapped with one of it's neighbors
#2)the entry is toggled to one of it's possible options
#3)the entry is deleted
#4)the entry is doubled
#it is interesting to note that mutations that occur on
#already interpreted elements are essentially irrelevant during growth
#(but not breeding..)
#probability is in 1/10000th
mutationRules (1,3000) (2,2000) (3,3000) (4,100)
growthRules Y:S,S,BL,BR,S,S,BU,SS,BD,SS.BL:[,L,S,S,Y,].BR:[,R,S,S,Y,].BU:[,U,S,S,Y,].BD:[,D,S,S,Y,]
numSteps 0
numSwaps 0
swappedLeft 0
swappedRight 0
numToggled 0
numDeleted 0
numDoubled 0
currentGene 0

The second file provides information on how the system should be interpreted:

LeftAngle -40
RightAngle 50
UpAngle 30
DownAngle -30
Mesh sphere.mesh
StartDirection 0,1,0.1
AtomicSize 10

These are both read in at start-up, and the first can be a saved snapshot of the lsystem state at any point in time (not, however, the fully interpreted state, that would be saved via a different component). Keep in mind, too, that all the trees (and the bouncing blobs of balls below), are growing over time. About one new ball pops out of a branch every second or so, up to the max size of the system.

Here is one with cubes, which makes rendering and getting the screenshot easier on the integrated graphics of my laptop:


And finally, here's one with physics. It much harder to get this across with a static image, but I haven't figured out how to make a movie yet..
All the balls in each system are joined to their previous ball via a ball joint, and if I could make a movie, you would see all four systems bounce away from each other, loosely staying together, but each arm swiveling here and there as they move away... It's kinda a neat, but will be even better with hinge joints, and best of all.. motors!


On top of all this, I've gotten lasik - totally awesome by the way - I hadn't been able to wear contacts for a year, and suddenly - 20/15 with no glasses!. Plus, they do the whole thing with lasers now, no microkeratome (little knife) to cut the flap. I'm a Tiger! Well, maybe not that far..

I also took a trip out east, and wandered around Detroit, DC, and New York, seeing the sites, and meeting up with old friends. It was really wonderful. It's a quite journey from West LA out to the snow and chill, but friends make it worth it. And after a few years of being single, a girl asked me if I wanted to date. So, while it all started off with some stumbling, I've manged to gradually stumble in the right direction at least. One little change at a time.

PS..
So right after abandoning the dynamic function bit back in Dec/Jan, I finally stumbled across this: libffi. When I found it, it seemed pretty much abandoned - the page hadn't been updated in ten years, no documentation, just an old tarball.

Now, granted, a little digging revealed it's largely maintained by gcc now, and they just updated the site a few days ago (sparking my interest in the whole thing again). However, if you think I'm bad at updating my site, and uploading tarballs - Try no updates for ten years.

Never say never

Well, it's been a while since my last post. Actually, I moved sometime in July, so it's been a while since I had internet at home, let alone posted anything. And I went to Usenix in Boston last month, and visited a friend in Colorado last week. Soo... anyways, I did manage to sneak in one upload of code a week or so ago, and ended up doing exactly what I said wouldn't happen in the last blog post, namely:

"Sooo, yah, dynamically creating functions - ain't gonna happen."

Now, granted I gave myself one little loophole:

"Unless you want to drop into asm and self-modifying code."

So, basically, I decided I really did want to get something like reflection in C++, to avoid the inevitable:

switch( request )
{
case 1:
function 1
case 2:
funciton 2
....
}

So, what the heck, I thought, I'll try writing some assembly to generate my function calls at runtime. To make a long story short, you create a buffer of machine code, basically your standard "trampoline" code, but on the heap, and call the buffer. The downside is the code needs to be ported if you want to run it on, say, a Mac. The upside is the code is relatively short (should be easy to port). Also, C is flexible enough that the transition from 32 bit -> 64 bit should be a matter of fixing bugs, not rewriting code. Which is good, because I'm writing this blog on my new 64 bit laptop :).

The real downside to this code is it can be a pain to debug. Luckily, I've worked with enough asm at this point, so when the debugger just gives up and spits out an address, it actually is giving me useful information. And I thought those asm/computer architecture classes I took so long ago would have no practical applications. Now I use them at work and for play. C'est la vie.

My first rough draft of dynamically generated function calls is up now. Still needs work to round it out, and I still need a way to generate metadata about my classes, so clients can generate requests appropriately (c++2xml?). Anywho, that's it for now.

cheers!

Reflections

Well, not a whole lot of coding happened this week. But I did see Tchaikovsky with fireworks Tues night. And the cops did show up at my door at 2am on Thurs, banged on it for awhile, and then wouldn't tell me why when I asked. *sigh*. LAPD. But the fireworks were really nice, and I hope to get pictures from one of my co-workers. That'll be my screenshot for this week :)

Anyways, I finally realized what I really needed to expose my objects interfaces was reflection. At first I had the fuzzy idea that maybe I could just map ids -> function pointers, but once I refreshed myself on function pointers in C++, I realized it just can't happen. Because you need something like this:

map<string,funcPointer*>

Where funcPointer can point at whatever function you want. Like Python :) But C++ makes you declare the args along your function pointer. At first I was like "Oh, there's probably a work around". Usually C++ has some sort of way of letting do what you want, if you really want to do it. I mean, just the keyword "friend" lets you violate encapsulation in all kinds of interesting ways. But then I ran across an erroneous statement somewhere along the lines of "function pointers aren't like normal pointers, they hold some extra information". My realization came in three states:

• Oh!, ok...


• Wait, no, bullocks!


• Hmm.. ok they're wrong, but I know what they mean, or a least what they repeated incorrectly

The Oh!, ok.. speaks for itself. The "Wait, no... bollocks!" comes from the fact that I look at a lot of disassembly in my line of work, and I realized that no, a function pointer is just an address, like any other pointer. A function call using a pointer is just going to look something like this:

call [address]

And that can even be broken into:

push eip+5
jump [address]

(jump is one byte, address is four, this is on a x86).
Hmm. So what's this business about "A function pointer has extra info"? Well, what they should have said is a function pointer declaration has extra info, namely, what arguments to push on the stack. This information is used by the compiler, and has nothing to do with the run-time encoding of function pointers. For a compiled language, figuring out what to push and pop on the stack pretty much has to happen at compile time. Unless you want to drop into asm and self-modifying code. Sooo, yah, dynamically creating functions - ain't gonna happen. RTTI is _not_ going to somehow magically create:

push c
push b
push a
call [address]

Just because it likes you.
In a statically compiled language, the only way you could do that is to have your arguments on the heap, not the stack. Oh well.... Or not?

Well, maybe there is a little wiggle room with that whole heap business. We could expose a function like:

invoke( string funcSpecification, void* args, void* reply )

Of course, now we have the whole messy business of, badabum:

• A means to specify and expose functions


• A standard way to parse and create those "void* args, void* reply" arguments

In other words, the same questions as my last post, but further along ;) Or Progress! Yes, the nagging first item: you do something highly structured and and formal like xml (WSDL), and it's just a bitch to specify everything, and send it across the network. But you have a nice standard parser, and a clear structure. You do something loose and custom, and you have wonky stuff like the IDL that just invents a new language, and not necessarily a good one.. So I think I'm going to start off having the objects expose something wonky like:

vector reflection;
reflection.push_back("getData( string, int )=myDataType")

to a library that gets loaded by the server. This library can load and generically manage any object that has the method vector getReflection() and invoke( string id, void* args, void* reply), and a little parsing logic... Clients will get something nicer and more structured from the server. Possibly even something that can be used to generate code. So not much different than the current setup. With one big difference: (1) that's _one_ library can manage any object with the right interface, instead of having a different library wrap every object. Yuck.

So what about that second point?

• A standard way to parse and create those args,reply arguments

Why my friends. That's just serialization! :)

One final reflection: an interesting effect of coding late at night, after you get home from work, is that it leads to things like this:

template<class T>
    bool serialize( T data, char*& state, unsigned int& size )
    {
        if( typeid( T ).name() == typeid( string ).name() )
        {
            //go ahead and give it to'm
            cout << "String!" << endl;
            return serializeString( data, state, size );
        }
        if( sizeof(T) > INT_MAX )
        {
            cout << "Data type way too big" << endl;
            return false;
        }
        state = new char[sizeof(T)];
        size = sizeof(T);
    
        //this is kind of silly, but a byte isn't always 8 bits,
        //so we can't just say tmp & 0xFF below
        //I really can't believe I bothered to handle this case.
        //However, this loop _is_ worth it just to think of the C coders screaming bloody murder
        //and pounding their heads in the desk
        char mask = 0x0; //a char is always 1 byte though
        for( int i = 0; i < CHAR_BIT; ++i )
        {
            mask = (mask << 1) | 0x01; 
        }
        //index needs to handle negative so we can count down, no unsigned int
        for( int i = sizeof(T) - 1; i >= 0; --i )
        {
            T tmp = data >> i * CHAR_BIT; 
            state[i] = tmp & mask;         
        }
    }

Then you come home the next night and realize that what you really meant was:

   template<class T>
   bool serialize( T data, char*& state, unsigned int& size )
   {
       size = sizeof(T);
       state = new char[size];
       memcpy( state, (void*)&data, size );
   }

Nothing like a good nights sleep!

Cheers!

PS. Blogger sucks at trying to put code up. Try Ctrl-Shift-+.

Smokin'

The smoke test is back, and using the ogre renderer. But the important thing is all the components are working together again. So now it's time to break stuff!

There's two major things I'm looking at:

- Simplify message passing.

I plan to do this by implementing object serialization for my hierarchy. I have to be careful, but I think I know how to do this. It might take some time, but it shouldn't be too bad.

- Exposing the interfaces of server-managed objects.

This is harder, and I'm just looking at mechanism right now, not even policy. The current deal is every method matches up with a command in the Commands object. Commands are specified in an xml file, and code is generated from them. Each command requests certain data, and has an expected response. Requests and responses are specified in xml files, and client interface code is generated from them. On the server, each object is wrapped in a shared library that can be loaded by the server depending on what behavior you want the server to take on. Each shared libary exposes the same interface to the server object - handleCommand( id, command ) I WANT SOMETHING SIMPLE! Oy. Or at least more transparent on the client end, and less work on the server end. Yes, I know this is an old problem with corba, rmi, soap, etc, littering the path before me. So this problem is harder :|

But that's what makes it fun! :)

Screenshots...
Unfortunately, I couldn't dig up some of the neater patterns I used to have, but they do demonstrate 3D hierarchal lsystems. The second has breeding turned on, so it looks a little more messed up.

Bouncing Ninjas!

So after trying OpenGL + ODE, Irrlicht + ODE, I've finally settled on OGRE + ODE + ? for the client rendering/smoke tests.

A little history.
A little while ago, after getting the lsystem, system server, interpreter, interpreter server, and my OpenGL + ODE client all displaying lsystems that were interpreted as 3D points in space, and could be evolved (via mutations and cross-breeding), I began to run up against the limits of how much time I could put into OpenGL. So I began looking for a 3D toolkit I was happy with. OGRE is it. However, now I need to find an input system I'm happy with. OIS is not it, and SDL may be it, but the OGRE codebase has drifted ever more and accomodating OIS/CEGUI, so that's what I'm using for now. I had SDL integrated in, but pulled it.

But for now, I'm eager to get back to fixing all the bugs I've found in the other components: subtle issues with mutexes, simplifying message passing (I'm not scared of hex anymore, so I'm going to make the objects serialize themselves instead of the current mess). And I'll just be glad to see those 3D l-systems again (especially coz I can't find any screenshots to prove I did it). For now, here's some screenshots of bouncing ninjas, and 2D systems generated from a codebase that has since been pretty much re-written.

Enjoy!

Ninjas:


Trying to see how big I could make the system (something like 100K elements/spheres in this one)

Detail:

Systems with the same start state, but one has a mutation rate > 1.

Playing around with different settings:

Bouncing Ninjas Refactored!

Intro

Hello, this is my blog about my project. I noticed I kept updating the intro page, and realized one day I'd want screenshots, musings and such. So I turned to this.
Maybe I'll even get around to sketching out the design in something other than code one day, and post it ;).
For now, the current status:

Cons

• Currently broken only has some working components (servers, lsystem, some physics).And bouncing ninjas!


• Undergoing a lot of rework.


• At the very least depends on Boost, ODE.


• Client will depend on OGRE.


• Client may end up depending on GDK/SDL, but currently trying OIS instead of SDL.


• I currently don't like OIS, but I need to focus on other things.


• SDL code written but pulled out temporarily.


• Currently only works on linux. I don't have windows to do the port (but I keep it in mind).


• No Design/Overview documentation.


• Some of the design kind of sucks.

Pros

• Has had all the parts working together in the past.


• Some of the design is kind of nice.


• platform dependencies are wrapped/abstracted.


• Build system is simple and straightforward. No automake/m4 magic.


• I always kept it moving forward, albeit often slowly, with long pauses, and many rewrites, over the last few years.