Never say never

Well, it's been a while since my last post. Actually, I moved sometime in July, so it's been a while since I had internet at home, let alone posted anything. And I went to Usenix in Boston last month, and visited a friend in Colorado last week. Soo... anyways, I did manage to sneak in one upload of code a week or so ago, and ended up doing exactly what I said wouldn't happen in the last blog post, namely:

"Sooo, yah, dynamically creating functions - ain't gonna happen."

Now, granted I gave myself one little loophole:

"Unless you want to drop into asm and self-modifying code."

So, basically, I decided I really did want to get something like reflection in C++, to avoid the inevitable:

switch( request )
{
case 1:
function 1
case 2:
funciton 2
....
}

So, what the heck, I thought, I'll try writing some assembly to generate my function calls at runtime. To make a long story short, you create a buffer of machine code, basically your standard "trampoline" code, but on the heap, and call the buffer. The downside is the code needs to be ported if you want to run it on, say, a Mac. The upside is the code is relatively short (should be easy to port). Also, C is flexible enough that the transition from 32 bit -> 64 bit should be a matter of fixing bugs, not rewriting code. Which is good, because I'm writing this blog on my new 64 bit laptop :).

The real downside to this code is it can be a pain to debug. Luckily, I've worked with enough asm at this point, so when the debugger just gives up and spits out an address, it actually is giving me useful information. And I thought those asm/computer architecture classes I took so long ago would have no practical applications. Now I use them at work and for play. C'est la vie.

My first rough draft of dynamically generated function calls is up now. Still needs work to round it out, and I still need a way to generate metadata about my classes, so clients can generate requests appropriately (c++2xml?). Anywho, that's it for now.

cheers!

Reflections

Well, not a whole lot of coding happened this week. But I did see Tchaikovsky with fireworks Tues night. And the cops did show up at my door at 2am on Thurs, banged on it for awhile, and then wouldn't tell me why when I asked. *sigh*. LAPD. But the fireworks were really nice, and I hope to get pictures from one of my co-workers. That'll be my screenshot for this week :)

Anyways, I finally realized what I really needed to expose my objects interfaces was reflection. At first I had the fuzzy idea that maybe I could just map ids -> function pointers, but once I refreshed myself on function pointers in C++, I realized it just can't happen. Because you need something like this:

map<string,funcPointer*>

Where funcPointer can point at whatever function you want. Like Python :) But C++ makes you declare the args along your function pointer. At first I was like "Oh, there's probably a work around". Usually C++ has some sort of way of letting do what you want, if you really want to do it. I mean, just the keyword "friend" lets you violate encapsulation in all kinds of interesting ways. But then I ran across an erroneous statement somewhere along the lines of "function pointers aren't like normal pointers, they hold some extra information". My realization came in three states:

• Oh!, ok...


• Wait, no, bullocks!


• Hmm.. ok they're wrong, but I know what they mean, or a least what they repeated incorrectly

The Oh!, ok.. speaks for itself. The "Wait, no... bollocks!" comes from the fact that I look at a lot of disassembly in my line of work, and I realized that no, a function pointer is just an address, like any other pointer. A function call using a pointer is just going to look something like this:

call [address]

And that can even be broken into:

push eip+5
jump [address]

(jump is one byte, address is four, this is on a x86).
Hmm. So what's this business about "A function pointer has extra info"? Well, what they should have said is a function pointer declaration has extra info, namely, what arguments to push on the stack. This information is used by the compiler, and has nothing to do with the run-time encoding of function pointers. For a compiled language, figuring out what to push and pop on the stack pretty much has to happen at compile time. Unless you want to drop into asm and self-modifying code. Sooo, yah, dynamically creating functions - ain't gonna happen. RTTI is _not_ going to somehow magically create:

push c
push b
push a
call [address]

Just because it likes you.
In a statically compiled language, the only way you could do that is to have your arguments on the heap, not the stack. Oh well.... Or not?

Well, maybe there is a little wiggle room with that whole heap business. We could expose a function like:

invoke( string funcSpecification, void* args, void* reply )

Of course, now we have the whole messy business of, badabum:

• A means to specify and expose functions


• A standard way to parse and create those "void* args, void* reply" arguments

In other words, the same questions as my last post, but further along ;) Or Progress! Yes, the nagging first item: you do something highly structured and and formal like xml (WSDL), and it's just a bitch to specify everything, and send it across the network. But you have a nice standard parser, and a clear structure. You do something loose and custom, and you have wonky stuff like the IDL that just invents a new language, and not necessarily a good one.. So I think I'm going to start off having the objects expose something wonky like:

vector reflection;
reflection.push_back("getData( string, int )=myDataType")

to a library that gets loaded by the server. This library can load and generically manage any object that has the method vector getReflection() and invoke( string id, void* args, void* reply), and a little parsing logic... Clients will get something nicer and more structured from the server. Possibly even something that can be used to generate code. So not much different than the current setup. With one big difference: (1) that's _one_ library can manage any object with the right interface, instead of having a different library wrap every object. Yuck.

So what about that second point?

• A standard way to parse and create those args,reply arguments

Why my friends. That's just serialization! :)

One final reflection: an interesting effect of coding late at night, after you get home from work, is that it leads to things like this:

template<class T>
    bool serialize( T data, char*& state, unsigned int& size )
    {
        if( typeid( T ).name() == typeid( string ).name() )
        {
            //go ahead and give it to'm
            cout << "String!" << endl;
            return serializeString( data, state, size );
        }
        if( sizeof(T) > INT_MAX )
        {
            cout << "Data type way too big" << endl;
            return false;
        }
        state = new char[sizeof(T)];
        size = sizeof(T);
    
        //this is kind of silly, but a byte isn't always 8 bits,
        //so we can't just say tmp & 0xFF below
        //I really can't believe I bothered to handle this case.
        //However, this loop _is_ worth it just to think of the C coders screaming bloody murder
        //and pounding their heads in the desk
        char mask = 0x0; //a char is always 1 byte though
        for( int i = 0; i < CHAR_BIT; ++i )
        {
            mask = (mask << 1) | 0x01; 
        }
        //index needs to handle negative so we can count down, no unsigned int
        for( int i = sizeof(T) - 1; i >= 0; --i )
        {
            T tmp = data >> i * CHAR_BIT; 
            state[i] = tmp & mask;         
        }
    }

Then you come home the next night and realize that what you really meant was:

   template<class T>
   bool serialize( T data, char*& state, unsigned int& size )
   {
       size = sizeof(T);
       state = new char[size];
       memcpy( state, (void*)&data, size );
   }

Nothing like a good nights sleep!

Cheers!

PS. Blogger sucks at trying to put code up. Try Ctrl-Shift-+.

Smokin'

The smoke test is back, and using the ogre renderer. But the important thing is all the components are working together again. So now it's time to break stuff!

There's two major things I'm looking at:

- Simplify message passing.

I plan to do this by implementing object serialization for my hierarchy. I have to be careful, but I think I know how to do this. It might take some time, but it shouldn't be too bad.

- Exposing the interfaces of server-managed objects.

This is harder, and I'm just looking at mechanism right now, not even policy. The current deal is every method matches up with a command in the Commands object. Commands are specified in an xml file, and code is generated from them. Each command requests certain data, and has an expected response. Requests and responses are specified in xml files, and client interface code is generated from them. On the server, each object is wrapped in a shared library that can be loaded by the server depending on what behavior you want the server to take on. Each shared libary exposes the same interface to the server object - handleCommand( id, command ) I WANT SOMETHING SIMPLE! Oy. Or at least more transparent on the client end, and less work on the server end. Yes, I know this is an old problem with corba, rmi, soap, etc, littering the path before me. So this problem is harder :|

But that's what makes it fun! :)

Screenshots...
Unfortunately, I couldn't dig up some of the neater patterns I used to have, but they do demonstrate 3D hierarchal lsystems. The second has breeding turned on, so it looks a little more messed up.

Bouncing Ninjas!

So after trying OpenGL + ODE, Irrlicht + ODE, I've finally settled on OGRE + ODE + ? for the client rendering/smoke tests.

A little history.
A little while ago, after getting the lsystem, system server, interpreter, interpreter server, and my OpenGL + ODE client all displaying lsystems that were interpreted as 3D points in space, and could be evolved (via mutations and cross-breeding), I began to run up against the limits of how much time I could put into OpenGL. So I began looking for a 3D toolkit I was happy with. OGRE is it. However, now I need to find an input system I'm happy with. OIS is not it, and SDL may be it, but the OGRE codebase has drifted ever more and accomodating OIS/CEGUI, so that's what I'm using for now. I had SDL integrated in, but pulled it.

But for now, I'm eager to get back to fixing all the bugs I've found in the other components: subtle issues with mutexes, simplifying message passing (I'm not scared of hex anymore, so I'm going to make the objects serialize themselves instead of the current mess). And I'll just be glad to see those 3D l-systems again (especially coz I can't find any screenshots to prove I did it). For now, here's some screenshots of bouncing ninjas, and 2D systems generated from a codebase that has since been pretty much re-written.

Enjoy!

Ninjas:


Trying to see how big I could make the system (something like 100K elements/spheres in this one)

Detail:

Systems with the same start state, but one has a mutation rate > 1.

Playing around with different settings:

Bouncing Ninjas Refactored!

Intro

Hello, this is my blog about my project. I noticed I kept updating the intro page, and realized one day I'd want screenshots, musings and such. So I turned to this.
Maybe I'll even get around to sketching out the design in something other than code one day, and post it ;).
For now, the current status:

Cons

• Currently broken only has some working components (servers, lsystem, some physics).And bouncing ninjas!


• Undergoing a lot of rework.


• At the very least depends on Boost, ODE.


• Client will depend on OGRE.


• Client may end up depending on GDK/SDL, but currently trying OIS instead of SDL.


• I currently don't like OIS, but I need to focus on other things.


• SDL code written but pulled out temporarily.


• Currently only works on linux. I don't have windows to do the port (but I keep it in mind).


• No Design/Overview documentation.


• Some of the design kind of sucks.

Pros

• Has had all the parts working together in the past.


• Some of the design is kind of nice.


• platform dependencies are wrapped/abstracted.


• Build system is simple and straightforward. No automake/m4 magic.


• I always kept it moving forward, albeit often slowly, with long pauses, and many rewrites, over the last few years.