Lets see if we can kickstart this blog with a quickie Cocoa tip. You know it's a bad sign that I always have to look up Markdown syntax every time I post because I've forgotten since the last time.

In Snow Leopard, Apple added some enumeration methods to several collection classes that use blocks. Several of those use a mysterious NSEnumerationOptions parameter. There aren't any links in the docs to the valid flag values for NSEnumerationOptions parameters.

NSArray: -enumerateObjectsWithOptions:usingBlock:, -enumerateObjectsAtIndexes:options:usingBlock:
NSDictionary: -enumerateKeysAndObjectsWithOptions:usingBlock:
NSSet: -enumerateObjectsWithOptions:usingBlock:
NSIndexSet: -enumerateIndexesWithOptions:usingBlock:, -enumerateIndexesInRange:options:usingBlock:

For anyone else Googling for some answers, I did eventual dig up the documentation for the values.

NSEnumerationConcurrent: invoke the Block on the selected elements concurrently; the order of invocation is nondeterministic and undefined; this flag is a hint and may be ignored by the implementation under some circumstances; the code of the Block must be safe against concurrent invocation

NSEnumerationReverse: invoke the Block on the selected elements in reverse of the natural order; available for NSArrays and NSIndexSets; undefined for NSDictionarys and NSSets, or when combined with the NSEnumerationConcurrent flag

Now, I'll go log a RADAR bug about the missing link...

Nokia has just announced that Qt v4.5 is going to be available in LGPL form. This is a huge announcement, and is interesting from a number of perspectives, many of which aren't necessarily obvious at first glance.

I was a recent guest on Late Night Cocoa where I extolled the virtues of developing applications in high level languages like Python, especially if you wish to do cross-platform development. Nokia's announcement could be a game changer.

I love Cocoa and the new Python bindings that Apple has supplied. However, if you need to do things cross-platform, Qt is the best alternative. I have concluded, from my many experiments, that Qt is the only cross-platform toolkit that approaches the polish of Cocoa. And, RiverBank's excellent PyQt is a very natural binding for Python. And, best of all, Qt 4.5 on the Mac is built on Cocoa.

However, up until now there has been a problem. If you licensed Qt for use in a commercial app, you would be forbidden to directly expose Qt via a scripting language to your customers. This makes sense from a certain perspective because a development license for Qt can run up to $3k. If you offer scripting in your app, and your app costs less than a $3k, then you would be undercutting Nokia. And, you would be keeping all the money.

In other words, until now you would have been prohibited from using PyQt in your commercial application if you exposed a scripting API, even if you were willing to pay the licensing fees for Qt and PyQt. However, with the LGPL, you'll be able to link to the Qt libraries from a commercial app without a commercial license from Nokia.

If people realize the inherent value in using Qt with a scripting language like Python, this could open up a whole new avenue for low-cost, high-quality, cross platform development.

I made my rookie podcast appearance on Late Night Cocoa a few days ago. My episode is called Porting Large Scale Applications to the Mac Platform (Note that one of the definitions of "episode" in my dictionary is "A finite period in which someone is affected by a specified illness").

During the episode I ramble on about many of the things I learned the hard way while working on the port of Maya, and more recently, the port of Houdini to the Mac platform.

I think I'm the first new "premium content" on the Mac Developer Network now Scotty has changed up the paid content model. I'm flattered, but hope he's got other plans to drive subscriptions.

Speaking of that, go out and support the MDN! It's a great service that I'd hate to lose.

If you are writing performance critical code for Linux, you should get Zoom. Period.

Note: I have no financial involvement in Zoom and do not receive commissions.

Most developers who have worked on the Mac will be familiar with Shark, the excellent profiling tool that Apple supplies to developers. Zoom is written by the same guys. Zoom is unfortunately not free ($199 USD), but if you are trying to optimize your code, it'll easily save you $200 worth of your time.

One element of making a solid application is to make it as easy as possible for users to report problems to you. But, it doesn't help you the developer much if you get a report such as "I received an error 42 when I tried to load my file." Wouldn't it be better to have the error report tell you the exact line of code.

I'm going to show you how to extract a stack trace from an NSException. Apple has kindly posted a guide to printing symbolic stack traces, however, their solution requires that the Developer Tools be installed because it uses the atos command-line utility.

The method I will outline below does not require any external command-line tools which may-or-may-not be installed. Instead, they require a couple of glibc functions.

And, just to get fancy, this code will also de-mangle C++ symbols in your stack traces. Those who are working in Objective-C only (lucky bastards) can delete half of the code with no repercussions.

This is the sort of problem best described with code, so here goes:

NSString * UTstackTraceForException(NSException * exception)
{
    NSArray * addresses = [exception callStackReturnAddresses];
    void * backtrace_frames[[addresses count]];
    int i = 0;
    NSString * trace = @"";
    NSString * frame_description, * symbol;

    for (NSNumber * address in addresses)
    {
        backtrace_frames[i] = (void *)[address unsignedLongValue];
        i++;
    }

    char **frameStrings = backtrace_symbols(&backtrace_frames[0], [addresses count]);

    if(frameStrings != NULL) {
        int x;
        for(x = 0; x < [addresses count]; x++) {
            // Loop over the description of each stack frame.
            // Try to find the symbol name and demangle it if possible

            if(frameStrings[x] == NULL) { break; }

            char * demangled = NULL;
            size_t length;
            int status;

            frame_description = [NSString stringWithUTF8String:frameStrings[x]];
            NSArray * components = [frame_description componentsSeparatedByCharactersInSet:[NSCharacterSet whitespaceCharacterSet]];

            int count = 0;
            for (NSString * component in components)
            {
                if ([component length] == 0)
                    continue;

                count++;
                if (count == 4)
                {
                    symbol = component;
                    demangled = __cxxabiv1::__cxa_demangle([symbol UTF8String], 
                                                           demangled, 
                                                           &length, &status);
                    break;
                }
            }

            if (demangled)
            {
                NSString * demangled_string = [NSString stringWithUTF8String:demangled];
                frame_description = [frame_description stringByReplacingOccurrencesOfString:symbol withString:demangled_string];
                free(demangled);
            }

            trace = [trace stringByAppendingFormat:@"%@\n", frame_description];
        }
        free(frameStrings);
    }
    return trace;
}

I'm off to WWDC in a weeks time. If anyone wants to meet up and chat about Houdini on OS X, drop me a line at mike@sidefx.com.

One of the things I've done for the upcoming version of Houdini for OS X is to fancy up some of the icons a bit. Our build system is still based on Makefiles. That means I've got to have some way to set a file's icon from the shell.

There are a number of solutions out there using AppleScript. However, Leopard's now standard inclusion of PyObjC makes things even more simple. Here's the Python code you need to set a files icon, given any image.

from AppKit import *

def set_icon(icon_path, file_path):
    image = NSImage.alloc().initWithContentsOfFile_(icon_path)

    workspace = NSWorkspace.sharedWorkspace()
    workspace.setIcon_forFile_options_(image, file_path, 
        NSExcludeQuickDrawElementsIconCreationOption)

Pretty simple. There is one thing to watch out for though. Since this is a Cocoa routine, it needs to access the window server. And therefore will not work unless the user running the script is logged into the console. That's a bit of a pain for automated builds, but not a big deal for most people.

The cat is out of the bag so to speak. My project since joining Side Effect Software in May has been porting Houdini to OS X.

We are accepting applications for the beta here.

More details will be forthcoming. It's been an amazing porting experience.

In my last post, High Speed Cometh to Those Who Wait, I mentioned that Bell had called my up to say that upgrades in my area meant that DSL was now available. I think I said something like "... won't truly believe it until the bits are flowing down the pipe." Well, it looks like all that got upgraded was Bell's optimism about the range of their budget DSL offering.

What follows is a letter I wrote to Bell, CC'ing the North Renfrew Times.

On February 23rd I received a voice mail that sounded too good to be true. Chad from Bell had called to say that upgrades in my area of Laurentian Hills meant that high speed internet was now available. My wife and I both work out of our house for software companies in Toronto. Our internet connection is our tether to our jobs. It would give us great joy to dispense with the restrictions, unreliability, and high cost that is the reality of our existing two-way satellite connection.

I returned the call and asked to be hooked up as soon as possible. Chad arranged for a Bell sales agent to call me on the Monday morning. When the call came I was informed that I could only get the "Essential" package, which is slower than our satellite link. Despite that, we still deemed it worth switching.

Two day later, when I checked the Bell web site, it had changed to say that we were now eligible for the faster "Total Internet Performance" package. So, I called back to confirm this and a Bell representative said I could have any package I wanted. When I asked if I needed a different modem for the faster package, I was told yes, and that I needed to send back the first one they had sent me. So I returned the first one to sender. It took me weeks of phone calls to get a replacement sent out. When the replacement finally arrived a few days ago, on my birthday, it was the same kind I had sent back in the first place. I found out later that they had never actually upgraded my account to the "Total Internet Performance" package.

When I finally plugged in my modem, the connection light remained blank. This did not surprise me because, by this point, I had learned that several of my neighbours had already experienced this. However, I gamely phoned up 310-Bell to see if anyone could do anything for me. After about an hour I managed to work my way up to the top level tech support who scheduled a service appointment for the next day. Ten minutes later she called back and said that, she was sorry, but due to "distance issues", high speed was not available in our area. When I asked to have the service removed from my account, the agent from technical support said that I had to call the business center again.

I spent another two hours of my birthday getting bounced back-and-forth between the business agents and the tech support agents in search of a promise that the service would be cancelled. The business agents refused to believe me when I told them that I could not receive high speed at my location, and they kept bouncing me back to the tech support side, except for one woman who clearly did not want to deal with the issue and just unceremoniously disconnected me.

The worst part is that I know that this drama is likely replaying itself all over the area with my neighbours. After canvasing the whole area looking for "takers", only to waste countless hours of our time, perhaps Bell could actually perform the upgrades that they promised to all of us. At this point, I do not have much faith that I will see that happen.

I got the call from Bell last weekend. They said "High Speed is now available in your neighbourhood." I couldn't believe it, and probably won't truly believe it until the bits are flowing down the pipe.

I should preface this by saying that I live in the middle of nowhere. Deep River, a metropolis of 4200, is the nearest town. It is about six kilometers away from me, far outside DSL range normally. I actually reside in the township of Laurentian Hills in a tiny place called Point Alexander.

We've been trying to get a wireless internet provider to come to the area for some time. I had written off DSL as unattainable.

Needless to say, I asked for a sales person to call me at the earliest possible moment. I said, I'll buy the fastest connection you offer. Here's the kicker, that turned out to be 512kbps with a 2Gb/month download cap. Not exactly a connection plan that gets a power user all rev'ed up. Still, it's better than my current 1.5Mbps connection from Xplornet in many ways. And, it's about $100/month cheaper than my satellite even if I totally blow the bandwidth cap.

However, the story continues...I went to the Bell web site today on a whim and plugged in my phone number. It said that I was eligible to upgrade to 7Mbps! So I called a Bell human being and said "Is this right? Can I really have the full speed deal?" and the woman assured me that I could, so I ordered it.

Now comes the strange part. I started plugging in my neighbour's phone numbers in the Bell site and it says that they are only eligible for 512kbps. The site stubbornly still says I'm good for 7Mbps though.

So, we will see how it goes. Hopefully Bell can honor this promise!!

Completely by accident, I just discovered that Terminal in Leopard supports middle mouse button paste. That was a staple in my workflow before I moved over to OS X. Now I can revive the old muscle memory, at least in Terminal.

This post has been sitting around waiting for my NDA to lift as Leopard is released into the wild. This post is about migrating from executable_path to rpath as a means of bundling relocatable libraries with your application. This is a technique that I used to make Maya relocatable.

The Problem

Sometimes developers find it is necessary to pack up a few extra dylib shared libraries inside an application bundle. In that case, it becomes necessary to do some fancy linking so that your application is relocatable.

Normally a Mach-O binary (eg a dylib, bundle, or executable) maintains a list of shared libraries that it links to. By default that list will contain a list of absolute paths to the libraries. Lets look at a random example using otool -L to dump the list of linked libraries.

#  otool -L /usr/bin/more
/usr/bin/more:
    /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0,...
    /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, cur...
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, cu...

You can see that the executable "more" links against three system dylibs. The absolute path to each dylib is listed. If you moved one of those dylibs, more would cease to function.

The system libraries are going to stay put. The real problem occurs when you use a non-system library (such as an open source library) in your application, and you'd like to stick it in your application bundle under .../Contents/Libraries. Lets say my example application is CodeShorts.app and I'm using the library liblua.dylib to use to embed the Lua scripting language (note I'd use Python, but it comes with OS X and therefore makes a terrible example). I'd like to have the following application layout.

CodeShorts.app/
    Contents/
        MacOS/
            CodeShorts
        Libraries/
            liblua.dylib

Now, you could link to liblua.dylib using the path /Applications/CodeShorts.app/Contents/Libraries/liblua.dylib, and hope that nobody moves your app. However, many Mac users love to spend hours a day reorganizing their beloved Macs, and who knows where CodeShorts.app will end up (hopefully not the trash).

The Old Way: executable_path

Apple has a couple of solutions for this. Until recently (pre Leopard), this solution was to stick the special name @executable_path into the library path as referenced by your application. So, CodeShorts would look for @executable_path/../Libraries/liblua.dylib. At runtime dyld (the dynamic linker) would substitute @executable_path for /Applications/CodeShorts.app/Contents/MacOS and you'd be off to the races. You could move CodeShorts.app around all afternoon and it would still run (unless you moved it to the Trash). This works reasonably well, but it does impose some restrictions. The two main restrictions are as follows: 1. the relative path between the executable and the library is fixed. If you want to use the same shared library from two executables, then both executables must share the same relative path to the dylib. 2. you can only specify one place to look for the shared library. You cannot have a multi-location search path

The New Way: rpath

If you are a developer who has used this technique, then you're in for a surprise when you try to use the old -executable_path flag on ld under Leopard. It's gone. Leaving you to wonder "what do I do now??"

One of the lovable, but frustrating things about Apple is that they force forward progress on their users and developers.

Now that Leopard is the prime cat, the new mechanism for this is rpath which will be familiar at least in name to developers from other platforms such as Linux. Apple has, of course, thrown in a few extras as they are wont to do. The main resources for the rpath changes are the dyld Release Notes for Mac OS X v10.5 and the ld manpage on Leopard.

At its most basic, the -rpath flag to ld allows you to specify a list of directories where dyld should look for library dependencies. So, for example if you link CodeShorts to liblua.dylib, you could specify -rpath /Applications/CodeShorts.app/Contents/Libraries -rpath /opt/local/lib. That would make the CodeShorts application look in two places for liblua.dylib and any other dependencies. That's a good start, but it does not address the relocation problem we set out to solve, unless you can guess every place the user might stick your application (-rpath /Users/JoeUser/.Trash).

To solve this problem, Apple has introduced two new special funky @ names that you can use. One is for use in your library's name that is embedded in the library. And one is for use in your rpath. We'll get to those in a minute.

First we need to look a bit deeper into library dependencies on OS X. When you link a shared library, framework, or bundle on OS X, its path is embedded into the Mach-O binary. That name is later used by other Mach-O binaries that reference your library. So, lets go back to our example above where we looked at /usr/bin/more.

#  otool -L /usr/bin/more
/usr/bin/more:
    /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0,...
    /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, cur...
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, cu...

You can see that more uses /usr/lib/libncurses.5.4.dylib. Now, where did that path come from? When more was linked, the linker looked at libncurses.5.4.dylib and read the path name for the library and then added it to the list of dependencies for more. We can look at /usr/lib/libncurses.5.4.dylib and see what the path name is:

# otool -L /usr/lib/libncurses.5.4.dylib
/usr/lib/libncurses.5.4.dylib:
    /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0,...
    /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, cur... 
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, cu...

The first entry is the name of the Mach-O binary. When more was linked, ld looked at libncurses.5.4.dylib and read the path /usr/lib/libncurses.5.4.dylib and added it to more's dependencies.

Now, back to those two special @ names. The first special name is @rpath. This name can be used to lookup a library at a path relative to the directories in the loader's rpath. You would use this when the library will live in a certain subdirectory somewhere along the rpath. While it might not be obvious why you would need this, it becomes quickly apparent when you think of Frameworks where the actual Mach-O library lives buried in a sub directory. You don't want to have to specify an rpath entry for each individual framework. You just want to add your Framework directory to the rpath and have each framework picked up via its own relative install name.

It's time to create a contrived example. So, lets say you have two frameworks: BigLibrary.framework and SmallLibrary.framework. Now, lets say that they both live in /opt/Frameworks (of course, you would usually put them in /Library/Frameworks, but OS X knows about that directory, so its not a good enough contrived example). Your two Mach-O binaries would be as follows:

/opt/
    Frameworks/
        SmallLibrary.framework/
            Versions/
                1.0/
                    SmallLibrary
        BigLibrary.framework/
            Versions/
                1.0/
                    BigLibrary

When you link to those Frameworks from your application, you would use the flag -rpath /opt/Frameworks. That way, your application will search /opt/Frameworks when it loads. However, there's no Mach-O dylibs in that directory. But, that's ok. Because, when you linked SmallLibrary you cleverly specified -install_name @rpath/SmallLibrary.framework/Versions/1.0/SmallLibrary. Likewise, you specified the relative path for BigLibrary as well. That means that dyld will look in /opt/Frameworks for SmallLibrary.framework/Versions/1.0/SmallLibrary and, it will find it. So, that takes care of @rpath.

Now, this still doesn't solve the problem of relocatable libraries (get on with it will you!).

This brings us to the second special name, @loader_path. This special name may be used at the beginning of a path that is passed to the -rpath flag. At runtime, dyld replaces @loader_path with the path to the Mach-O binary that loaded the library. The nice thing here is that the loader can be an executable or another shared library of some sort. This is much more flexible than the old @executable_path solution, which only referred to the main executable.

Now, we have the tools to do what we really want to do. We can now hide libraries and framework bundles inside our application bundle and have them found no matter where the user drags the application.

Lets say that we want the following disk structure for our application CodeShorts, using all of the frameworks and libraries that we described above.

CodeShorts.app/
    Contents/
        MacOS/
            CodeShorts
        Libraries/
            liblua.dylib
        Frameworks/
            SmallLibrary.framework/
                Versions/
                    1.0/
                        SmallLibrary
            BigLibrary.framework/
                Versions/
                    1.0/
                        BigLibrary

We've got two frameworks and a library embedded in our application. We need to tell the application CodeShorts what directories to look in when searching for the libraries and frameworks, and we've got to give the libraries and frameworks the correct relative paths so that they'll be found.

When we link the application CodeShorts we are going to give it two places to look for libraries and frameworks. We'll do that using the -rpath flag something like this:

Mach-O Binary: 
  CodeShorts.app/Contents/MacOS/CodeShorts

Linker Flags:  
  -rpath @loader_path/../Libraries -rpath @loader_path/../Frameworks

NOTE: gcc doesn't know the -rpath flag, so you've got to pass it through to ld like this: -Wl,-rpath,@loader_path/...

This means that, when dyld loads the CodeShorts application, it will look for libraries in the Libraries and Frameworks directories inside the application.

When the libraries are linked (obviously needs to happen before the app is linked) they should be told that their install name is relative to the rpath. For liblua.dylib this is easy.

Mach-O Binary: 
  CodeShorts.app/Contents/Libraries/liblua.dylib

Linker Flags:  
  -install_name @rpath/liblua.dylib

For the frameworks, this is a little trickier, but not much. You need to give the location of the actual dylib, which is burried in a sub directory of the framework. But, you need to give it relative to the rpath.

Mach-O Binary: 
  CodeShorts.app/Contents/Frameworks/SmallLibrary.framework/Versions/1.0/SmallLibrary
Linker Flags:  
  -install_name @rpath/SmallLibrary.framework/Versions/1.0/SmallLibrary

Mach-O Binary: 
  CodeShorts.app/Contents/Frameworks/BigLibrary.framework/Versions/1.0/BigLibrary
Linker Flags:  
  -install_name @rpath/BigLibrary.framework/Versions/1.0/BigLibrary

That should be it. This is probably a long winded explanation of a simple concept. However maybe my verbosity balances out the terseness of the Apple documentation.

About three months ago I made a hard decision and left the Maya development team at Autodesk. It was a hard decision because I'd been part of the Maya team since January 1994, when I started as a student intern. The main reason I left is that I had been working on the same piece of software for thirteen years, which is an eternity in the software industry. The Maya team is a great group of people and they've got lots of big plans for Maya stretching off for years into the future. But for me, its time for a change.

My new gig is Side Effects Software, makers of the Houdini 3D animation software. While Autodesk is the fifth largest software company in the world, Side Effects is at least in the top five makers of software named after famous escape artists. That is too say, they are a much smaller shop. For a developer, that means a less overhead and more connection to the customers.

For anyone who's written Houdini off as a niche product should have a look at the Houdini 9 Public Beta. There are some great things happening in Houdini.

Now that I've gotten this post out of the way, I hope to get back to writing up bits and bobs that I come across, probably unrelated to my daily work for the most part.

This is the story of how I learned that it is a good idea to plan the deployment of your web projects ahead of time, and then match up your development environment with your chosen deployment environment as much as possible.

When developing this blog, I decided to use Python 2.5 for development because I wanted to use Sqlite which comes with Python 2.5. I liked the idea of being able to nuke the database by deleting a file.

When I went to deploy, I did so on a Linux box that was running mod_python hooked up to version 2.4 of Python. Rather than trying to rebuilt mod_python and whatnot, I decided to just back port my code.

It turned out to be non-trivial. And, problems are a lot harder to diagnose on a remote server.

The main problem turned out to be my XMLRPC code. It would appear that xmlrpclib got a big upgrade in Python 2.5. Which is good, unless you are back porting...

The first problem is that they changed SimpleXMLRPCDispatcher to take constructor arguments in 2.5 to deal with string encoding issues. That was pretty easy to find and fix.

# In Python 2.4 
dispatcher = SimpleXMLRPCDispatcher()

# In Python 2.5 
dispatcher = SimpleXMLRPCDispatcher(False,'UTF-8')

Next, came the hard part. I was getting an obscure error about an argument requiring a float. Over XMLRPC you only get the final error message and not a complete stack trace. I eventually tracked this down to xmlrpclib.DataTime. After digging through the Python source I figured out that the constructor for xmlrpclib.DataTime was extended in Python 2.5 to allow initializing with a Python datetime.datetime object, such as the ones Django returns. It seemed pretty logical that xmlrpclib.DataTime would support that, so it took me a while to figure out that it didn't support it in Python 2.4.

Anyway, for anyone still interested, here's the fix:

# In Python 2.4 (Note that I stole the strftime format string right from 
# Python 2.5's xmlrpclib source)
datetime = xmlrpclib.DateTime( post.date.strftime("%Y%m%dT%H:%M:%S") )

# In Python 2.5 
datetime = xmlrpclib.DateTime( post.date )

This is the inaugural post to Code Shorts. It is a weekend project gone a bit out of control. I could say that I wrote Code Shorts in Python using Django, but that is like saying that I built a table when I really just assembled one that I bought at Ikea.

I treated Code Shorts as a pedagogical exercise. My goal was to learn a bit about Django and bone-up on database-backed web development. I was pretty up on CGI scripting in the day, but the generation following it (to which J2EE belonged) was not really for the weekend warrior. Django and others like Ruby on Rails and Turbo Gears have really put the fun back in web app development.

In the end, I got a bit carried away. I did all of the blog HTML and CSS by hand. I had been meaning to get a better grip on CSS. Whether, I achieved that is debatable, but the results are better than anything I've done before. The logo was done in Maya of course (see the about page to understand why I say "of course").

Other nice fiddly bits include hooking up tags, comments, and RSS feeds. And I even added on an XMLRPC interface so that I could blog from TextMate, the best editor I have run across to date.