This post has been sitting around waiting for my NDA to lift as Leopard is released into the wild. This post is about migrating from executable_path to rpath as a means of bundling relocatable libraries with your application. This is a technique that I used to make Maya relocatable.

The Problem

Sometimes developers find it is necessary to pack up a few extra dylib shared libraries inside an application bundle. In that case, it becomes necessary to do some fancy linking so that your application is relocatable.

Normally a Mach-O binary (eg a dylib, bundle, or executable) maintains a list of shared libraries that it links to. By default that list will contain a list of absolute paths to the libraries. Lets look at a random example using otool -L to dump the list of linked libraries.

#  otool -L /usr/bin/more
/usr/bin/more:
    /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0,...
    /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, cur...
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, cu...

You can see that the executable "more" links against three system dylibs. The absolute path to each dylib is listed. If you moved one of those dylibs, more would cease to function.

The system libraries are going to stay put. The real problem occurs when you use a non-system library (such as an open source library) in your application, and you'd like to stick it in your application bundle under .../Contents/Libraries. Lets say my example application is CodeShorts.app and I'm using the library liblua.dylib to use to embed the Lua scripting language (note I'd use Python, but it comes with OS X and therefore makes a terrible example). I'd like to have the following application layout.

CodeShorts.app/
    Contents/
        MacOS/
            CodeShorts
        Libraries/
            liblua.dylib

Now, you could link to liblua.dylib using the path /Applications/CodeShorts.app/Contents/Libraries/liblua.dylib, and hope that nobody moves your app. However, many Mac users love to spend hours a day reorganizing their beloved Macs, and who knows where CodeShorts.app will end up (hopefully not the trash).

The Old Way: executable_path

Apple has a couple of solutions for this. Until recently (pre Leopard), this solution was to stick the special name @executable_path into the library path as referenced by your application. So, CodeShorts would look for @executable_path/../Libraries/liblua.dylib. At runtime dyld (the dynamic linker) would substitute @executable_path for /Applications/CodeShorts.app/Contents/MacOS and you'd be off to the races. You could move CodeShorts.app around all afternoon and it would still run (unless you moved it to the Trash). This works reasonably well, but it does impose some restrictions. The two main restrictions are as follows: 1. the relative path between the executable and the library is fixed. If you want to use the same shared library from two executables, then both executables must share the same relative path to the dylib. 2. you can only specify one place to look for the shared library. You cannot have a multi-location search path

The New Way: rpath

If you are a developer who has used this technique, then you're in for a surprise when you try to use the old -executable_path flag on ld under Leopard. It's gone. Leaving you to wonder "what do I do now??"

One of the lovable, but frustrating things about Apple is that they force forward progress on their users and developers.

Now that Leopard is the prime cat, the new mechanism for this is rpath which will be familiar at least in name to developers from other platforms such as Linux. Apple has, of course, thrown in a few extras as they are wont to do. The main resources for the rpath changes are the dyld Release Notes for Mac OS X v10.5 and the ld manpage on Leopard.

At its most basic, the -rpath flag to ld allows you to specify a list of directories where dyld should look for library dependencies. So, for example if you link CodeShorts to liblua.dylib, you could specify -rpath /Applications/CodeShorts.app/Contents/Libraries -rpath /opt/local/lib. That would make the CodeShorts application look in two places for liblua.dylib and any other dependencies. That's a good start, but it does not address the relocation problem we set out to solve, unless you can guess every place the user might stick your application (-rpath /Users/JoeUser/.Trash).

To solve this problem, Apple has introduced two new special funky @ names that you can use. One is for use in your library's name that is embedded in the library. And one is for use in your rpath. We'll get to those in a minute.

First we need to look a bit deeper into library dependencies on OS X. When you link a shared library, framework, or bundle on OS X, its path is embedded into the Mach-O binary. That name is later used by other Mach-O binaries that reference your library. So, lets go back to our example above where we looked at /usr/bin/more.

#  otool -L /usr/bin/more
/usr/bin/more:
    /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0,...
    /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, cur...
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, cu...

You can see that more uses /usr/lib/libncurses.5.4.dylib. Now, where did that path come from? When more was linked, the linker looked at libncurses.5.4.dylib and read the path name for the library and then added it to the list of dependencies for more. We can look at /usr/lib/libncurses.5.4.dylib and see what the path name is:

# otool -L /usr/lib/libncurses.5.4.dylib
/usr/lib/libncurses.5.4.dylib:
    /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0,...
    /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, cur... 
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, cu...

The first entry is the name of the Mach-O binary. When more was linked, ld looked at libncurses.5.4.dylib and read the path /usr/lib/libncurses.5.4.dylib and added it to more's dependencies.

Now, back to those two special @ names. The first special name is @rpath. This name can be used to lookup a library at a path relative to the directories in the loader's rpath. You would use this when the library will live in a certain subdirectory somewhere along the rpath. While it might not be obvious why you would need this, it becomes quickly apparent when you think of Frameworks where the actual Mach-O library lives buried in a sub directory. You don't want to have to specify an rpath entry for each individual framework. You just want to add your Framework directory to the rpath and have each framework picked up via its own relative install name.

It's time to create a contrived example. So, lets say you have two frameworks: BigLibrary.framework and SmallLibrary.framework. Now, lets say that they both live in /opt/Frameworks (of course, you would usually put them in /Library/Frameworks, but OS X knows about that directory, so its not a good enough contrived example). Your two Mach-O binaries would be as follows:

/opt/
    Frameworks/
        SmallLibrary.framework/
            Versions/
                1.0/
                    SmallLibrary
        BigLibrary.framework/
            Versions/
                1.0/
                    BigLibrary

When you link to those Frameworks from your application, you would use the flag -rpath /opt/Frameworks. That way, your application will search /opt/Frameworks when it loads. However, there's no Mach-O dylibs in that directory. But, that's ok. Because, when you linked SmallLibrary you cleverly specified -install_name @rpath/SmallLibrary.framework/Versions/1.0/SmallLibrary. Likewise, you specified the relative path for BigLibrary as well. That means that dyld will look in /opt/Frameworks for SmallLibrary.framework/Versions/1.0/SmallLibrary and, it will find it. So, that takes care of @rpath.

Now, this still doesn't solve the problem of relocatable libraries (get on with it will you!).

This brings us to the second special name, @loader_path. This special name may be used at the beginning of a path that is passed to the -rpath flag. At runtime, dyld replaces @loader_path with the path to the Mach-O binary that loaded the library. The nice thing here is that the loader can be an executable or another shared library of some sort. This is much more flexible than the old @executable_path solution, which only referred to the main executable.

Now, we have the tools to do what we really want to do. We can now hide libraries and framework bundles inside our application bundle and have them found no matter where the user drags the application.

Lets say that we want the following disk structure for our application CodeShorts, using all of the frameworks and libraries that we described above.

CodeShorts.app/
    Contents/
        MacOS/
            CodeShorts
        Libraries/
            liblua.dylib
        Frameworks/
            SmallLibrary.framework/
                Versions/
                    1.0/
                        SmallLibrary
            BigLibrary.framework/
                Versions/
                    1.0/
                        BigLibrary

We've got two frameworks and a library embedded in our application. We need to tell the application CodeShorts what directories to look in when searching for the libraries and frameworks, and we've got to give the libraries and frameworks the correct relative paths so that they'll be found.

When we link the application CodeShorts we are going to give it two places to look for libraries and frameworks. We'll do that using the -rpath flag something like this:

Mach-O Binary: 
  CodeShorts.app/Contents/MacOS/CodeShorts

Linker Flags:  
  -rpath @loader_path/../Libraries -rpath @loader_path/../Frameworks

NOTE: gcc doesn't know the -rpath flag, so you've got to pass it through to ld like this: -Wl,-rpath,@loader_path/...

This means that, when dyld loads the CodeShorts application, it will look for libraries in the Libraries and Frameworks directories inside the application.

When the libraries are linked (obviously needs to happen before the app is linked) they should be told that their install name is relative to the rpath. For liblua.dylib this is easy.

Mach-O Binary: 
  CodeShorts.app/Contents/Libraries/liblua.dylib

Linker Flags:  
  -install_name @rpath/liblua.dylib

For the frameworks, this is a little trickier, but not much. You need to give the location of the actual dylib, which is burried in a sub directory of the framework. But, you need to give it relative to the rpath.

Mach-O Binary: 
  CodeShorts.app/Contents/Frameworks/SmallLibrary.framework/Versions/1.0/SmallLibrary
Linker Flags:  
  -install_name @rpath/SmallLibrary.framework/Versions/1.0/SmallLibrary

Mach-O Binary: 
  CodeShorts.app/Contents/Frameworks/BigLibrary.framework/Versions/1.0/BigLibrary
Linker Flags:  
  -install_name @rpath/BigLibrary.framework/Versions/1.0/BigLibrary

That should be it. This is probably a long winded explanation of a simple concept. However maybe my verbosity balances out the terseness of the Apple documentation.

Completely by accident, I just discovered that Terminal in Leopard supports middle mouse button paste. That was a staple in my workflow before I moved over to OS X. Now I can revive the old muscle memory, at least in Terminal.

The cat is out of the bag so to speak. My project since joining Side Effect Software in May has been porting Houdini to OS X.

We are accepting applications for the beta here.

More details will be forthcoming. It's been an amazing porting experience.

One of the things I've done for the upcoming version of Houdini for OS X is to fancy up some of the icons a bit. Our build system is still based on Makefiles. That means I've got to have some way to set a file's icon from the shell.

There are a number of solutions out there using AppleScript. However, Leopard's now standard inclusion of PyObjC makes things even more simple. Here's the Python code you need to set a files icon, given any image.

from AppKit import *

def set_icon(icon_path, file_path):
    image = NSImage.alloc().initWithContentsOfFile_(icon_path)

    workspace = NSWorkspace.sharedWorkspace()
    workspace.setIcon_forFile_options_(image, file_path, 
        NSExcludeQuickDrawElementsIconCreationOption)

Pretty simple. There is one thing to watch out for though. Since this is a Cocoa routine, it needs to access the window server. And therefore will not work unless the user running the script is logged into the console. That's a bit of a pain for automated builds, but not a big deal for most people.

I'm off to WWDC in a weeks time. If anyone wants to meet up and chat about Houdini on OS X, drop me a line at mike@sidefx.com.

One element of making a solid application is to make it as easy as possible for users to report problems to you. But, it doesn't help you the developer much if you get a report such as "I received an error 42 when I tried to load my file." Wouldn't it be better to have the error report tell you the exact line of code.

I'm going to show you how to extract a stack trace from an NSException. Apple has kindly posted a guide to printing symbolic stack traces, however, their solution requires that the Developer Tools be installed because it uses the atos command-line utility.

The method I will outline below does not require any external command-line tools which may-or-may-not be installed. Instead, they require a couple of glibc functions.

And, just to get fancy, this code will also de-mangle C++ symbols in your stack traces. Those who are working in Objective-C only (lucky bastards) can delete half of the code with no repercussions.

This is the sort of problem best described with code, so here goes:

NSString * UTstackTraceForException(NSException * exception)
{
    NSArray * addresses = [exception callStackReturnAddresses];
    void * backtrace_frames[[addresses count]];
    int i = 0;
    NSString * trace = @"";
    NSString * frame_description, * symbol;

    for (NSNumber * address in addresses)
    {
        backtrace_frames[i] = (void *)[address unsignedLongValue];
        i++;
    }

    char **frameStrings = backtrace_symbols(&backtrace_frames[0], [addresses count]);

    if(frameStrings != NULL) {
        int x;
        for(x = 0; x < [addresses count]; x++) {
            // Loop over the description of each stack frame.
            // Try to find the symbol name and demangle it if possible

            if(frameStrings[x] == NULL) { break; }

            char * demangled = NULL;
            size_t length;
            int status;

            frame_description = [NSString stringWithUTF8String:frameStrings[x]];
            NSArray * components = [frame_description componentsSeparatedByCharactersInSet:[NSCharacterSet whitespaceCharacterSet]];

            int count = 0;
            for (NSString * component in components)
            {
                if ([component length] == 0)
                    continue;

                count++;
                if (count == 4)
                {
                    symbol = component;
                    demangled = __cxxabiv1::__cxa_demangle([symbol UTF8String], 
                                                           demangled, 
                                                           &length, &status);
                    break;
                }
            }

            if (demangled)
            {
                NSString * demangled_string = [NSString stringWithUTF8String:demangled];
                frame_description = [frame_description stringByReplacingOccurrencesOfString:symbol withString:demangled_string];
                free(demangled);
            }

            trace = [trace stringByAppendingFormat:@"%@\n", frame_description];
        }
        free(frameStrings);
    }
    return trace;
}

I made my rookie podcast appearance on Late Night Cocoa a few days ago. My episode is called Porting Large Scale Applications to the Mac Platform (Note that one of the definitions of "episode" in my dictionary is "A finite period in which someone is affected by a specified illness").

During the episode I ramble on about many of the things I learned the hard way while working on the port of Maya, and more recently, the port of Houdini to the Mac platform.

I think I'm the first new "premium content" on the Mac Developer Network now Scotty has changed up the paid content model. I'm flattered, but hope he's got other plans to drive subscriptions.

Speaking of that, go out and support the MDN! It's a great service that I'd hate to lose.

Nokia has just announced that Qt v4.5 is going to be available in LGPL form. This is a huge announcement, and is interesting from a number of perspectives, many of which aren't necessarily obvious at first glance.

I was a recent guest on Late Night Cocoa where I extolled the virtues of developing applications in high level languages like Python, especially if you wish to do cross-platform development. Nokia's announcement could be a game changer.

I love Cocoa and the new Python bindings that Apple has supplied. However, if you need to do things cross-platform, Qt is the best alternative. I have concluded, from my many experiments, that Qt is the only cross-platform toolkit that approaches the polish of Cocoa. And, RiverBank's excellent PyQt is a very natural binding for Python. And, best of all, Qt 4.5 on the Mac is built on Cocoa.

However, up until now there has been a problem. If you licensed Qt for use in a commercial app, you would be forbidden to directly expose Qt via a scripting language to your customers. This makes sense from a certain perspective because a development license for Qt can run up to $3k. If you offer scripting in your app, and your app costs less than a $3k, then you would be undercutting Nokia. And, you would be keeping all the money.

In other words, until now you would have been prohibited from using PyQt in your commercial application if you exposed a scripting API, even if you were willing to pay the licensing fees for Qt and PyQt. However, with the LGPL, you'll be able to link to the Qt libraries from a commercial app without a commercial license from Nokia.

If people realize the inherent value in using Qt with a scripting language like Python, this could open up a whole new avenue for low-cost, high-quality, cross platform development.

Lets see if we can kickstart this blog with a quickie Cocoa tip. You know it's a bad sign that I always have to look up Markdown syntax every time I post because I've forgotten since the last time.

In Snow Leopard, Apple added some enumeration methods to several collection classes that use blocks. Several of those use a mysterious NSEnumerationOptions parameter. There aren't any links in the docs to the valid flag values for NSEnumerationOptions parameters.

NSArray: -enumerateObjectsWithOptions:usingBlock:, -enumerateObjectsAtIndexes:options:usingBlock:
NSDictionary: -enumerateKeysAndObjectsWithOptions:usingBlock:
NSSet: -enumerateObjectsWithOptions:usingBlock:
NSIndexSet: -enumerateIndexesWithOptions:usingBlock:, -enumerateIndexesInRange:options:usingBlock:

For anyone else Googling for some answers, I did eventual dig up the documentation for the values.

NSEnumerationConcurrent: invoke the Block on the selected elements concurrently; the order of invocation is nondeterministic and undefined; this flag is a hint and may be ignored by the implementation under some circumstances; the code of the Block must be safe against concurrent invocation

NSEnumerationReverse: invoke the Block on the selected elements in reverse of the natural order; available for NSArrays and NSIndexSets; undefined for NSDictionarys and NSSets, or when combined with the NSEnumerationConcurrent flag

Now, I'll go log a RADAR bug about the missing link...