DDL and xf.linker

November 18, 2008

There seems to be a growing demand for dynamic linking of D code in a way that would not be a complete PITA. While *nix users could work with SO to some extent (as it apparently works fine with GDC), Windows blokes are left to the mercy of DLLs. As we all know, DLLs are scary. Yet still not many folks are taking a look at DDL.

The current status is that the version in the official SVN is not very stable or maintained – the linker has some issues, updates for new DMD and Tango version aren’t folded in regularly. I’ve been maintaining a separate branch of DDL with some of the issues fixed, but most importantly, I’ve created a custom linker that seems to function much better than the one in the official DDL repo. It sits at http://team0xf.com:1024/linker/ and is released under the MIT license, thus you have no excuse for not using it.

While my presentation at the Tango Conference 2008 described the linker in some detail, it lacked step-by-step instructions for using it. This post fixes that issue.

1. Preparations

First things first: we need to fetch xf.linker and my branch of DDL. If you have a unix-ish shell like MSYS or Cygwin, you can run the following commands:

wget -O linker.zip http://team0xf.com:1024/linker/archive/tip.zip
unzip linker.zip
mkdir xf
mv linker-* xf/linker
rm linker.zip

wget -O ext.zip http://team0xf.com:1024/ext/archive/tip.zip
unzip ext.zip
mv ext-*/ddl ./
rm ext.zip
rm -R ext-*

They will create two directories: xf and ddl in the current working path. If you don’t have any unix-ish shell (MSYS for the win), please slap yourself first, then interpret them using your favorite web browser and some sort of a package manager.

To make things easier for you, I’ve created a package with the examples from this article packed up nicely along with dependencies. It probably won’t be updated along with the repositories, though.

2. Hello world

The setup we’ll create consists of two .d files: one for the host application and one for the plugin. They will be compiled separately and then the host will load a class from the plugin. Let’s start with the host program:

 1 module Host;
 2 
 3 private {
 4     import xf.linker.DefaultLinker;
 5     import tango.io.Stdout;
 6 }
 7 
 8 void main() {
 9     auto linker = createDefaultLinkerCfgString(`
10         type    host        regex .*\.map
11         type    lib         regex .*\.obj
12         order   host        self
13         order   lib         host self
14         load    Host.map    .
15     `);
16 
17     auto lib = linker.load("Plugin.obj", ".");
18     auto pluginClass = lib.getClass!(Object, "Plugin.MyPlugin");
19 
20     Stdout.formatln("Loaded class: {}", pluginClass.name);
21 
22     auto pluginObject = pluginClass.newObject();
23     Stdout.formatln("Instantiated an object: '{}'", pluginObject);
24 }

Let’s also quickly create the plugin:

1 class MyPlugin {
2     char[] toString() {
3         return "I am a plugin object";
4     }
5 }

Ok now, the plugin should be pretty self-explanatory. The host code most probably isn’t. But before we dissect it, let me introduce some basic concepts behind the linker.

The main design goal of xf.linker has been flexibility. It’s supposed to work well in situations where you only have to load a single .obj file, a few plugins, or even in a system composed of dozens of dynamically reloadable components. At the same time, a single program may have different types of plugins. Some may be dependent upon others, plugins might even have their own plugins, etc. The linker approaches this issue by defining lib types and letting the user define the sources of symbols for each lib type in terms of other lib types. This is exactly what happens at the beginning of the main() function. The long string passed to createDefaultLinkerCfgString is mostly a configuration script for the library types and their link order. We define two types of components here:

• host – the host application
• lib – the plugin

… so nothing fancy here. When loading a component, its filename is matched against the defined rules and the first one that matches is chosen. In this case, we use regular expressions to match the files. The plugin(s) will be given in .obj files, while the host data will be loaded from a .map file. Why do we need a .map? It maps symbol names into addresses inside the executable, so we’ll be using it to resolve symbol names into addresses when looking them up during the linking process. The .map file is not strictly necessary, since some other reflection mechanism could be used, e.g. parsing the process image using the dbghelp API. There’s preliminary support for it in DDL, but I haven’t looked into it for a while, so we’ll stick to maps for the sake of this tutorial.

The next two lines of the configuration script define linking order for the library types. The host will not be relinked (executable programs are always fully linked), so we just tell it to self-resolve symbols when someone asks it for them. However, the plugin will need to get symbols from the host application, e.g. static instances of ClassInfo, TypeInfo, exception hooks, etc. Thus the linking order for the plugin will be host followed by self. Whatever symbols it can’t find in the host, it will try to find in itself. If that doesn’t work, we’ll get an exception.

Finally, the end of the config file contains a “load Host.map .” statement. It’s equivalent to calling linker.load(”Host.map”, “.”). We need to load the host so the plugin may link itself to it. “.” is the working directory, which is mostly important with on-demand compilation.

The rest of the code should be more or less self-explanatory. We load the plugin library with linker.load(”plugin.obj”, “.”); then we find the MyPlugin class within it via lib.getClass. The actual object that’s returned here is an instance of ddl.ExportClass. It contains a few useful properties, including name, classInfo and isAbstract, as well as methods to create instances of the class: newObject.

The getClass function template’s first parameter is the type to which the instantiated object should be cast; the second must be a fully qualified name of the class we wish to load.ExportClass.newObject is a variadic template function, so we can actually use it to pass arguments to a constructor. Please note that it uses the types provided to it via IFTI in order to locate the proper constructor – casts won’t be carried out automatically. For instance, if your class has a constructor with an int parameter and you call newObject(’a'), you’ll get an error, since DDL will be looking for a ctor taking a char. Thus, make sure to always explicitly provide the types for the ctor when calling newObject, e.g. newObject(cast(int)‘a’);

We can now compile the plugin and the host application. Nothing fancy here:

dmd -c Plugin.d
rebuild -L/M -clean -Iddl Host.d

After running these commands, we should be given three files: Host.exe, Host.map and Plugin.obj.

Executing Host.exe should yield the following output:

Loaded class: 6Plugin8MyPlugin
Instantiated an object: 'I am a plugin object'

( There may be some destructor spam as well, you can safely ignore it. )

3. Class extraction and exceptions

This time we’ll show how DDL can do class loading and that it doesn’t crash with exceptions as DLLs love to. We’ll start from making a Common.d file containing a single definition of an IPlugin interface:

1 module Common;
2 
3 interface IPlugin {
4     void doStuff();
5 }

Then we’ll make a Plugin.d module which contains two sample implementations of IPlugin:

 1 module Plugin;
 2 
 3 private {
 4     import tango.io.Stdout;
 5     import Common;
 6 }
 7 
 8 class GoodPlugin : IPlugin {
 9     void doStuff() {
10         Stdout.formatln("GoodPlugin is being good");
11     }
12 }
13 
14 class BadPlugin : IPlugin {
15     void doStuff() {
16         throw new Exception("BadPlugin blows up");
17     }
18 }

The host executable will want to load the Plugin.obj and instantiate all classes that implement IPlugin within it. Then we shall call doStuff() on the instances:

 1 module Host;
 2 
 3 private {
 4     import xf.linker.DefaultLinker;
 5     import Common;
 6     import tango.io.Stdout;
 7 }
 8 
 9 void main() {
10     auto linker = createDefaultLinkerCfgString(`
11         type        host        regex .*\.map
12         type        lib         regex .*\.obj
13         order       host        self
14         order       lib         host self
15         load        Host.map    .
16     `);
17 
18     auto lib = linker.load("Plugin.obj", ".");
19     foreach (sub; lib.getSubclasses!(IPlugin)) {
20         Stdout.formatln("Loaded class: {}", sub.name);
21         auto pluginObject = sub.newObject();
22         Stdout.formatln("Instantiated an object: '{}'", pluginObject);
23 
24         Stdout.formatln("Calling IPlugin.doStuff");
25         try {
26             pluginObject.doStuff();
27         } catch (Exception exc) {
28             Stdout.formatln("ONOZ, Exception caught while running doStuff: '{}'", exc);
29         }
30     }
31 }

As you can see, the beginning of the host is almost exactly the same as in the previous example. But then, instead of loading a single class the library, we iterate through all ’subclasses’ of a given class/interface: foreach (sub; lib.getSubclasses!(IPlugin)). This yields us a list of ddl.ExportClass objects. Calling sub.newObject() gives us an instance of IPlugin.

The rest of the module is formality. We build it just as before and upon running, should get:

Loaded class: 6Plugin10GoodPlugin
Instantiated an object: 'Plugin.GoodPlugin'
Calling IPlugin.doStuff
GoodPlugin is being good
Loaded class: 6Plugin9BadPlugin
Instantiated an object: 'Plugin.BadPlugin'
Calling IPlugin.doStuff
ONOZ, Exception caught while running doStuff: 'BadPlugin blows up'

As we can see, the IPlugin implementations got instantiated and their methods were called. In the second case we were able to safely catch an exception. This is already more than what SO and DLL can provide. Using exceptions across host/plugin boundaries with DLLs would result in an immediate crash, while neither SO or DLLs would be able to give us a list of classes implementing our IPlugin interface.

Ok now, this stuff could be done with DDL a long time ago. Let’s now try something xf.linker – specific:

4. On-demand compilation

For the sake of simplicity, we’ll reuse the Plugin.d from the first example here:

1 class MyPlugin {
2     char[] toString() {
3         return "I am a plugin object";
4     }
5 }

The host app will also be very similar to the first one, with just two slight differences – changing “.obj” to “.d” in the linker config and in the linker.load() call:

 1 module Host;
 2 
 3 private {
 4     import xf.linker.DefaultLinker;
 5     import tango.io.Stdout;
 6 }
 7 
 8 void main() {
 9     auto linker = createDefaultLinkerCfgString(`
10         type    host        regex .*\.map
11         type    lib         regex .*\.d
12         order   host        self
13         order   lib         host self
14         load    Host.map    .
15     `);
16 
17     auto lib = linker.load("Plugin.d", ".");
18     auto pluginClass = lib.getClass!(Object, "Plugin.MyPlugin");
19 
20     Stdout.formatln("Loaded class: {}", pluginClass.name);
21 
22     auto pluginObject = pluginClass.newObject();
23     Stdout.formatln("Instantiated an object: '{}'", pluginObject);
24 }

That’s it when it comes to the configuration of the D code. But we also have to get a few more things: DMD, the Build tool and object.di from Tango, because as it is now, the dynamic compilation system in xf.linker expects them to be present in the directory with the host app. We only need two files from DMD: dmd.exe and sc.ini. They shall reside in the bin/ directory along with build.exe. Build is used to find dependencies between modules (it can scan the list of imports of a given module pretty quickly).

This gives us the following directory layout:

• Host.d
• Plugin.d
• bin/build.exe
• bin/dmd.exe
• bin/sc.ini
• import/object.di
• xf/ddl/

This time we won’t build the Plugin.d manually. We simply compile the Host executable …

rebuild -L/M -clean -Iddl Host.d

… and run it:

Compiling 'some-path/compile/Plugin.d'
Loaded class: 6Plugin8MyPlugin
Instantiated an object: 'I am a plugin object'

Let’s run it immediately again:

Loaded class: 6Plugin8MyPlugin
Instantiated an object: 'I am a plugin object'

It was compiled only the first time. When we ran Host.exe the second time, the D library provider for xf.linker.LazyLinker used the cached .obj and dependency files stored in _objCache and _depCache. It also creates two junk files: “.rsp” and “.def”, but that’s only temporary.

5. Closing words

This concludes the introduction to DDL and xf.linker. For a more advanced example, please refer to testLazy.d in the xf.linker repository, as it’s the semi-real-time ray-tracing example I’ve demoed at the Tango conference. I hope that my ramblings were more or less understandable and enough to get you excited about this linking technology. It’s got potential to change the way D programmers approach component-based software development and really distinguish D from more conventional system-level programming languages, such as C/C++.

Filed under: DDL | Comments (8)