I’ve been thinking of writing a dbus client library for a while. And recently I’ve even been succeeding.
I’ve been using dbus for a while as part of Rhine where I use it to send notifications on some WM actions.
Though great criticism has been levied at dbus. I do think that that the situation is salvageable. Many desktop components currently use dbus as their primary form of IPC, so I don’t have much choice either.
So when I first implemented notification support in rhine, I turned to libnotify. This is a powerful library and it does the job but it also pulls in large amounts of glib/gobject. This is especially unfortunate considering my ambitions for static linking.
I thus turned to the specification and libdbus. And implemented the essential functionality of libnotify myself. Much credit is to be given to the excellent blogpost by Shem Sedrick for providing an example to build on. As part of this endeavour I ended up writing my own (incomplete) zig bindings for libdbus.
Though I certainly succeeded at relatively comfortable bindings with more type safety than the original, I hungered for more. libdbus was not packaged on allyourcodebase so I couldn’t easily link it dynamically. It also lacked some type safety and convenience that I was sure zig could provide. Thus I began looking for zig libraries.
So let’s see what the ecosystem looks like and if our needs can be met. The first thing to note is that many people seem to have tried. But few implementations seem ready enough to actually be used. These are the ones have I tried:
I wasn’t happy with any of these. But fixing some of their issues did give me the confidence to try writing my own dbus client library.
I can do better
I knew that these words would — as they so often do — mark the dawn of brilliance … or insanity.
So I commenced this endeavour with a few goals in mind. My library should
To solve a problem we must first get familiar with its substrate — with the space it lives in. So let’s get into our tooling.
Like all my recent problems I will be writing this in the zig programming language. It’s important to know what we have available to us.
Zig (unlike C) supports return-value-errors as a first-class language construct. This will improve the correctness of our code since We will be forced to handle errors wherever they can occur. This is something that southporter fails at and that my own bindings do slightly better. Zig forces us to be careful about which parts of our code can fail and which can’t. Let us try to only fail in ways which are recoverable.
Zig demands manual memory management and explicit passing of allocators. This not only allows for enhanced control but also for improved debugging and better code reuse. I thus want to follow this convention as much as possible. Because we become so aware of our allocations, where they happen and that they can fail, I want to keep the number of allocations to a minimum. There is not point being wasteful with out ressources and every allocation we make, we also need to clean.
The zig standard library has Interfaces for streaming readers and writers with a user-provided buffer size. This is exciting because it potentially provides us a way of avoiding allocations entirely — especially when it comes to reading from and writing to a socket or file. There are backends for writers and readers backed by a fixed amount of memory, or by an allocator. There’s even a writer which only counts the number of bytes written and discards their contents. I want this flexibility.
Version 0.16 of zig introduced an Io interface, which abstracts all Io interfaces and allows any code using it to seamlessly participate in different synchronous or asynchronous userspace scheduling schemes. I hope that this too will improve the flexibility of my code.
The zig standard library quite prominently features iterator patterns. These have the following shape:
var iter = mem.splitScalar(u8, "Hello,There,People,,yoooo", ',');
while (iter.next()) |str| {
// do something with the string str
}
So we create a small stack object for book keeping while we all the memory and heft associated with it is only referenced by pointer. These make for pretty nice low-cost abstractions. We gain a lot of type safety with very little overhead, since the iterator object only tracks whatever state we would have needed for our operation in the first place.
The dbus protocol is mysterious to many and understood by few. Let’s dive into the actual wire protocol.
Dbus is an IPC mechanism that allows different processes on a system to expose a variety of functionality to each other. There are dbus clients, which use a dbus server in order to communicate with eatch other. This communication is broken up into messages which are then routed by the bus. The bus uses the same protocol to allow clients to configure the routing of these messages.
Messages are the most basic unit of information sent between clients. Messages consist of two part: a header and a body. Both are serialized in the same way and consist of a collection of different types. Similar to wayland, A client needs to know the signature of a message in order to parse it.
Dbus supports a variety of different types in its message format. Each type has a string representation used to identify it on the bus. This is used both for generic code and because the message header contains the signature of the body.
Dbus has the following types:
| Type | Description | string |
|---|---|---|
| Byte | A single unsigned 8-bit integer | ‘y’ |
| Boolean | A boolean value | ‘b’ |
| Int16 | A signed 16-bit integer | ‘n’ |
| Uint16 | An unsigned 16-bit integer | ‘q’ |
| Int32 | A signed 32-bit integer | ‘i’ |
| Uint32 | An unsigned 32-bit integer | ‘l’ |
| Int64 | A signed 64-bit integer | ‘x’ |
| Uint64 | An unsigned 64-bit integer | ‘t’ |
| Double | A 64-bit IEEE 754 floating point number | ‘d’ |
| String | A string | ‘s’ |
| Object Path | A string with special meaning | ‘o’ |
| Signature | A type signature string | ‘s’ |
| UnixFd | A unix filedescriptor | ‘h’ |
These seem fairly unintimidating, so where is my headache coming from? Well, dbus wants to communicate more complex information too, so aside from these Basic Types there are also the more complex compound types. These are harder to deal with:
| Type | Description | string |
|---|---|---|
| Array(T) | An array containing an arbitrary number of elements of a given type T | “a” + typestring of T |
| Dict(K,V) | A dictionary with key type K and value type V | “a{” + typestring of K + typestring of V + “}” |
| Struct(T_1, …, T_n) | A tuple of values of types T_1, …, T_n | “(” + typestring of T_1 + … + typestring of T_n + “)” |
| Variant | Any Value of Any Type …. WHAT? | “v” |
These are more complex to serialize and especially that last one is a serious problem. It’s not like these aren’t encountered in the wild either. A dictionary of the form “a{sv}” is often used to pass optional arguments of different types. Array, Dict and Struct will cause difficulties in API design. Variant will make typesafety harder. But let’s talk about that in the next part.
Dbus Doesn’t actually have dictionaries as first-class types. There is only a dict element type and it’s only allowed to appear in arrays. For our purposes it is easier to consider dictionaries directly.
Not only does a dbus type need to be serialized but it also needs to have a certain byte alignment relative to the start of the message. The reason for this is fairly clear: computers usually read multi-byte values from aligned memory. This is what alignments look like:
| Type | Alignment | Commentary |
|---|---|---|
| Byte | 1 | aligned to its size |
| Boolean | 4 | actually serialized as a Uint32 |
| Int16 | 2 | aligned to its size |
| Uint16 | 2 | aligned to its size |
| Int32 | 4 | aligned to its size |
| Uint32 | 4 | aligned to its size |
| Int64 | 8 | aligned to its size |
| Uint64 | 8 | aligned to its size |
| Double | 8 | aligned to its size |
| String | 4 | we start by writing the length of the string as a Uint32, hence the alignment |
| Object Path | 4 | same as String |
| Signature | 1 | same as String but the length is only 1 byte |
| Array(T) | 4 | the alignment of its length. note that the array elements get alignment padding |
| Dict(K,V) | 4 | same as Array |
| Struct(T_1, …, T_n) | 8 | even if the alignment of the fields is less |
| Variant | 1 | the alignment of the type signature of the value, the value itself has its own alignment |
Every Value is preceded by enough 0 bytes to ensure their correct alignment but no more.
Now that we’ve talked about types we can talk about the message header. The message header is essentially just a block of serialized dbus values and it has the following signature:
“yyyyuua{yv}”
I know that this is slightly different to what the docs say, but it serializes identically and makes more sense this way.
But what do the fields actually mean?
| Byte | Description |
|---|---|
| 1 | the endianness of the message |
| 2 | the kind of message, there are 4 types of messages |
| 3 | 3 bitflags providing some auxiliary information |
| 4 | the major version of the dbus protocol. always 1 |
| 5 - 8 | The length of the message body in bytes |
| 9 - 12 | A message serial, this is used to correlate responses to a message |
| 13 - end | A dictionary containing several options that depend on the message type |
The main thing to keep in mind is that the header encodes the length of the message. This is incredibly desirable, especially for long messages since it allows the reader of the message to know know when to stop rather than falling into unbounded reading.
Every process that connects to the dbus message gets assigned a name. This is used to identify it among bus participants. A dbus client can ask for additional names like org.freedesktop.Notifications. This allows for different system components to indentify themselves as a generic component. If I want to send a notification to the notification daemon, I don’t need to know which daemon is running I already know what bus name it will have. A bus participant can have multiple names.
Dbus is very OOP-pilled but it is crucial that we understand this. The hierarchy goes as follows:
Code snippets in the following section will be missing certain parts I will leave out some runtime checks and I will leave out some struct fields. This is to remove some of the clutter and better illustrate my points.
Now that we’ve talked about some challenges, let’s talk about serializations. The first thing to notice here is that we can’t serialize the message directly into the socket. The protocol requires us to put the message length into the header and we cannot write the message body before the header.
What I wish to utilize is std.Io.Writer.Allocating from the zig standard library. It provides a writer interface backed by an allocator.
Dbus Messages care about both endianness and memory alignment. Here we are actually lucky: unlike c or c++, zig has good support for both in the standard library. we can freely use std.builtin.Endian and std.mem.Alignment. We can even automatically use our native endianness.
Implementing dbus Types is surprisingly hard. Ideally I want to be able to construct them in-place as function arguments and do type checking based on them. The issue is, that dbus types are also very much runtime values. I played around with several representations until I folded and settled on the following:
pub const Type = union(enum) {
Byte: void,
Boolean: void,
Int16: void,
Uint16: void,
Int32: void,
Uint32: void,
Int64: void,
Uint64: void,
Double: void,
String: void,
ObjectPath: void,
Signature: void,
UnixFd: void,
Variant: void,
tuple: []const @This(),
array: *const @This(),
dict: struct { *const Type, *const Type },
pub fn Array(comptime T: Type) Type {
return comptime .{ .array = &T };
}
pub fn Dictionary(comptime K: Type, comptime V: Type) Type {
comptime {
if (!K.isBasic()) @compileError(fmt.comptimePrint("dict keys need to be basic types, got {}", .{K}));
return .{ .dict = .{ &K, &V } };
}
}
pub fn Struct(comptime Ts: []const Type) Type {
return .{ .tuple = Ts };
}
}
A tagged union (basically an algebraic data type) approach allows us to recursively destructure our dbus type. At this point I have conceded that dbus type values are usually only going to be available at compile time. This brings some drawbacks in the flexibility of the API but it gives one huge benefit: type safety.
So we define the following member function:
pub fn toType(comptime T: Type) type {
switch (T) {
.Byte => return u8,
.Boolean => return bool,
.Int16 => return i16,
.Uint16 => return u16,
.Int32 => return i32,
.Uint32 => return u32,
.Int64 => return i64,
.Uint64 => return u64,
.Double => return f64,
.String, .ObjectPath => return []const u8,
.Signature => return []const u8,
.UnixFd => return UnixFd,
.tuple => |s| {
var out: [s.len]type = undefined;
for (&out, s) |*t, tt| t.* = tt.toType();
return std.meta.Tuple(&out);
},
else => @compileError(fmt.comptimePrint("cannot express dbus type {} as type", .{T})),
}
}
This is a surprise tool that we’ll use later
So let’s use all ofthis to put together a writer interface for messages. After some deliberation, this is what I came up with:
pub const MessageWriter = struct {
endian: std.builtin.Endian,
writer: Io.Writer.Allocating,
types: std.ArrayList(u8) = .empty
}
There is a field to remember our endianness, a field for our allocating writer and one to remember the type signature string of the message we’re writing;
So let’s start by writing values to the MessageWriter. We can define a function with the following signature:
pub fn write(writer: *@This(), comptime T: types.Type, value: T.toType()) !void
Type safety at last :). Note how this approach on one hand forces the type of the value we write to be known at compile time but on the other hand also guarantees that our write function stays fully typesafe.
While this works well for basic types it still hasn’t saved the issue for the more complex array types. While it would certainly be possible to simply define the zig type corresponding to Array(T) as []const T.toType(), this would be bad.
The data we wish to serialize may not be in the correct format, this could force expensive allocations. Let’s simply imagine a scenario where we have a value of a type like so:
[]const packed struct{lower: u8, higher: u8}
and we wish to serialize all lower bytes into a dbus type with this signature:
“ay”
This corresponds to
[]const u8
We can immediately see the issue: in order to get a slice to pass to the write function we’d need to allocate it — possibly on the heap — and since we’re going to write all of this data to a big allocated block anyway, all we do is copy the data over from the memory we just allocated.
This is a really crap way of using memory. Can we do better?
Let’s take our iterator approach to heart and write something that’s similar MessageWriter but for serializing and array. Since an array serialization needs to start with the length of the array (in bytes), my first instinct was another allocating reader. This however is wasteful. We allocate the entire array before copying it over to the main message. We are allocating memory again just to copy it elsewhere. We have moved the point at which we allocate but we haven’t addressed the issue.
There is another problem too. To see it we need to look at what its usage looked like:
var writer: MessageWriter = .{ .writer = .init(allocator) };
var array = writer.startArray(.Uint32);
try array.write(5);
try array.write(6);
try array.end();
Starting an array cannot error and does not need a try since we’ve not allocated anything yet. The more problematic thing cmes with array.end(). Since we not only need to write the length of the array but also copy over the array data we need to allocate in it. This causes array.end() to become fallible and there is nothing we can do about it. Failing on array.end() is pretty awkward, could we just not do that?
But how do we fix the issue of having to write the length of the array before writing its elements?
This is the bit where we remember that we’re not just writing to a socket. Every write of the MessageWriter writes into a continuous (usually) heap-allocated block of memory. The beautiful part about memory? We can write to it even after the initial allocation. So let’s simply write the array length first, remember the pointer and then simply write to the MessageWriter directly.
This is vastly superior: We prevent all auxiliary allocations, we don’t do any copying and array.end() is no longer fallible.
This is what the new API looks like:
var writer: MessageWriter = .{ .writer = .init(allocator) };
{
var array = try writer.startArray(.Uint32);
defer array.end();
try array.write(5);
try array.write(6);
}
Since ending the array can no longer fail, we can use defer to end the array at the end of the block. This allows us to be generally nicer about scoping the lifetime of our array writers. We can ensure that the array writer is out of scope by the time the array is done.
You will also have noticed a while ago that we’re using Zigs comptime functionality to embed the type of the array content directly into the type of the array writer. This gives us yet more type safety since we never need to make sure that we write the correct type into the array later. We specify the type of every value only once.
This is what the relevant parts of our ArrayWriter type look like (omitting the write function):
fn ArrayWriter(comptime T: types.Type) type {
return struct {
parent: *MessageWriter,
start: usize,
pub fn init(parent: *MessageWriter) !@This() {
try parent.writeNoType(.Uint32, 0);
return .{
.parent = parent,
.start = parent.writer.writer.end - 4,
};
}
pub fn end(writer: *const @This()) void {
var buf = writer.parent.writer.written();
mem.writeInt(
u32,
@ptrCast(buf[writer.start..][0..4]),
@intCast(buf.len - writer.start),
writer.parent.endian,
);
}
};
}
We can of course immediately see why end cannot fail. We even manage to reduce the amount the amount of data required for both the address of the array length and to compute the value to write there to a single field — pretty neat, huh?
We also note that this is technically a function returning a type but that’s simply how generics work in Zig. This is what the startArray method on MessageWriter looks like:
pub fn startArray(writer: *MessageWriter, comptime child: types.Type) !ArrayWriter(child) {
try writer.types.appendSlice(writer.writer.allocator,
comptime types.Type.getString(.Array(child)));
return .init(writer);
}
Since dictionaries are just fancy arrays, we can just add an analogous DictWriter. Nothing new is happening there aside from every write function having and additional parameter for the key value.
This is what startDict looks like:
pub fn startDict(writer: *MessageWriter,
comptime Key: types.Type,
comptime Val: types.Type) !DictWriter(Key, Val) {
try writer.types.appendSlice(writer.writer.allocator,
comptime types.Type.getString(.Dictionary(Key, Val)));
return .init(writer);
}
How do we serialize variants you ask? Well, variants simply consist of the signature of their contained type followed by the serialized value. We can just add another method to MessageWriter and win.
pub fn writeVariant(writer: *@This(), comptime T: types.Type, value: T.toType()) !void
Note that this too is typesafe.
To ensure alignment, we can simply define a padding function:
pub fn padAlignment(writer: *MessageWriter, alignment: mem.Alignment) !void {
const end = writer.writer.writer.end;
try writer.writer.writer.splatByteAll(0, alignment.forward(end) - end);
}
We’ll call this at every required place.
Now that we seem ready let’s try this out. I will skip over how we connect to the bus. Just remember that it’s a unix socket that we read and write from.
This is what our program looks like:
pub fn main(init: std.process.init) !void {
const io = init.io;
var conn = try Conn.connect(io, init.environ_map, .Session);
defer conn.disconnect(io);
try conn.authenticate(io);
var hello = conn.newCall(init.gpa, .{
.destination = "org.freedesktop.DBus",
.path = "/org/freedesktop/DBus",
.interface = "org.freedesktop.DBus",
.member = "Hello",
});
try conn.sendCall(io, &hello, null);
var message = conn.newCall(init.gpa, .{
.destination = "org.freedesktop.Notifications",
.path = "/org/freedesktop/Notifications",
.interface = "org.freedesktop.Notifications",
.member = "Notify",
});
var body = message.makeWriter();
try body.write(.String, "dtram");
try body.write(.Uint32, 0);
try body.write(.String, "");
try body.write(.String, "dtram test");
try body.write(.String, "we're winning!!!");
{
const actions = try body.startArray(.String);
defer actions.end();
}
{
var hints = try body.startDict(.String, .Variant);
defer hints.end();
try hints.write("urgency", .Byte, 1);
}
try body.write(.Int32, -1);
try conn.sendCall(io, &message, &body);
}
Let’s go through it step-by-step:
We fist establish a connection to the dbus session bus and authenticate.
var conn = try Conn.connect(io, init.environ_map, .Session);
defer conn.disconnect(io);
try conn.authenticate(io);
Then we send a Hello message to the session bus. This is important for us to be allowed to send more messages.
var hello = conn.newCall(init.gpa, .{
.destination = "org.freedesktop.DBus",
.path = "/org/freedesktop/DBus",
.interface = "org.freedesktop.DBus",
.member = "Hello",
});
try conn.sendCall(io, &hello, null);
And finally, we use the org.freedesktop.Notifications interface to send ourselves a desktop notification.
var message = conn.newCall(init.gpa, .{
.destination = "org.freedesktop.Notifications",
.path = "/org/freedesktop/Notifications",
.interface = "org.freedesktop.Notifications",
.member = "Notify",
});
var body = message.makeWriter();
try body.write(.String, "dtram");
try body.write(.Uint32, 0);
try body.write(.String, "");
try body.write(.String, "dtram test");
try body.write(.String, "we're winning!!!");
{
const actions = try body.startArray(.String);
defer actions.end();
}
{
var hints = try body.startDict(.String, .Variant);
defer hints.end();
try hints.write("urgency", .Byte, 1);
}
try body.write(.Int32, -1);
try conn.sendCall(io, &message, &body);
This really shows the writer interface in its full glory.
Sadly this moment of bliss was not my experience. Like most programs, I did not get it correct on the first try. But how do you even debug this? The session bus itself does not do much debug logging and even the dbus-monitor command only logs well-formed messages.
Clearly this is no way to live, but how do we inspect the bytes that we send? We can’t just write a simple socket dumper since the authentication part of the bus is interactive.
And I really don’t feel like writing my own mock-dbus server.
Suprisingly enough — after a bit of googling, I found the answer on stackoverflow. The socat utility (seems to be short-hand for socket cat) has a solution.
The following command will create a mock-socket that we can monitor:
$ socat -x UNIX-LISTEN:/run/user/1000/bus2 /run/user/1000/bus 2> log
We can now inject our proxy-socket via an environment variable:
$ export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus2
and finally run our program:
$ zig run dtram.zig
inevitably the program fails.
which is why we can repeat the same setup with a working dbus client:
$ socat -x UNIX-LISTEN:/run/user/1000/bus2 /run/user/1000/bus 2> log_working
$ export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus2
$ notify-send "Hello"
We can then go ahead and compare the hex values one by one.
This allows us to learn interesting things about the message protocol. Some of these I should have already known:
But most interestingly of all:
This means that if you have alignment padding at the start of the array (this is only possible if the array starts at an 8-byte alignment boundary and the array elements are 8-byte aligned. It just so happens that every dictionary is of this form), you will report the wrong array size and silently fail.
Incorrect Messages are punished via immediate socket closure.
Tune in next time when I deserialize messages (or write about something entirely different).