What ? PUSK ? What’s that ? This is the ProLiant USB Setup Key

This is a USB key you can now use to capture a hardware configuration of your HP ProLiant server (tested with G7 and Gen8). For that you just have to boot on the USB key and type “capture” at the boot prompt. Configuration is stored on the key, with the operation logs.

Then you can modify the conf files, or just use the single one we provide for what is really specific in a server (iLO credentials and IP conf), and redeploy that hardware configuration on a new server. For that just boot the new server on the key, and voilà !!

More over to deploy, you don’t need a keyboard, mouse, screen attached to the server, so if you are working in a place where your server is just electrically and networkly connected, that sufficient, and at the end the server will shutdown once the gardware configuration is done. Just restart it, and start controlling it remotely from the iLO do perform whatever further installation/customization you need to do.

So this is an easy way to have an operator perform the operation, doesn’t need Linux knowledge, nor platform knowledge. He just has to send the logs back to the dev team in case a problem occurs so they can debug.

And more over, thanks to HP, this is all GPLv2 Free Software

Now the important part, where to download it ?

The full 0.9.7 key is available at ftp://ftp.project-builder.org/PUSK/pusk-0.9.7.img

Just use dd to burn it onto your key and boot with it (WARNING: default mode is to deploy !)

For those of you who want to hack on the code, the entry point is at http://pusk.project-builder.org/browser/trunk and the Wiki (Home page) at http://pusk.project-builder.org

Hope you’ll find it useful. Let us know what you think of it.

I previously wrote a post with some calculations about the power supplied to laptops from a car battery [1]. A comment on the post suggested that I might have made a mistake in testing the Voltage because leaving the door open (and thus the internal lights on) will cause a Voltage drop.

So I’ve done some more tests:

In my previous tests I recorded 12.85V inside my car (from the front power socket which although having the same connector as a cigarette lighter isn’t designed for lighting cigarettes) and 13.02V from the battery terminals – a 0.17V difference. In my tests today I was unable to reproduce that but I think that my biggest mistake was to take the reading too quickly. Today I noticed that it took up to a minute for the Voltage to stabilise after opening a door (the Voltage dips after any current draw and takes time to recover) so a quick reading isn’t going to be accurate.

My car is a Kia Carnival which has two sockets in the front for power and for actually lighting cigarettes. The one for lighting cigarettes has a slightly lower Voltage and only works when the ignition is turned on. The car also has a power socket in the boot (the trunk for US readers) which delivers the same Voltage as the power socket in the front.

Also one thing to note is that today is a reasonably cold day (16.5C outside right now) and my car hasn’t been driven since last night so the battery would be quite cold (maybe 12C or less). My previous measurements were taken in summer so the battery would have been a lot warmer and therefore working more effectively.

Conclusion

The Voltage drop from turning on the internal lights surprised me, I had expected that a car battery which is designed to supply high current wouldn’t be affected by such things. Certainly not to give a 2% Voltage drop! The Voltage difference from reading inside the car and at the battery terminals might be partly due to the apparent lead coating on the terminals, I pushed the probes of my multimeter beneath the surface of the metal and got a really good connection.

The 14% Voltage increase when the engine was running was also a surprise. It seems to me that if you are running a power hungry device (such as a laptop) it would be a good idea to disconnect it when the engine is turned off. A 14% higher voltage will give a 14% lower current if the PSU is efficient and therefore less problems with heat in the wiring and less risk of blowing a fuse.

Also it’s a good idea to be more methodical about performing tests than I was before my last post. There are lots of other tests I could run (such as testing after the engine has been running for a while) but at the moment I don’t have enough interest in this topic to do more tests. Please leave a comment if there’s something interesting that you think I missed.

• [1] http://etbe.coker.com.au/2013/01/24/power-supplies-wires/

I’ve been part of many conference calls for work and found them seriously lacking. Firstly there’s a lack of control over the call, so when someone does something stupid like putting an unmuted phone handset near a noise source there’s no way to discover who did it and disconnect them.

Another problem is that of noise on the line when some people don’t mute their phones, which is related to the lack of control as it’s impossible to determine who isn’t muting their phone.

Possibly the biggest problem is how to determine who gets to speak next. When group discussions take place in person non-verbal methods are used to determine who gets to speak next. With a regular phone call (two people) something like the CSMACD algorithm for network packets works well. But when there are 8+ people involved it becomes time consuming to resolve issues of who speaks next even when there are no debates. This is more difficult for multinational calls which can have a signal round trip time of 700ms or more.

I think that we need a VOIP based conference call system for smart phones to manage this. I think that an ideal system would be based on the push to talk concept with software control that only allows one phone to transmit at a time. If someone else is speaking and you want to say something then you would push a button to indicate your desire but your microphone wouldn’t go live while the other person was speaking. The person speaking would be notified of your request and one of the following things would happen:

• You are added to the queue of people wishing to speak. When the other person finished speaking the next person in the queue gets a turn.
• You are added to the queue and the moderator of the call chooses who gets to speak next. This isn’t what I’d prefer but would probably be desired by managers for corporate calls.
• You get to interrupt the person who’s speaking. This may not be ideal but is similar to what currently happens.

Did I miss any obvious ways for the system to react to a talk request?

Is there any free software to do something like this? A quick search of the Google Play store didn’t find anything that seems to match.

Today I had the need to install OpenWrt on a Ubiquiti UniFi device. What I thought was going to be an easy process was unfortunately very painful due to bug 10597 in OpenWrt, which has gone unfixed for months.

The problem is that OpenWrt have the “magic” ID of “XM” identifying all the Ubiquiti firmware images, suitable for Nano M devices. UniFi devices, however, use the “BZ” magic. So the UniFi build of OpenWrt will not actually work on a stock UniFi device.

The fix is very simple — download the OpenWrt image builder, edit the target/linux/ar71xx/image/Makefile file, and change the line

$(eval $(call SingleProfile,UBNTXM,$(fs_64k),UBNTUNIFI,ubnt-unifi,UBNT-UF,ttyS0,115200,XM,XM,ar7240))

to instead be

$(eval $(call SingleProfile,UBNTXM,$(fs_64k),UBNTUNIFI,ubnt-unifi,UBNT-UF,ttyS0,115200,XM,BZ,ar7240))

and then run make image PROFILE=UBNTUNIFI to build the image. I flashed the resulting image (openwrt-ar71xx-generic-ubnt-unifi-squashfs-factory.bin) onto my UniFi by first scp’ing it over, and then typing “fwupdate.real -m /tmp/openwrt-ar71xx-generic-ubnt-unifi-squashfs-factory.bin -d“.

For those who don’t want to go through the process, especially if you are getting an error like “Invalid version ‘XM.ar7240.v6.0.0-OpenWrt-r36088′”, I have uploaded pre-made images to my site.

It’s the same stock 12.09 build with those two magic bytes tweaked so it actually works on the UniFi.

For the last couple of months I’ve been working for Infoxchange, a not-for-profit that provides technology to other not for profit. I’ve been working in the webapp development team, where we mostly work on webapps in the health and community sector using both Perl (for the older stuff) and Python/Django (for the newer stuff).

The government didn’t really work out for me, so this is a nice change. It’s relaxed, people wear jeans to work. We have fair trade tea and coffee and a delivery of CSA fruit every week.

It is busy though. We’ve got a lot going on, and not enough people to do it, so we’re looking for more. So if you’re a committed, talented Perl, Python or Web/Javascript programmer or devops who is in (or willing to move to) Melbourne, who wants to make a difference, you should get in touch with me. The pay is good (you will not be working for peanuts) and the team is fun.

We love open source, we contribute upstream, we have an organisation Github account. You can run what you want on your desktop. Developers have technical ownership over their work. Development is Agile.

Some technologies we love are Perl, Mojolicious, Python, Django, Javascript, jQuery, Bootstrap, Less, NodeJS, AngularJS, Github, Puppet, Debian and PostgreSQL. If you love any of those too, you should totally get in touch. If you love more technologies, bring those along too.

I’ve recently been spending a bit of my spare time playing Ingress (see the Wikipedia page if you haven’t heard of it). A quick summary is that Ingress is an Android phone game that involves geo-location of “portals” that you aim to control and most operations on a portal can only be performed when you are within 40 meters – so you do a lot of travelling to get to portals at various locations. One reasonably common operation that can be performed remotely is recharging a portal by using it’s key, after playing for a while you end up with a collection of keys which can be difficult to manage.

Until recently the set of portal keys was ordered alphabetically. This isn’t particularly useful given the fact that portal names are made up by random people who photograph things that they consider to be landmarks. If people tried to use a consistent geographic naming system (which was short enough to fit in large print on a phone display) then it would be really difficult to make it usable. But as joke names are accepted there’s just no benefit in having a sort by name.

A recent update to the Ingress client (the program which runs on the Android phone and is used for all game operations) changed the sort order to be by distance. This makes it really easy to see the portals which are near you (which is really useful) but also means that the order changes whenever you move – which isn’t such a good idea for use on a mobile phone. It’s quite common for Ingress players to recharge portals while on public transport. But with the new Ingress client the list order will change as you move so anyone who does recharging on a train will find the order of the list changing during the process and it’s really difficult to find items in a list which is in a different order each time you look at it.

This problem of ordering by location has a much greater scope than Ingress. One example is collections of GPS tagged photographs, it wouldn’t make any sense to mix the pictures of two different sets of holiday pictures because they were both taken in countries that are the same distance from my current location (as the current Ingress algorithm would do).

It seems to me that the best way of sorting geo-tagged items (Ingress portals, photos, etc) is to base it on the distance from a fixed point which the user can select. It could default to the user’s current location but in that case the order of the list should remain unchanged at least until the user returns to the main menu and I think it would be ideal for the order to remain unchanged until the user requests it.

I think that most Ingress players would agree with me that fixing annoying mis-features of the Ingress client such as this one would be better for the game than adding new features. While most computer games have some degree of make-work (in almost every case a computer could do things better than a person) I don’t think that finding things in a changing list should be part of the make-work.

Also it would be nice if Google released some code for doing this properly to reduce the incidence of other developers implementing the same mistakes as the Ingress developers in this regard.

A few related (or semi-related) bits and pieces which turned up over the last month or so. The first is a video I was sent from onlinemba.com, entitled Telecommuting is Good for You and Good for Business. It cites a few studies showing greater productivity, reduced turnover and eco-friendliness (which I would tend to agree with based on personal experience). Unfortunately I can’t seem to see links to the studies themselves, but you should go watch it anyway, because it’s one of those cute “live animated” things, which I’m a sucker for.

That leads neatly to the second thing, which is from the iiNet blog, entitled The benefits of working from home. As with the above video, Yahoo gets a prominent mention for not allowing telework (although reports on that seem to be somewhat mixed). Anyway, benefits cited include lack of commute, lower overhead and fewer distractions, but the blog post does also make the very good point that you need to figure out whether or not telework is actually right for you. It also offers some tips I tend to agree with.

Now we get to the semi-related bit. Assuming telework is right for you, you want a good internet connection. As I’ve said before, “Yay NBN! Bring it on!“. As much disenchantment as I have with the major parties, this is something Labor is actually doing right. For an easy-to-appreciate comparison of what Labor and the LNP are proposing, check out How Fast is the NBN. Teleworkers, pay particular attention to the simulated Dropbox sync.

It’s not very often that I separate mindi from mondo in the publication of releases. But this time it was needed as I had a customer who suffered from bugs that were only needing a mindi realease, and I thought it would help many other users ,so here you are !

Mindi 2.1.5 is there, and is principally solving kernel support detection for the type of initrd possible (solves an abort of mindi on RHEL3/4), and also reduces the number of error messages when dealing with links containing more than 2 references to .. Should help with some recent reports.

Also I had a report that the -H option and RESTORE keyword were not completely without interaction, so this is now solved as well.

Finally, this version supports better HP ProLiant Gen8 and future platforms by also using hp-rcu and hp-fm tools.

Now available on ftp://ftp.mondorescue.org for more than 120 distribution tuples ! And for those who ask why I do that: first because I like it, then because I have the tools to do it, and also because I do have users who are using Fedora 7, RHEL 3 or even Red Hat 6.2.

I’m looking for a PaaS provider that isn’t going to cost me very much (or anything at all) and supports Flask and PostGIS. Based on J5′s recommendation in my blog the other day, I created an OpenShift account.

A free account OpenShift gives you three small gears¹ which are individual containers you can run an app on. You can either run an app on a single gear or have it scale to multiple gears with load balancing. You then install components you need, which OpenShift refers to by the pleasingly retro name of cartridges. So for instance, Python 2.7 is one cartridge and PostgreSQL is another. You can either install all cartridges on one gear or on separate gears based on your resource needs².

You choose your base platform cartridge (i.e. Python-2.6) and you optionally give it a git URL to do an initial checkout from (which means you can deploy an app that is already arranged for OpenShift very fast). The base cartridge sets up all the hooks for setting up after a git push (you get a git remote that you can push to to redeploy your app). The two things you need are a root setup.py containing your pip requirements, and a wsgi/application file which is a Python blob containing an WSGI object named application. For Python it uses virtualenv and all that awesome stuff. I assume for node.js you’d provide a package.json and it would use npm, similarly RubyGems for Ruby etc.

There’s a nifty command line tool written in Ruby (what happened to Python-only Redhat?) that lets you do all the sort of cloud managementy stuff, including reloading cartridges and gears, tailing app logs and SSHing into the gear. I think an equivalent of dbshell would be really useful based on your DB cartridge, but it’s not a big deal.

There are these deploy hooks you can add to your git repo to do things like create your databases. I haven’t used them yet, but again it would make deploying your app very fast.

There are also quickstart scripts for deploying things like WordPress, Rails and a Jenkins server onto a new gear. Speaking of Jenkins there’s also a Jenkins client cartridge which I think warrants experimentation.

So what’s a bit crap? Why isn’t my app running on OpenShift yet? Basically because the available cartridges are a little antique. The supported Python is Python 2.6, which I could port my app too; or there are community-supported 2.7 and 3.3 cartridges, so that’s fine for me (TBH, I thought my app would run on 2.6) but maybe annoying for others. There is no Celery cartridge, which is what I would have expected, ideally so you can farm tasks out to other gears, and although you apparently can use it, there’s very little documentation I could find on how to get it running.

Really though the big kick in the pants is there is no cartridge for Postgres 9.2/PostGIS 2.0. There is a community cartridge you can use on your own instance of OpenShift Origin, but that defeats the purpose. So either I’m waiting for new Postgres to be made available on OpenShift or backporting my code to Postgres 8.4.

Anyway, I’m going to keep an eye on it, so stay tuned.

1. small gears have 1GB of disk and 512MB of RAM allocated
2. I think if you have a load balancing (scalable) application, your database needs to be on its own gear so all the other gears can access it.

Recently, professors at a San Jose State refused to use a lecture series by Michael Sandel at their university: it’s well worth reading their explanation of this decision. After a long and somewhat frustrating discussion about this, I think it’s worth teasing out some of the issues surrounding MOOCs. Much of this draws on the conversation which I just had, but mostly because these views are representative of much more widely-held opinions.

Image courtesy of opensourceway

There’s the assumption that just because something is ‘open source’, it must be good. This is tied to other assumptions about what openness means, such as the assumption that ‘open source’ necessarily means more participatory and more accessible. While MOOCs certainly have the potential to make interesting, useful, learning material widely available so that students (and others) can enrich their learning, we do need to bear in mind the context in which they’re being developed. Context matters. ‘Open source’ doesn’t necessarily mean ‘good’ in all contexts, because other considerations must be taken into account.

In this case, we need to remember that MOOCs are being developed in the context of cuts to university funding around the world, and in the context of university systems which tend to privilege publishing over teaching, with ever-increasing class sizes and workloads for lecturers. We’re seeing a massive casualisation of the workforce as we shift from full-time lecturers doing most of the teaching to the use of underpaid teaching assistants who are usually on short-term, precarious contracts. Funding for students is also limited, making it harder for students from disadvantaged backgrounds to get a university education (in the US far more than in Australia).

What does this mean for our evaluation of MOOCs? Firstly, we need to be aware of the probability that requests (or demands) that lecturers use content from MOOCs hosted at other universities are motivated more by a desire on the part of university management to cut costs than by a concern for quality teaching. Secondly, there is a strong chance that the use of lecture content from MOOCs will be used to justify further casualisation of the academic workforce on the basis that as the backbone of the unit is there, all that’s needed will be teaching assistants/tutors rather than full-time lecturers. Thirdly, this is likely to contribute to and reinforce the existing two-tier system (more so in the US than Australia): some students will have access to lecturers who develop units, have funding for research, and engage in hands-on teaching, while poorer students at under-resourced universities will get content developed elsewhere, taught by tutors who are unlikely to have the resources and support necessary to develop themselves as teachers and as researchers.

There’s also the issue of what we use as the standard. While I’m sure Sandel is an engaging lecturer with many valuable points to make, the outline for the ‘Justice’ unit which the San Jose professors declined to use states that, “principal readings for the course are texts by Aristotle, John Locke, Immanuel Kant, John Stuart Mill, and John Rawls.” In a unit covering “affirmative action, income distribution, same-sex marriage, the role of markets, debates about rights (human rights and property rights), arguments for and against equality, dilemmas of loyalty in public and private life”, it’s worth questioning whether a backbone consisting purely of dead white men is most appropriate.

Some of the books taught in Ethnic Studies units in Arizona – those usually left out of mainstream curricula.

Universities, and particularly the most prestigious and well-funded US universities, are still disproportionately accessible to privileged groups within society. If unit content is increasingly produced primarily by these universities, and then farmed out to other places, we are likely to hear a more and more narrow range of perspectives. The existing constraints on marginalised voices within academia will be reinforced: women and minority groups will, in all likelihood, be those who are pushed (further) into precarious employment as short-term teaching staff unable to create their own units.

I’m not against the idea of MOOCs. But we need to think about the broader context in which they’re developed, and take active steps to shape them in positive directions. We need to hold open spaces for participatory, accessible learning that values a diversity of voices – including those of both students and teachers. In order to do this, we can’t take the discourse of ‘openness’ associated with MOOCs at face value.

I’ve been working on the data collection part of my cycle route modelling. I’m hoping that I can, as a first output, put together a map of where people are cycling in Melbourne. A crowd-sourced view of the best places to cycle, if you will. Given I will probably be running this in the cloud¹, I thought it was best to actually store the data in a GIS database, rather than lots and lots of flat files.

A quick Google turned up GeoAlchemy, which are GIS extensions for SQLAlchemy. Provides lots of the standard things you want to do as methods on fields, but this is only a limited set of what you can do with PostGIS. Since I’m going to be wanting to do things like binning data, I thought it was worth figuring out how hard it was to call other PostGIS methods.

GeoAlchemy supports subclassing to create new dialects, but you have to subclass 3 classes, and it’s basically a pain in the neck when you just want to extend the functionality of the PostGIS dialect. Probably what I should do is submit a pull request with the rest of the PostGIS API as extensions, but I’m lazy. Henceforth, for the second time this week I am employing monkey patching to get the job done (and for the second time this week, kittens cry).

Functions in GeoAlchemy require two things, a method stub saying how we collect the arguments and the return (look at geoalchemy.postgis.pg_functions) and a mapping from this to the SQL function. Since we only care about one dialect, we can make this easier on ourselves by combining these two things. Firstly we monkeypatch in the method stubs:

from geoalchemy.functions import BaseFunction
from geoalchemy.postgis import pg_functions

@monkeypatchclass(pg_functions)
class more_pg_functions:
    """
    Additional functions to support for PostGIS
    """

    class length_spheroid(BaseFunction):
        _method = 'ST_Length_Spheroid'

Note the _method attribute which isn’t something used anywhere else. We can then patch in support for this:

from geoalchemy.dialect import SpatialDialect

@monkeypatch(SpatialDialect)
def get_function(self, function_cls):
    """
    Add support for the _method attribute
    """

    try:
        return function_cls._method
    except AttributeError:
        return self.__super__get_function(function_cls)

The monkeypatching functions look like this:

def monkeypatch(*args):
    """
    Decorator to monkeypatch a function into class as a method
    """

    def inner(func):
        name = func.__name__

        for cls in args:
            old = getattr(cls, name)
            setattr(cls, '__super__{}'.format(name), old)

            setattr(cls, name, func)

    return inner


def monkeypatchclass(cls):
    """
    Decorator to monkeypatch a class as a baseclass of @cls
    """

    def inner(basecls):
        cls.__bases__ += (basecls,)

        return basecls

    return inner

Finally we can do queries like this:

>>> track = session.query(Track).get(1)
>>> session.scalar(track.points.length_spheroid('SPHEROID["WGS 84",6378137,298.257223563]'))
6791.87502950043

Code on GitHub.

1. your recommendations for cloud-based services please, must be able to run Flask and PostGIS and be super cheap

I’m visiting my parents for the long weekend. Sitting in the airport I decided I should use my spare time to write some documentation, so sitting in the airport was the first time I’d tried to connect to work’s OpenVPN server. While it’s awesome that Network Manager can now import OpenVPN configs, it didn’t work because NM doesn’t support the crucial keysize parameter.

Rather than work around the problem, which some people have done, but would annoyingly break my other OpenVPNs, I used the fact that it’s open source to fix the problem properly.

My Dad asked if I was working. No, well, not really. I’m fixing the interface to my VPN client so I can connect to work’s VPN, I replied. Unglaublich! my father remarked. Not unbelievable, because it’s open source!

Shared Memory Fences

In our last adventure, dri3k first steps, one of the ‘future work’ items was to deal with synchronization between the direct rendering application and the X server. DRI2 “handles” this by performing a round trip each time the application starts using a buffer that was being used by the X server.

As DRI3 manages buffer allocation within the application, there’s really no reason to talk to the server, so this implicit serialization point just isn’t available to us. As I mentioned last time, James Jones and Aaron Plattner added an explicit GPU serialization system to the Sync extension. These SyncFences serializing rendering between two X clients, but within the server there are hooks provided for the driver to use hardware-specific serialization primitives.

The existing Linux DRM interfaces queue rendering to the GPU in the order requests are made to the kernel, so we don’t need the ability to serialize within the GPU, we just need to serialize requests to the kernel. Simple CPU-based serialization gating access to the GPU will suffice here, at least for the current set of drivers. GPU access which is not mediated by the kernel will presumably require serialization that involves the GPU itself. We’ll leave that for a future adventure though; the goal today is to build something that works with the current Linux DRM interfaces.

SyncFence Semantics

The semantics required by SyncFences is for multiple clients to block on a fence which a single client then triggers. All of the blocked clients start executing requests immediately after the trigger fires.

There are four basic operations on SyncFences:

• Trigger. Mark the fence as ready and wake up all waiting clients

• Await. Block until the fence is ready.

• Query. Retrieve the current state of the fence.

• Reset. Unset the fence; future Await requests will block.

SyncFences are the same as Events as provided by Python and other systems. Of course all of the names have been changed to keep things interesting. I’ll call them Fences here, to be consistent with the current X usage.

Using Pthread Primitives

One fact about pthreads that I recently learned is that the synchronization primitives (mutexes, barriers and semaphores) are actually supposed to work across process boundaries, if those objects are in shared memory mapped by each process. That seemed like a great simplification for this project; allocate a page of shared memory, map into the X server and direct rendering application and use the existing pthreads APIs.

Alas, the pthread objects are architecture specific. I’m pretty sure that when that spec was written, no-one ever thought of running multiple architectures within the same memory space. I went and looked at the code to check, and found that each of these objects has a different size and structure on x86 and x86_64 architectures. That makes it pretty hard to use this API within X as we often have both 32- and 64- bit applications talking to the same (presumably 64-bit) X server.

As a last resort, I read through a bunch of articles on using futexes directly within applications and decided that it was probably possible to implement what I needed in an architecture-independent fashion.

Futexes

Linux Futexes live in this strange limbo of being a not-quite-public kernel interface. Glibc uses them internally to implement locking primitives, but it doesn’t export any direct interface to the system call. Certainly they’re easy to use incorrectly, but it’s unusual in the Linux space to have our fundamental tools locked away ‘for our own safety’.

Fortunately, we can still get at futexes by creating our own syscall wrappers.

static inline long sys_futex(void *addr1, int op, int val1,
                 struct timespec *timeout, void *addr2, int val3)
{
    return syscall(SYS_futex, addr1, op, val1, timeout, addr2, val3);
}

For this little exercise, I created two simple wrappers, one to block on a futex:

static inline int futex_wait(int32_t *addr, int32_t value) {
    return sys_futex(addr, FUTEX_WAIT, value, NULL, NULL, 0);
}

and one to wake up all futex waiters:

static inline int futex_wake(int32_t *addr) {
    return sys_futex(addr, FUTEX_WAKE, MAXINT, NULL, NULL, 0);
}

Atomic Memory Operations

I need atomic memory operations to keep separate cores from seeing different values of the fence value, GCC defines a few such primitives and I picked _syncboolcompareandswap and _syncvalcompareandswap. I also need fetch and store operations that the compiler won’t shuffle around:

#define barrier() __asm__ __volatile__("": : :"memory")

static inline void atomic_store(int32_t *f, int32_t v)
{
    barrier();
    *f = v;
    barrier();
}

static inline int32_t atomic_fetch(int32_t *a)
{
    int32_t v;
    barrier();
    v = *a;
    barrier();
    return v;
}

If your machine doesn’t make these two operations atomic, then you would redefine these as needed.

Futex-based Fences

These wake-all semantics of Fences greatly simplify reasoning about the operation as there’s no need to ensure that only a single thread runs past Await, the only requirement is that no threads pass the Await operation until the fence is triggered.

A Fence is defined by a single 32-bit integer which can take one of three values:

• 0 - The fence is not triggered, and there are no waiters.
• 1 - The fence is triggered (there can be no waiters at this point).
• -1 - The fence is not triggered, and there are waiters (one or more).

With those, I built the fence operations as follows. Here’s Await:

int fence_await(int32_t *f)
{
    while (__sync_val_compare_and_swap(f, 0, -1) != 1) {
        if (futex_wait(f, -1)) {
            if (errno != EWOULDBLOCK)
                return -1;
        }
    }
    return 0;
}

The basic requirement that the thread not run until the fence is triggered is met by fetching the current value of the fence and comparing it with 1. Until it is signaled, that comparison will return false.

The compareandswap operation makes sure the fence is -1 before the thread calls futex_wait, either it was already -1 in the case where there were other waiters, or it was 0 before and is now -1 in the case where there were no waiters before. This needs to be an atomic operation so that the fence value will be seen as -1 by the trigger operation if there are any threads in the syscall.

The futex_wait call will return once the value is no longer -1, it also ensures that the thread won’t block if the trigger occurs between the swap and the syscall.

Here’s the Trigger function:

int fence_trigger(int32_t *f)
{
    if (__sync_val_compare_and_swap(f, 0, 1) == -1) {
        atomic_store(f, 1);
        if (futex_wake(f) < 0)
            return -1;
    }
    return 0;
}

The atomic compareandswap operation will make sure that no Await thread swaps the 0 for a -1 while the trigger is changing the value from 0 to 1; either the Await switches from 0 to -1 or the Trigger switches from 0 to 1.

If the value before the compareandswap was -1, then there may be threads waiting on the Fence. An atomic store, constructed with two memory barriers and a regular store operation, to mark the Fence triggered is followed by the futex_wake call to unblock all Awaiting threads.

The Query function is just an atomic fetch:

int fence_query(int32_t *f)
{
    return atomic_fetch(f) == 1;
}

Reset requires a compareandswap so that it doesn’t disturb things if the fence has already been reset and there are threads waiting on it:

void fence_reset(int32_t *f)
{
    __sync_bool_compare_and_swap(f, 1, 0);
}

A Request for Review

Ok, so we’ve all tried to create synchronization primitives only to find that our ‘obvious’ implementations were full of holes. I’d love to hear from you if you’ve identified any problems in the above code, or if you can figure out how to use the existing glibc primitives for this operation.

Maybe I’m wrong. I generally don’t relay that much the movements (justified !) that are happening on Internet around Open Source and Open Data. I’m much more in a mood to promote stuff rather than to rant against what is not working (with exceptions as everybody )

Now having recently been elected at the board of the AFUL association for the defence of Open Source and Linux, I also need to become more vocal with regards to these subjects, and some areas are really frightening so need more voices to support them.

If you look back in our short IT history, you can see that each time standards have been promoted (for small fees such as the PC, Unix, or for free such as the Internet, the W3C) it has allowed our industry to flourish and develop itself in an incomparable way. And of course, FLOSS has been a clear accelerator of the Internet development.

DRM are by nature incompatible with an Open Internet, and Open Source. HTML5 shows great promises, especially its new agentless video conferencing system. So we should keep what is good in it, and stop bloating it with useless and jail-full features.

Lots of entities have now publish a letter in order to promote a DRM free Web. Forward and promote these information to your own networks. You can sign the petition available at http://www.defectivebydesign.org/no-drm-in-html5/ if you agree with that vision.

For french speaking people, also read the latest LinuxFR.org article.

If you think that Internet can’t develop itself without open access to content, please act and sign this.

The stitching is based on the graph of the data that is in the table. By Jessica Kelly

There’s been a significant push in Internet Studies over the last few years towards ‘big data’ studies, which aggregate huge volumes of information (such as tweets or website linking patterns) and subject them to analysis, often quantitative analysis. Much of this research provides valuable insights into how people are using the Internet and its impacts on society, politics, and economics. At IR13 last year there were plenty of projects which took a ‘big data’ approach to the study of social movements, particularly the Arab Spring and Occupy, and provided important analysis about how they organise and communicate. And, of course, my collaborator on the Mapping Movements project, Tim Highfield, is doing excellent work in the area.

However, I do think that there are important aspects of this shift towards big data that we need to maintain a critical approach towards. Part of the reason why ‘big data’ is so appealing is that it looks like Science: there are numbers! and statistical analysis! There have been claims that it will allow us to ‘do away with the need for hypothesis and theory’ (presumably ridding ourselves of the biases contained in these processes). It fits within our perceptions of what ‘proper’ science should be: more objective, less reliant on qualitative methods like participant observation and interviews. This notion of science has, of course, been critiqued from a number of perspectives. Emily Martin’s ‘The Egg and the Sperm‘, for example, provided an excellent demonstration of how profoundly even ‘hard’, supposedly objective, science, is shaped by cultural assumptions, including those surrounding gender.

The shift towards ‘big data’ is not only linked to the uptake of new analytical tools, it is also linked to our (gendered) ideas of what science should look like. As more funding becomes available for big data research, it is important to bear in mind the ways in which our assumptions structure the value we place on different research, and the ways in which access to different research fields is gendered. While many women provide vital contributions in STEM fields, there continue to be significant structural barriers to participation by women and minority groups in these areas. Devaluing qualitative research in favour of quantitative big data not only builds on misplaced assumptions about the value of ‘hard sciences’, it also adds to the factors excluding marginalised perspectives from academia.

By Jessica Kelly

This is not to say that we should abandon big data approaches. As I said, I believe that they provide many helpful insights. There’s also some fascinating work out there that uses big data in ways that undermine the assumptions that this research must be ‘objective’ – Zizi Papacharissi and Maria de Fatime Oliviera’s work on affective publics springs to mind here. Tim and I are approaching the use of big data by drawing together big data approaches and participant observation, interviews, and other qualitative methods. So the issue is not so much whether we use big data, as whether we remain aware of the ways in which its use is structured by our assumptions about what constitutes ‘science’, and of ways in which this may privilege some groups’ participation over others.

I recently helped organise the inaugural Paris edition of the popular DevOpsDays series of conferences.  It was a great event, and I’d like to share some of my thoughts and observations here.

The DevOpsDays series of events is, as the name implies, centred around the “devops” movement, and is intended as a way to introduce people to this style of IT workflow, project, and people management.  Since this is not related to a programming language, this conference falls outside of the normal sorts of events that Mozilla generally finds itself involved in.  I believe this to be a very good thing, and I am proud to have represented Mozilla both as a sponsor of the event, and as a European devops community member.

The audience was mostly French in composition with a not-insignificant number of attendees from both Francophone and non-Francophone European countries.  Said audience was a healthy mix of developers, IT operations, and managers across a wide spectrum of company sizes, types, and even industries; frankly, I was impressed at how broad the composition was, and it was refreshing to see interest in the devops movement from such a wide group.

Though the event was held in Paris, all of the talks (with the exception of some of the ignites) were done in English.  This was by design – a internally contentious decision that, in my opinion, ultimately proved itself to be the correct one.  The open spaces during the afternoon were in a mix of English and French in order to ensure that everybody could participate equally.  Concerning the open spaces, we weren’t sure if the format would work here in France, but they were a smash success!  Everybody seemed to really enjoy the format as a platform for discussion, debate, and idea-generation.  I’d wager that for many of the attendees, it was the first time they’d ever been exposed to such a thing, and my hope is that they can bring the format to others in the future.

Since devops is so new to France, the majority of the presentations themselves were entry-level, and thus not particularly interesting to me directly.  That said, there were two presentations that really stood out (and would have held their own even at a more “advanced” event): “CustomerOps” by Alexis Lê-Quôc, and “Map & Territory” by Pierre-Yves Ritschard.

Alexis’ presentation on “CustomerOps” centred around the concept of providing customer support using engineering principles – and, indeed, delivered by engineers themselves.  This really hit home for me because in Mozilla IT/Ops, we’re not only the people who build and provide technical infrastructure, but are also the people who provide direct support to the consumers of that infrastructure – a situation that is absolutely not a given in many other companies (i.e. the admins and the customer reps are not the same people).  Alexis illustrated the importance of communication, and how to measure success (read: customer satisfaction) in meaningful ways.

Pierre-Yves’ presentation was based on a very interesting philosophical conjecture: that our mental model of the world is not the same as the reality said model attempts to describe.  Put another way, a map isn’t actually land, it’s a representation of the territory it describes (hence the title).  Therefore, the most valuable models are the ones that can describe reality in useful ways, and it’s in defining “usefulness” that the real effort must be made.  In a more applicable sense his thesis was simple: identify your “key metrics” – the numbers which literally describe the success or failure of your business – and make sure you are collecting, analysing, and modelling them above all.  Every other metric is either secondary or potentially uninteresting in the first place.

Personally, I spent a lot of time mingling with the attendees, talking about Mozilla, our projects, and our mission.  Generally speaking, the first question was, “Can I have one of those Firefox stickers?”, but the second question was, “When can I get my hands on a FirefoxOS phone?”  As usual, everybody wanted to see one, and (unfortunately), as usual, I didn’t have one to show them.  The more events that I attend on behalf of Mozilla, the more I realise that continues to be a wasted opportunity to promote our most exciting new project.  I’ll have to work on this going forward.

Of course, since this was a devops-related event, people were also very curious about if and how Mozilla is implementing devops internally.  The overarching theme of devops is communication, so this event was an excellent opportunity to talk about IT at Mozilla, and to promote not only our successes, but dig into our failures as well.  This sort of interaction is vital in order to avoid stagnation.

In summary, it was a fine showing for our first Parisian event, and I am looking forward to the next edition.  Hopefully I’ll see you there!

Here in Athens, I’m starting to get a much  better idea of who I need to speak to and where I need to go, in part because I’ve had some fortuitous introductions from friends-of-friends-of-friends. This aspect of research on social movements is, I think, often under-acknowledged. You can, when doing research, make a more-or-less disinterested decision about the case study that will serve your research project best, show up, and then make contact with activists through websites or by showing up to protests. But activists in many circumstances are understandable cautious about talking to strange people who show up out of nowhere. As well as the security concerns that accompany state surveillance of many movements, activists have many competing demands on their time and talking to academics who may retell the story of the movement in ways which activists aren’t comfortable with.

Gift economy by London Permaculture

So I’m tremendously grateful to the people who take some time to introduce me and to tell me who it’s important that I speak to. Our work in Tunisia relied heavily on connections from friends and colleagues, just as my work in Athens does. In return, I try to ensure that my work is relevant for activists, that I write in a way that’s clear and accessible rather than all wrapped up in academic jargon, and that my work is publicly-available (preferably for free as open-access publications). I’m not sure that I always succeed in each of these, but I do try.

As well as attempting to give something of worth back in return for the introductions, explanations, and time taken for interviews which activists give me, academia relies on a complex web of gifting. This morning I spent two and a half hours going through over two hundred applications for Adacamp SF, and after I finish this post I will review conference abstracts for the Internet Research: resistance and appropriation conference. In fact, none of the work I’m doing at the moment is paid, as I’m taking a break from teaching to do research. The book review I’m writing is unpaid, the three book chapters and one article I have due soon are unpaid, the research I’m currently doing is for a book that will only be published a long way down the line and is unlikely to bring me any substantial income in royalties.

‘mad-scientist type stuff’ by sarkasmo

Much of this ‘gifting’ is not entirely altruistic – it brings me benefits in one form or another, even if that isn’t financial (just as gifting does in other gift economies). Publishing is essential to getting an academic position, even a teaching-focused position (which is sad, given the importance of good teachers, and the extent to which this is gendered). Some of it also feeds into exploitative systems, like the use of academic articles and peer-reviews (both unpaid) to build a massively profitable academic journal business - one reason why I won’t do reviews for journals which aren’t open access.

I can’t do my work without the gifts – of time, energy, and reputation – that others lend me. And I wouldn’t want to do my work if I didn’t feel like I was gifting something in return (although sometimes I do worry that I’m giving the academic equivalent of an ugly sweater that will never get worn). The context of that gifting, and the relationships involved, need careful thought.

I’ve just got off a call to the UK Digital TV Group, for which I gave a talk on HTML5 video accessibility (slides best viewed in Google Chrome).

The slide provide a high-level summary of the accessibility features that we’ve developed in the W3C for HTML5, including:

• Subtitles & Captions with WebVTT and the track element
• Video Descriptions with WebVTT, the track element and speech synthesis
• Chapters with WebVTT for semantic navigation
• Audio Descriptions through synchronising an audio track with a video
• Sign Language video synchronized with a main video

I received some excellent questions.

The obvious one was about why WebVTT and not TTML. While for anyone who has tried to implement TTML support, the advantages of WebVTT should be clear, for some the decision of the browsers to go with WebVTT still seems to be bothersome. The advantages of CSS over XSL-FO in a browser-context are obvious, but not as much outside browsers. So, the simplicity of WebVTT and the clear integration with HTML have to speak for themselves. Conversion between TTML and WebVTT was a feature that was being asked for.

I received a question about how to support ducking (reduce the volume of the main audio track) when using video descriptions. My reply was to either use video descriptions with WebVTT and do ducking during the times that a cue is active, or when using audio descriptions (i.e. actual audio tracks) to add an additional WebVTT file of kind=metadata to mark the intervals in which to do ducking. In both cases some JavaScript will be necessary.

I received another question about how to do clean audio, which I had almost forgotten was a requirement from our earlier media accessibility document. “Clean audio” consists of isolating the audio channel containing the spoken dialog and important non-speech information that can then be amplified or otherwise modified, while other channels containing music or ambient sounds are attenuated. I suggested using the mediagroup attribute to provide a main video element (without an audio track) and then the other channels as parallel audio tracks that can be turned on and off and attenuated individually. There is some JavaScript coding involved on top of the APIs that we have defined in HTML, but it can be implemented in browsers that support the mediagroup attribute.

Another question was about the possibilities to extend the list of @kind attribute values. I explained that right now we have a proposal for a new text track kind=”forced” so as to provide forced subtitles for sections of video with foreign language. These would be on when no other subtitle or caption tracks are activated. I also explained that if there is a need for application-specific text tracks, the kind=”metadata” would be the correct choice.

I received some further questions, in particular about how to apply styling to captions (e.g. color changes to text) and about how closely the browser are able to keep synchronization across multiple media elements. The earlier was easily answered with the ::cue pseudo-element, but the latter is a quality of implementation feature, so I had to defer to individual browsers.

Overall it was a good exercise to summarize the current state of HTML5 video accessibility and I was excited to show off support in Chrome for all the features that we designed into the standard.

The folk at Packt Publishing sent me an e-copy of GNOME 3 Application Development Beginners Guide a month or so ago.

I’ve been putting off this review because I don’t think this is an very good book and it’s hard to write bad reviews.

First off, the book’s Javascript sections use Seed. I think this is an unconventional choice given that the shell and most of GNOME uses gjs. It had been my experience with the Javascript bindings that gjs was significantly more mature, a view which is confirmed by the fact that Seed has had very little development in the last 18 months.

The book does not seem to use GTK+ best practice, like using Gtk.Grid or Gtk.Application and not using c_new constructor. It is full of things like use of Vala’s [CCode] pragma, but I don’t see why. I felt important and powerful facilities in GLib like properties were not properly explained, especially property binding. There was also a lack of understanding, for example, referring to Timeout objects, which don’t exist (the structure you’re looking for is a Source).

I do like that it uses Anjuta. It’s a shame that it requires unexplained hacks to get things building.

The Clutter section was very poor. Comparing Clutter to GTK+ is simply not reasonable. Clutter is a scene graph API, which doesn’t really have a comparison in the GTK+ stack, which goes from drawing layer to widget layer with no intermediate layer. I immediately noticed the Clutter examples hardcoded layout instead of using a layout manager.

The multimedia section had the user installing non-free codecs. Then it uses alsasink and not auto*sink. It spends a lot of time setting up GStreamer pipelines, rather than using decodebin and playbin, maybe this improves understanding, but I think it mostly will lead to the creation of very rigid apps.

I stopped reading and started skimming at this point. I did again notice weird things like creating JSON using append methods and not the handy JSON-GLib. The examples of HTML5 applications with WebKit perhaps would explain why Seed, except the wrapper is written in Vala, so there’s no problem of conflicting JS engines (I think it works fine anyway, right?). Similarly the application accesses applications by looking in /usr/share/applications rather than libgnome-menus. Again, this will lead to very rigid apps that don’t work very well and doesn’t teach beginners the best practice for GNOME development.

There’s stuff that’s just weird, the system requirements are significantly more powerful than my last computer, on which I was doing GNOME development just fine. There is discussion of how to switch to GNOME Shell, as if it’s required, whereas you can develop GNOME apps in Unity and XFCE just fine.

The typesetting of the book is poor. The source code is weirdly indented and I feel like it lacked readability (there’s great syntax highlighting available for printed text). There are grammatical mistakes that really should have been picked up in editing. The screenshots are blurry in the PDF (looks like some kind of busted bilinear filtering?). Also I can see the resize indicator on the mouse. Generally these serve to make the book look unprofessional.

Finally I don’t think the book really leads the new developer into the community as the best source to get help, which they will undoubtedly need. Of course, the community has already produced some excellent tutorials, which I think new developers would be much better off with.

All up I’m giving it 1 star.

GNOME 3 Application Development: Beginner’s Guide: ★☆☆☆☆

Into the Fire came out yesterday, and the creators are asking people to embed it and distribute it widely. The documentary looks at the impacts of austerity on migrants in Greece, who are facing not only dire economic circumstances but also widespread racism.

Day 1

it turns out that this week wasn’t the best possible to hold a hackfest in Boston and Cambridge; actually, it was the worst. what was supposed to be the first day passed with us hacking in various locations, mostly from home, or hotel lobbies. nevertheless, there were interesting discussions on experimental work, like a rework of the drawing and content scrolling model that Alex is working on.

Day 2

or Day 1 redux

with the city-wide lockdown revoked, we finally managed to meet up at the OLPC offices and start the discussion on Wayland, input, and compatibility; we took advantage of Kristian attending so we could ask questions about client-side decorations, client-side shadows, and Wayland compatibility. we also discussed clipboard, and drag and drop, and the improvements in the API that will be necessary when we switch to Wayland — right now, both clipboard and DnD are pretty tied to the X11 implementation and API.

after lunch, the topic moved to EggListBox and EggFlowBox: scalability, selection, row containers, CSS style propagation, and accessibility.

we also went over a whole set of issues, like positioning popups; high resolution displays; input methods; integrating the Gd widgets into GTK+, and various experimental proposals that I’m sure will be reported by their authors on Planet GNOME soon. it was mostly high level discussion, to frame the problems and bring people up to speed with each problem space and potential/proposed solutions.

we’d all want to thank OLPC, and especially Walter Bender, for being gracious hosts at their office in Cambridge, even on a weekend and the GNOME Foundation.

this is a PSA: if you’re thinking about submitting a talk for GUADEC 2013 in Brno, you have a week to do so.

In my previous post, I wrote about the growing neonazi movement in Greece and links with the police. However, I also mentioned that there were many signs of resistance to this. This weekend, antiracist and antifascist groups from Greece and around the world will hold a variety of actions that remember the coup which brought the junta to power on 21st April, 1967. I’m trying to put together an overview of some of these events, and I thought I’d share a little of it (perhaps some foreigners in Athens this weekend may even appreciate some English information) – click the pictures for more information (in Greek): [Please excuse any terrible translations, my Greek is struggling with political jargon and slang - feel free to add corrections in the comments, or to suggest other events]

Throughout the weekend, the Embros Theatre in Psirri will host a range of activities. These include documentaries, discussions, and workshops. It’s also great to see that there are many activities for children, especially on the Saturday.

On the 19th and the 20th, there will be events at Antiviosi [Antibiotic] Squat from 7pm on. On the 19th, this will focus on experiences from the Russian antifascist movement, and they will show the documentary “Antifa attitude”. On the 20th, there will be a discussion around ‘stories from the extremes’, with a video from Antifa Salonica.

On the 21st, a gathering will be held in Messina Square at 11:00 in the morning.

On the 21st, there will be speeches and a concert in Freedom Park on Vasilissis Sofias Avenue from 5pm onwards.

Having completed our work in Tunisia, I’m now in Athens for the next part of the ‘Mapping Movements’ project that Tim Highfield and I are working on. This work looks at the relationships between online and offline spaces, connecting quantitative and qualitative methods to get a better understanding of how activists are using digital technologies. Obviously (for those who know me and perhaps also those who have read my ‘about’ page), this is not just a matter of an abstract academic question: I’m interested in these issues as an activist as well as an academic, because we need to know what works and what doesn’t (and when), and how to be as effective as possible. It’s also important for people outside movements to understand them, to know what they’re facing.

‘Crush the fascists in every neighbourhood’ (if my dodgy greek is right)

While in Greece, I want to learn more about the antifascist organising here. Before I came, I had some sense of what was going on. Golden Dawn, a neonazi party that holds seats in the Greek parliament, are not only engaging in racist rhetoric – they are also involved in violent acts against immigrants and anarchists, as well as (at the very least) threatening queer people. Meanwhile, there is evidence of a close connection between the Greek police and Golden Dawn, and a recent Amnesty Report shows evidence of widespread and systematic police violence.

Reading about this is bad enough, but it is a different experience walking the streets of Athens. There’s a lot of racist graffiti. Sometimes just the Golden Dawn initials or name, but there’s also more explicit messages. One neighbourhood I walked through was full of “kill blacks”. A lot of this has been crossed out or covered with antifascist messages, thankfully, but I can’t imagine how scary it must be to be visibly non-white or, worse, non-white and in a precarious situation, in Greece right now. (Yesterday’s attacks on migrant workers in Nea Manolada who asked for their unpaid back wages are just one manifestation of this.)

Greek riot (and everywhere-else) police

The police are also very present. In Perth I’m not used to seeing large blocks of police – despite the massive over-policing of CHOGM in 2011 and the huge police presence at Lizard’s Revenge in South Australia last year, I’ve been lucky enough to mostly avoid spending too much time in the presence of heavily-armed riot police. My time with Occupy Oakland has done little to lower the anxiety about police that I inherited from my parents (who grew up in apartheid South Africa). So when I see a whole heap of police, all with riot shields and clubs and body armour, and a caged police bus, I tend to think something big must be just around the corner. In fact, it looks like the police bring out this kind of presence for demonstrations of any size. Yesterday I came across a demonstration by hospital workers against unpaid wages and cuts to services: everyone seemed very polite, and I didn’t see any trouble. Nevertheless, there were at least two busloads of police. Someone I spoke to at the protest said, “maybe they want people to think we are dangerous?”

(based on an illustration by Clay Rodery)

It was a similar case today at the action against the recent shut-down of Athens Indymedia and two radio stations: maybe thirty people at the demonstration, at least one busload of police and a few squads on motorbikes. This is the other part of the story. While the Greek government rejects calls to ban Golden Dawn for its racist rhetoric and action, Athens Indymedia and two radio stations have been shut down, apparently after pressure from the Greek Ministry of Public Order. It is important, for understanding this, to bear in mind claims that there are links between Golden Dawn and more centrist political parties.

Indymedia Athens have called for a week of international solidarity actions to highlight the shutdown, pointing out that while they may have technical solutions (such as hosting Indymedia on TOR) we cannot ignore or just ‘route around’ government attempts to silent dissent.

More broadly, there is the question of what those overseas might do to help deal with the spread of fascism in Greece. One person at the protest suggested that foreigners should boycott Greece, since tourist dollars don’t help those who are having the hardest time anyway. I’ve been thinking about this more. Individual action wouldn’t work – simply deciding not to go to Greece sends no message. A boycott would need to have specific demands (such as independent investigations and action taken on police brutality and racism). In the meantime, at the very least it is important to watch what’s happening here, because this is not just Greece’s problem: racism spreads when unchecked, and Golden Dawn is now looking to connect with racists and neonazis in other countries (including Australia). We can’t afford to step back and treat this as something happening only ‘over there’.

Young women for transitional justice at the opening march of the WSF

As Tunisia deals with the legacy of the Ben Ali regime, the framework of transitional justice is being used by many within the government and in civil society to guide the transformation. This includes the creation of a Tunisian Ministry of Human Rights and Transitional Justice, the involvement of international groups such as the ICTJ, and a plethora of local groups which have begun to work under the umbrella of transitional justice. There is considerable debate surrounding the transitional justice process in Tunisia, especially around the specific form the process should take.

One of my friends, Dr. Christalla Yakinthou, has considerable experience working in the practice and theory of transitional justice, and while we were in Tunisia we were interested in looking more closely at the process. In order to connect our interests, we focused specifically on the shifts in Internet governance which have happened since the fall of the Ben Ali regime. Internet governance is not traditionally seen as being a part of transitional justice, but it’s a vital part of the process of rebuilding and dealing with past human rights abuses.

As Tunisians struggle to deal with the ongoing challenges of reconfiguring the state, including dealing with ongoing police violence and IMF ‘reforms‘ that are unlikely to help the Tunisian people, Internet governance seems to be one area in which definite progress has been made. We talked to a range of people, including activists, bureaucrats, and those working within the Tunisian Internet Agency: most were very pleased with the changes made so far, and the way in which the legacy of online surveillance and censorship had been dealt with.

This doesn’t mean that the Internet freedom is secure, of course, just as it isn’t secure in Australia or other nominal democracies.. While the previous mechanisms of censorship and surveillance have been largely dismantled, Tunisian courts continue to attempt censorship or other coercive measures to silence dissent, as well as ‘objectionable content’ such as pornography. Concerns also exist about the potential affects of the intellectual property provisions slated for the new Tunisian constitution, and the creation of a new ‘cybercrime‘ unit.

As we wind up the final interviews in the first phase of the research, I’ll be hoping for the best for those in Tunisia working on these issues. Everyone we spoke to was very generous with their time, especially given what a busy period it is there. We have some great material, and I’m looking forward to putting it together for publication. It’ll be interesting: it’s the first time I’ve focused on such a state-centric process. I think that there are important critiques to be made of the transitional justice process, and particularly of the top-down nature of much of the work in the area. While there are great hopes for the transitional justice framework, it’s important to understand it as part of broader structures, including the coercive mechanisms of the state and the international system. At the same time, many Tunisians are attempting to mould existing frameworks to meet their own needs, including by questioning existing models for transitional justice.

Carrying on from my last post, the failed chef-client run came down to the init script in ceph 0.56 not yet knowing how to iterate /var/lib/ceph/{mon,osd,mds} and automatically start the appropriate daemons. This functionality seems to have been introduced in 0.58 or so by commit c8f528a. So I gave it another shot with a build of ceph 0.60.

On each of my ceph nodes, a bit of upgrading and cleanup. Note the choice of ceph 0.60 was mostly arbitrary, I just wanted the latest thing I could find an RPM for in a hurry. Also some of the rm invocations won’t be necessary, depending on what state things are actually in:

# zypper ar -f http://download.opensuse.org/repositories/home:/dalgaaf:/ceph:/extra/openSUSE_12.3/home:dalgaaf:ceph:extra.repo
# zypper ar -f http://gitbuilder.ceph.com/ceph-rpm-opensuse12-x86_64-basic/ref/next/x86_64/ ceph.com-next_openSUSE_12_x86_64
# zypper in ceph-0.60
# kill $(pidof ceph-mon)
# rm /etc/ceph/*
# rm /var/run/ceph/*
# rm -r /var/lib/ceph/*/*

That last gets rid of any half-created mon directories.

I also edited the Ceph environment to only have one mon (one of my colleagues rightly pointed out that you need an odd number of mons, and I had declared two previously, for no good reason). That’s knife environment edit Ceph on my desktop, and set "mon_initial_members": "ceph-0" instead of "ceph-0,ceph-1".

I also had to edit each of the nodes, to add an osd_devices array to each node, and remove the mon role from ceph-1. That’s knife node edit ceph-0.example.com then insert:

  "normal": {
    ...
    "ceph": {
      "osd_devices": [  ]
    }
  ...

Without the osd_devices array defined, the osd recipe fails (“undefined method `each_with_index’ for nil:NilClass”). I was kind of hoping an empty osd_devices array would allow ceph to use the root partition. No such luck, the cookbook really does expect you to be doing a sensible deployment with actual separate devices for your OSDs. Oh, well. I’ll try that another time. For now at least I’ve demonstrated that ceph-0.60 does give you what appears to be a clean mon setup when using the upstream cookbooks on openSUSE 12.3:

knife ssh name:ceph-0.example.com -x root chef-client
[2013-04-15T06:32:13+00:00] INFO: *** Chef 10.24.0 ***
[2013-04-15T06:32:13+00:00] INFO: Run List is [role[ceph-mon], role[ceph-osd], role[ceph-mds]]
[2013-04-15T06:32:13+00:00] INFO: Run List expands to [ceph::mon, ceph::osd, ceph::mds]
[2013-04-15T06:32:13+00:00] INFO: HTTP Request Returned 404 Not Found: No routes match the request: /reports/nodes/ceph-0.example.com/runs
[2013-04-15T06:32:13+00:00] INFO: Starting Chef Run for ceph-0.example.com
[2013-04-15T06:32:13+00:00] INFO: Running start handlers
[2013-04-15T06:32:13+00:00] INFO: Start handlers complete.
[2013-04-15T06:32:13+00:00] INFO: Loading cookbooks [apache2, apt, ceph]
[2013-04-15T06:32:13+00:00] INFO: Processing template[/etc/ceph/ceph.conf] action create (ceph::conf line 6)
[2013-04-15T06:32:13+00:00] INFO: template[/etc/ceph/ceph.conf] updated content
[2013-04-15T06:32:13+00:00] INFO: template[/etc/ceph/ceph.conf] mode changed to 644
[2013-04-15T06:32:13+00:00] INFO: Processing service[ceph_mon] action nothing (ceph::mon line 23)
[2013-04-15T06:32:13+00:00] INFO: Processing execute[ceph-mon mkfs] action run (ceph::mon line 40)
creating /var/lib/ceph/tmp/ceph-ceph-0.mon.keyring
added entity mon. auth auth(auid = 18446744073709551615 key=AQC8umZRaDlKKBAAqD8li3u2JObepmzFzDPM3g== with 0 caps)
ceph-mon: mon.noname-a 192.168.4.118:6789/0 is local, renaming to mon.ceph-0
ceph-mon: set fsid to f80aba97-26c5-4aa3-971e-09c5a3afa32f
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-ceph-0 for mon.ceph-0
[2013-04-15T06:32:14+00:00] INFO: execute[ceph-mon mkfs] ran successfully
[2013-04-15T06:32:14+00:00] INFO: execute[ceph-mon mkfs] sending start action to service[ceph_mon] (immediate)
[2013-04-15T06:32:14+00:00] INFO: Processing service[ceph_mon] action start (ceph::mon line 23)
[2013-04-15T06:32:15+00:00] INFO: service[ceph_mon] started
[2013-04-15T06:32:15+00:00] INFO: Processing ruby_block[tell ceph-mon about its peers] action create (ceph::mon line 64)
mon already active; ignoring bootstrap hint

[2013-04-15T06:32:16+00:00] INFO: ruby_block[tell ceph-mon about its peers] called
[2013-04-15T06:32:16+00:00] INFO: Processing ruby_block[get osd-bootstrap keyring] action create (ceph::mon line 79)
2013-04-15 06:32:16.872040 7fca8e297780 -1 monclient(hunting): authenticate NOTE: no keyring found; disabled cephx authentication
2013-04-15 06:32:16.872042 7fca8e297780 -1 unable to authenticate as client.admin
2013-04-15 06:32:16.872400 7fca8e297780 -1 ceph_tool_common_init failed.
[2013-04-15T06:32:18+00:00] INFO: ruby_block[get osd-bootstrap keyring] called
[2013-04-15T06:32:18+00:00] INFO: Processing package[gdisk] action upgrade (ceph::osd line 37)
[2013-04-15T06:32:27+00:00] INFO: package[gdisk] upgraded from uninstalled to
[2013-04-15T06:32:27+00:00] INFO: Processing service[ceph_osd] action nothing (ceph::osd line 48)
[2013-04-15T06:32:27+00:00] INFO: Processing directory[/var/lib/ceph/bootstrap-osd] action create (ceph::osd line 67)
[2013-04-15T06:32:27+00:00] INFO: Processing file[/var/lib/ceph/bootstrap-osd/ceph.keyring.raw] action create (ceph::osd line 76)
[2013-04-15T06:32:27+00:00] INFO: entered create
[2013-04-15T06:32:27+00:00] INFO: file[/var/lib/ceph/bootstrap-osd/ceph.keyring.raw] owner changed to 0m
[2013-04-15T06:32:27+00:00] INFO: file[/var/lib/ceph/bootstrap-osd/ceph.keyring.raw] group changed to 0
[2013-04-15T06:32:27+00:00] INFO: file[/var/lib/ceph/bootstrap-osd/ceph.keyring.raw] mode changed to 440
[2013-04-15T06:32:27+00:00] INFO: file[/var/lib/ceph/bootstrap-osd/ceph.keyring.raw] created file /var/lib/ceph/bootstrap-osd/ceph.keyring.raw
[2013-04-15T06:32:27+00:00] INFO: Processing execute[format as keyring] action run (ceph::osd line 83)
creating /var/lib/ceph/bootstrap-osd/ceph.keyring
added entity client.bootstrap-osd auth auth(auid = 18446744073709551615 key=AQAOl2tR0M4bMRAAatSlUh2KP9hGBBAP6u5AUA== with 0 caps)
[2013-04-15T06:32:27+00:00] INFO: execute[format as keyring] ran successfully
[2013-04-15T06:32:28+00:00] INFO: Chef Run complete in 14.479108446 seconds
[2013-04-15T06:32:28+00:00] INFO: Running report handlers
[2013-04-15T06:32:28+00:00] INFO: Report handlers complete

Witness:

ceph-0:~ # rcceph status
=== mon.ceph-0 ===
mon.ceph-0: running {"version":"0.60-468-g98de67d"}

On the note of building an easy-to-deploy Ceph appliance, assuming you’re not using Chef and just want something to play with, I reckon the way to go is use config pretty similar to what would be deployed by this Chef cookbook, i.e. an absolute minimal /etc/ceph/ceph.conf, specifying nothing other than initial mons, then use the various Ceph CLI tools to create mons and osds on each node and just rely on the init script in Ceph >= 0.58 to do the right thing with what it finds (having to explicitly specify each mon, osd and mds in the Ceph config by name always bugged me). Bonus points for using csync2 to propagate /etc/ceph/ceph.conf across the cluster.

I’ve recently completed a couple of research projects in Tunisia, which was, on the whole a very good experience. Tunisians are currently in the process of trying to change the direction of their country, which is of course a huge task, and I really appreciate the time and energy that people gave to help us and to talk to us about their work. However, the constant harassment that my friend/colleague/collaborator and I faced really wore me down after a while.

The harassment ranges from cars honking as they drive past to men walking behind me and whispering in my ear, “hey, beautiful!” to casual “ca va?”s to shouts of “hey, wanna fuck me?” to very obvious leering to ‘friendly’ attempts to talk us by men who wouldn’t leave when we politely (and then less politely) told them we weren’t interested to cars that slowed and followed us when we were walking down the street at night. And this is constant. In the space of a few metres in a busy area, we might have four or five groups of men shout at us.

I want to be clear here: this is not limited to Tunisia. It happens plenty in Perth, although most of the time when I’m walking or cycling around there I’m wearing headphones so I miss it. I got far more harassment in Tunisia than I get in Australia, partly because I’m obviously foreign. But then, the situation is reversed in Australia: Aboriginal Australians face constant racist harassment in Australia (including from the police), and many others (including Australian citizens who aren’t Anglo-Saxon or who speak a language other than English) face outright racist abuse or more subtle racism. And this is not to mention the sexism that even relatively privileged women in Australia face.

I don’t often write about this aspect of my work, and perhaps I should. As I get more confident as a researcher, I want to write more about the process, to be more present within the final published piece. For now, this is a start. The work is not only the interviews that will make it into the final publication, but also this context that surrounds them: trying different strategies for dealing with the harassment (ignoring it, shouting back, wondering if shouting back will lead to trouble). Not wanting to leave the hotel, some days, because I was just too sick of dealing with it.

And, at the same time, being very aware of my privilege, being aware that I am lucky enough to have access to international travel, and that I have a voice (however small) within the authority of academia. Knowing that however unpleasant I may find street harassment, my work is temporary and soon I will be elsewhere, and trying to present an analysis that will somehow be useful in dealing with all this.

I’m not going anywhere in particular with this. I’m sick, and very tired, and in a new country with new challenges. So it’s best to finish by letting the last words go to introducing awesome Tunisian feminists, who like all Tunisian women deal with this every day and are both in a better position to understand the situation there and to work out what to do about it: check out Feminism Attack!

[If you know of any feminist groups working on street harassment in Tunisia who need a signal boost, feel free to mention them in the comments and I'll add them in here.]

DRI3K — First Steps

Here’s an update on DRI3000. I’ll start by describing what I’ve managed to get working and then summarize discussions that happened on the xorg-devel mailing list.

Private Back Buffers

One of the big goals for DRI3000 is to finish the job of moving buffer management out of the X server and into applications. The only thing still allocated by DRI2 in the X server are back buffers; everything else moved to the client side. Yes, I know, this breaks the GLX requirement for sharing buffers between applications, but we just don’t care anymore.

As a quick hack, I figured out how to do this with DRI2 today — allocate our back buffers separately by creating X pixmaps for them, and then using the existing DRI2GetBuffersWithFormat request to get a GEM handle for them.

Of course, now that all I’ve got is a pixmap, I can’t use the existing DRI2 swap buffer support, so for now I’m just using CopyArea to get stuff on the screen. But, that works fine, as long as you don’t care about synchronization.

Handling Window Resize

The biggest pain in DRI2 has been dealing with window resize. When the window resizes in the X server, a new back buffer is allocated and the old one discarded. An event is delivered to ‘invalidate’ the old back buffer, but anything done between the time the back buffer is discarded and when the application responds to the event is lost.

You can easily see this with any GL application today — resize the window and you’ll see occasional black frames.

By allocating the back buffer in the application, the application handles the resize within GL; at some point in the rendering process the resize is discovered, and GL creates a new buffer, copies the existing data over, and continues rendering. So, the rendered data are never lost, and every frame gets displayed on the screen (although, perhaps at the wrong size).

The puzzle here was how to tell that the window was resized. Ideally, we’d have the application tell us when it received the X configure notify event and was drawing the frame at the new size. We thought of a cute hack that might do this; track GL calls to change the viewport and make sure the back buffer could hold the viewport contents. In theory, the application would receive the X configure notify event, change the viewport and render at the new size.

Tracking the viewport settings for an entire frame and constructing their bounding box should describe the size of the window; at least it should describe the intended size of the window.

There’s at least one serious problem with this plan — applications may well call glClear before calling glViewport, and as glClear does not use the current viewport, instead clearing the “whole” window, we couldn’t use the viewport as an indication of the current window size.

However, what this exercise did lead us to realize was that we don’t care what size the window actually is, we only care what size the application thinks it is. More accurately, the GL library just needs to be aware of any window configuration changes before the application, so that it will construct a buffer that is not older than the application knowledge of the window size.

I came up with two possible mechanisms here; the first was to construct a shared memory block between application and X server where the X server would store window configuration changes and signal the application by incrementing a sequence number in the shared page; the GL library would simply look at the sequence number and reallocate buffers when it changed.

The problem with the shared memory plan was that it wouldn’t work across the network, and we have a future project in mind to replace GLX indirect rendering with local direct rendering and PutImage which still needs accurate window size tracking. More about that project in a future post though…

X Events to the Rescue

So, I decided to just have the X server send me events when the window size changed. I could simply use the existing X configure notify events, but that would require a huge infrastructure change in the application so that my GL library could get those events and have the application also see them. Not knowing what the application is up to, we’d have to track every ChangeWindowAttributes call and make sure the event_mask included the right bits. Ick.

Fortunately, there’s another reason to use a new event — we need more information than is provided in the ConfigureNotify event; as you know, the Swap extension wants to have applications draw their content within a larger buffer that can have the window decorations placed around it to avoid a copy from back buffer to window buffer. So, our new ConfigureNotify event would also contain that information.

Making sure that ConfigureNotify event is delivered before the core ConfigureNotify event ensures that the GL library should always be able to know about window size changes before the application.

Splitting the XCB Event Stream

Ok, so I’ve got these new events coming from the X server. I don’t want the application to have to receive them and hand them down to the GL library; that would mean changing every application on the planet, something which doesn’t seem very likely at all.

Xlib does this kind of thing by allowing applications to stick themselves into the middle of the event processing code with a callback to filter out the events they’re interested in before they hit the main event queue. That’s how DRI2 captures Invalidate events, and it “works”, but using callbacks from the middle of the X event processing code creates all kinds of locking nightmares.

As discussed above, I don’t care when GL sees the configure events, as long as it gets them before the application finds about about the window size change. So, we don’t need to synchronously handle these events, we just need to be able to know they’ve arrived and then handle them on the next call to a GL drawing function.

What I’ve created as a prototype is the ability to identify specific events and place them in a separate event queue, and when events are placed in that event queue, to bump a ‘sequence number’ so that the application can quickly identify that there’s something to process.

Making the Event Mask Per-API Instead of Per-Client

The problem described above about using the core ConfigureNotify events made me think about how to manage multiple APIs all wanting to track window configuration. For core events, the selection of which events to receive is all based on the client; each client has a single event mask, and each client receives one copy of each event.

Monolithic applications work fine with this model; there’s one place in the application selecting for events and one place processing them. However, modern applications end up using different APIs for 3D, 2D and media. Getting those libraries to cooperate and use a common API for event management seems pretty intractable. Making the X server treat each API as a separate entity seemed a whole lot easier; if two APIs want events, just have them register separately and deliver two events flagged for the separate APIs.

So, the new DRI3 configure notify events are created with their own XID to identify the client-side owner of the event. Within the X server, this required a tiny change; we already needed to allocate an XID for each event selection so that it could be automatically cleaned up when the client exited, so the only change was to use the one provided by the client instead of allocating one in the server.

On the wire, the event includes this new XID so that the library can use it to sort out which event queue to stick the event in using the new XCB event stream splitting code.

Current Status

The above section describes the work that I’ve got running; with it, I can run GL applications and have them correctly track window size changes without losing a frame. It’s all available on the ‘dri3’ branches of my various repositories for xcb proto, libxcb, dri3proto and the X server.

Future Directions

The first obvious change needed is to move the configuration events from the DRI3 extension to the as-yet-unspecified new ‘Swap’ extension (which I may rename as ‘Present’, as in ‘please present this pixmap in this window’). That’s because they aren’t related to direct rendering, but rather to tracking window sizes for off-screen rendering, either direct, indirect or even with the CPU to memory.

DRI3 and Fences

Right now, I’m not synchronizing the direct rendering with the CopyArea call; that means the X server will end up with essentially random contents as the application may be mid-way through the next frame before it processes the CopyArea. A simple XSync call would suffice to fix that, but I want a more efficient way of doing this.

With the current Linux DRI kernel APIs, it is sufficient to serialize calls that post rendering requests to the kernel to ensure that the rendering requests are themselves serialized. So, all I need to do is have the application wait until the X server has sent the CopyArea request down to the kernel.

I could do that by having the X server send me an X event, but I think there’s a better way that will extend to systems that don’t offer the kernel serialization guarantee. James Jones and Aaron Plattner put together a proposal to add Fences to the X Sync extension. In the X world, those offer a method to serialize rendering between two X applications, but of course the real goal is to expose those fences to GL applications through the various GL sync extensions (including GLARBsync and GLNVfence).

With the current Linux DRI implementation, I think it would be pretty easy to implement these fences using pthread semaphores in a block of memory shared between the server and application. That would be DRI-specific; other direct rendering interfaces would use alternate means to share the fences between X server and application.

Swap/Present — The Second Extension

By simply using CopyArea for my application presentation step, I think I’ve neatly split this problem into manageable pieces. Once I’ve got the DRI3 piece working, I’ll move on to fixing the presentation issue.

By making that depend solely on existing core Pixmap objects as the source of data to present, I can develop that without any reference to DRI. This will make the extension useful to existing X applications that currently have only CopyArea for this operation.

Presentation of application contents occurs in two phases; the first is to identify which objects are involved in the presentation. The second is to perform the presentation operation, either using CopyArea, or by swapping pages or the entire frame buffer. For offscreen objects, these can occur at the same time. For onscreen, the presentation will likely be synchronized with the scanout engine.

The second form will mean that the Fences that mark when the presentation has occurred will need to signaled only once the operation completes.

A CopyArea operation means that the source pixmap is “ready” immediately after the Copy has completed. Doing the presentation by using the source pixmap as the new front buffer means that the source pixmap doesn’t become “ready” until after the next swap completes.

What I don’t know now is whether we’ll need to report up-front whether the presentation will involve a copy or a swap. At this point, I don’t think so — the application will need two back buffers in all cases to avoid blocking between the presentation request and the presentation execution. Yes, it could use a fence for this, but that still sticks a bubble in the 3D hardware where it’s blocked waiting for vblank instead of starting on the next frame immediately.

Plan of Attack

Right now, I’m working on finishing up the DRI3 piece:

• Replace the DRI2 buffer allocation kludge with actual local buffer allocation, mapping them into pixmaps using FD passing.

• Replace the DRI2 authentication scheme with having the X server open the DRI object, preparing it for rendering and passing it back to the application.

• Working on the XCB pieces to get the split event-queue stuff landed upstream.

• Implementing the Fencing stuff to correctly serialize access to the pixmap.

The first three seem fairly straight forward. The fencing stuff will involve working with James and Aaron to integrate their XSync changes into the server.

After that, I’ll start working on the presentation piece. Foremost there is figuring out the right name for this new extension; I started with the name ‘Swap’ as that’s the GL call it implements. However, ‘Swap’ is quite misleading as to the actual functionality; a name more like ‘Present’ might provide a better indication of what it actually does. Of course, ‘Present’ is both a verb and a noun, with very different connotations. Suggestions on this most complicated part of the project are welcome!

Sometimes it’s most interesting to just dive in and see what breaks. There’s a Chef cookbook for Ceph on github which seems rather more recently developed than the one in SUSE-Cloud/barclamp-ceph, and seeing as its use is documented in the Ceph manual, I reckon that’s the one I want to be using. Of course, the README says “Tested as working: Ubuntu Precise (12.04)”, and I’m using openSUSE 12.3…

First things first, need a Chef server, so I installed openSUSE 12.3 on a VM, then installed Chef 10 on that, roughly following the manual installation instructions. Note for those following along at home – sometimes the blocks I’ve copied here are just commands, sometimes they include command output as well. You’ll figure it out

# zypper ar -f http://download.opensuse.org/repositories/systemsmanagement:/chef:/10/openSUSE_12.3/systemsmanagement:chef:10.repo
# zypper in rubygem-chef-server
# chkconfig couchdb on
# rccouchdb start
# chkconfig rabbitmq-server on
# rcrabbitmq-server start
# rabbitmqctl add_vhost /chef
# rabbitmqctl add_user chef testing
# rabbitmqctl set_permissions -p /chef chef ".*" ".*" ".*"
# for service in solr expander server server-webui; do
      chkconfig chef-$service on
      rcchef-$service start
  done

I didn’t bother editing /etc/chef/server.rb, the config as shipped works fine (not that the AMQP password is very secure, mind). The only catch is the web UI didn’t start. IIRC this is due to /etc/chef/webui.pem not existing yet (chef-server creates it, but this doesn’t finish until later).

Then configured knife:

# knife configure -i
WARNING: No knife configuration file found
Where should I put the config file? [/root/.chef/knife.rb]
Please enter the chef server URL: [http://os-chef.example.com:4000]
Please enter a clientname for the new client: [root]
Please enter the existing admin clientname: [chef-webui]
Please enter the location of the existing admin client's private key: [/etc/chef/webui.pem]
Please enter the validation clientname: [chef-validator]
Please enter the location of the validation key: [/etc/chef/validation.pem]
Please enter the path to a chef repository (or leave blank):
Creating initial API user...
Created client[root]
Configuration file written to /root/.chef/knife.rb

And make a client for me:

# knife client create tserong -d -a -f /tmp/tserong.pem
Created client[tserong]

Then set up my desktop as a Chef workstation (roughly following these docs, and again pulling Chef from systemsmanagement:chef:10 on OBS):

# sudo zypper in rubygem-chef
# cd ~
# git clone git://github.com/opscode/chef-repo.git
# cd chef-repo
# mkdir -p ~/.chef
# scp root@os-chef:/etc/chef/validation.pem ~/.chef/
# scp root@os-chef:/tmp/tserong.pem ~/.chef/
# knife configure
WARNING: No knife configuration file found
Where should I put the config file? [/home/tserong/.chef/knife.rb]
Please enter the chef server URL: [http://desktop.example.com:4000] http://os-chef.example.com:4000
Please enter an existing username or clientname for the API: [tserong]
Please enter the validation clientname: [chef-validator]
Please enter the location of the validation key: [/etc/chef/validation.pem] /home/tserong/.chef/validation.pem
Please enter the path to a chef repository (or leave blank): /home/tserong/chef-repo
[...]
Configuration file written to /home/tserong/.chef/knife.rb

Make sure it works:

# knife client list
chef-validator
chef-webui
root
tserong

Grab the cookbooks and upload them to the Chef server. The Ceph cookbook claims to depend on apache and apt, although presumably the former is only necessary for RADOSGW, and the latter for Debian-based systems. Anyway:

# cd ~/chef-repo
# git submodule add git@github.com:opscode-cookbooks/apache2.git cookbooks/apache2
# git submodule add git@github.com:opscode-cookbooks/apt.git cookbooks/apt
# git submodule add git@github.com:ceph/ceph-cookbooks.git cookbooks/ceph
# knife cookbook upload apache2
# knife cookbook upload apt
# knife cookbook upload ceph

Boot up a couple more VMs to be Ceph nodes, using the appliance image from last time. These need chef-client installed, and need to be registered with the chef server. knife bootstrap will install chef-client and dependencies for you, but after looking at the source, if /usr/bin/chef doesn’t exist, it actually uses wget or curl to pull http://opscode.com/chef/install.sh and runs that. How this is considered a good idea is completely baffling to me, so again I installed our chef build from OBS on each of my Ceph nodes (note to self: should add this to appliance image on Studio):

# zypper ar -f http://download.opensuse.org/repositories/systemsmanagement:/chef:/10/openSUSE_12.3/systemsmanagement:chef:10.repo
# zypper in rubygem-chef

And ran the now-arguably-safe knife bootstrap from my desktop:

# knife bootstrap ceph-0.example.com
Bootstrapping Chef on ceph-0.example.com
[...]
# knife bootstrap ceph-1.example.com
Bootstrapping Chef on ceph-1.example.com
[...]

Then, roughly following the Ceph Deploying with Chef document.

Generate a UUID and monitor secret (had to do the latter on one of my Ceph VMs, as ceph-authtool is conveniently already installed):

# uuidgen -r
f80aba97-26c5-4aa3-971e-09c5a3afa32f
# ceph-authtool /dev/stdout --name=mon. --gen-key
[mon.]
key = AQC8umZRaDlKKBAAqD8li3u2JObepmzFzDPM3g==

Then on my desktop:

knife environment create Ceph

This I filled in with:

{
  "name": "Ceph",
  "description": "",
  "cookbook_versions": {
  },
  "json_class": "Chef::Environment",
  "chef_type": "environment",
  "default_attributes": {
    "ceph": {
      "monitor-secret": "AQC8umZRaDlKKBAAqD8li3u2JObepmzFzDPM3g==",
      "config": {
        "fsid": "f80aba97-26c5-4aa3-971e-09c5a3afa32f",
        "mon_initial_members": "ceph-0,ceph-1",
        "global": {
        },
        "osd": {
          "osd journal size": "1000",
          "filestore xattr use omap": "true"
        }
      }
    }
  },
  "override_attributes": {
  }
}

Uploaded roles:

# knife role from file cookbooks/ceph/roles/ceph-mds.rb
# knife role from file cookbooks/ceph/roles/ceph-mon.rb
# knife role from file cookbooks/ceph/roles/ceph-osd.rb
# knife role from file cookbooks/ceph/roles/ceph-radosgw.rb

Assigned roles to nodes:

# knife node run_list add ceph-0.example.com 'role[ceph-mon],role[ceph-osd],role[ceph-mds]'
# knife node run_list add ceph-1.example.com 'role[ceph-mon],role[ceph-osd],role[ceph-mds]'

I didn’t bother with recipe[ceph::repo] as I don’t care about installation right now (Ceph is already installed in my VM images).

Had to set "chef_environment": "Ceph" for each node by running:

# knife node edit ceph-0.example.com
# knife node edit ceph-1.example.com

Didn’t set Ceph osd_devices per node – I’m just playing, so can sit on top of the root partition.

Now let’s see if it works:

# knife ssh name:ceph-0.example.com -x root chef-client
[2013-04-11T13:44:47+00:00] INFO: *** Chef 10.24.0 ***
[2013-04-11T13:44:48+00:00] INFO: Run List is [role[ceph-mon], role[ceph-osd], role[ceph-mds]]
[2013-04-11T13:44:48+00:00] INFO: Run List expands to [ceph::mon, ceph::osd, ceph::mds]
[2013-04-11T13:44:48+00:00] INFO: HTTP Request Returned 404 Not Found: No routes match the request: /reports/nodes/ceph-0.example.com/runs
[2013-04-11T13:44:48+00:00] INFO: Starting Chef Run for ceph-0.example.com
[2013-04-11T13:44:48+00:00] INFO: Running start handlers
[2013-04-11T13:44:48+00:00] INFO: Start handlers complete.
[2013-04-11T13:44:48+00:00] INFO: Loading cookbooks [apache2, apt, ceph]
No ceph-mon found.

[2013-04-11T13:44:48+00:00] INFO: Processing template[/etc/ceph/ceph.conf] action create (ceph::conf line 6)
[2013-04-11T13:44:48+00:00] INFO: template[/etc/ceph/ceph.conf] backed up to /var/chef/backup/etc/ceph/ceph.conf.chef-20130411134448
[2013-04-11T13:44:48+00:00] INFO: template[/etc/ceph/ceph.conf] updated content
[2013-04-11T13:44:48+00:00] INFO: template[/etc/ceph/ceph.conf] owner changed to 0
[2013-04-11T13:44:48+00:00] INFO: template[/etc/ceph/ceph.conf] group changed to 0
[2013-04-11T13:44:48+00:00] INFO: template[/etc/ceph/ceph.conf] mode changed to 644
[2013-04-11T13:44:48+00:00] INFO: Processing service[ceph_mon] action nothing (ceph::mon line 23)
[2013-04-11T13:44:48+00:00] INFO: Processing execute[ceph-mon mkfs] action run (ceph::mon line 40)
creating /var/lib/ceph/tmp/ceph-ceph-0.mon.keyring
added entity mon. auth auth(auid = 18446744073709551615 key=AQC8umZRaDlKKBAAqD8li3u2JObepmzFzDPM3g== with 0 caps)
ceph-mon: mon.noname-a 192.168.4.118:6789/0 is local, renaming to mon.ceph-0
ceph-mon: set fsid to f80aba97-26c5-4aa3-971e-09c5a3afa32f
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-ceph-0 for mon.ceph-0
[2013-04-11T13:44:49+00:00] INFO: execute[ceph-mon mkfs] ran successfully
[2013-04-11T13:44:49+00:00] INFO: execute[ceph-mon mkfs] sending start action to service[ceph_mon] (immediate)
[2013-04-11T13:44:49+00:00] INFO: Processing service[ceph_mon] action start (ceph::mon line 23)
[2013-04-11T13:44:49+00:00] INFO: service[ceph_mon] started
[2013-04-11T13:44:49+00:00] INFO: Processing ruby_block[tell ceph-mon about its peers] action create (ceph::mon line 64)
connect to
/var/run/ceph/ceph-mon.ceph-0.asok
failed with
(2) No such file or directory

connect to
/var/run/ceph/ceph-mon.ceph-0.asok
failed with
(2) No such file or directory

[2013-04-11T13:44:49+00:00] INFO: ruby_block[tell ceph-mon about its peers] called
[2013-04-11T13:44:49+00:00] INFO: Processing ruby_block[get osd-bootstrap keyring] action create (ceph::mon line 79)
2013-04-11 13:44:49.928800 7f58e9677700 0
-- :/23863 >> 192.168.4.117:6789/0 pipe(0x18f0d30 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault

2013-04-11 13:44:52.928739 7f58efc1c700 0 -- :/23863 >> 192.168.4.118:6789/0 pipe(0x7f58e0000c00 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault
2013-04-11 13:44:55.929375 7f58e9677700 0 -- :/23863 >> 192.168.4.117:6789/0 pipe(0x7f58e0003010 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault
2013-04-11 13:44:58.929211 7f58efc1c700 0 -- :/23863 >> 192.168.4.118:6789/0 pipe(0x7f58e00039f0 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault
2013-04-11 13:45:01.929787 7f58e9677700 0 -- :/23863 >> 192.168.4.117:6789/0 pipe(0x7f58e00023b0 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault
[...]

And it’s stuck there, trying and failing to talk to something.

See those “no such file or directory” errors after “service[ceph_mon] started”? Yeah? Well, the mon isn’t started, hence the missing sockets in /var/run/ceph.

Why isn’t the mon started? Turns out the ceph init script won’t start any mon (or osd or mds for that matter) if you don’t have entries in the config file with some suffix, e.g. [mon.a]. And all I’ve got is:

[global]
  fsid =  f80aba97-26c5-4aa3-971e-09c5a3afa32f
  mon initial members = ceph-0,ceph-1
  mon host = 192.168.4.118:6789, 192.168.4.117:6789

[osd]
    osd journal size = 1000
    filestore xattr use omap = true

But given the mon recipe triggers ceph-mon-all-starter if using upstart (which it would be, on the “Tested as working: Ubuntu Precise”), and ceph-mon-all-starter seems to just ultimately run something like ceph-mon --cluster=ceph -i ceph-0 regardless of what’s in the config file… Maybe I can cheat.

Directly starting ceph-mon from a shell on ceph-0 before the chef-client run turned out to be a bad idea (bit of a chicken and egg problem figuring out what to inject into the “mon host” line of the config file). So I put a bit of evil into the mon recipe:

diff --git a/recipes/mon.rb b/recipes/mon.rb
index 5cd76de..a518830 100644
--- a/recipes/mon.rb
+++ b/recipes/mon.rb
@@ -61,6 +61,10 @@ EOH
   notifies :start, "service[ceph_mon]", :immediately
 end

+execute 'hack to force mon start' do
+  command "ceph-mon --cluster=ceph -i #{node['hostname']}"
+end
+
 ruby_block "tell ceph-mon about its peers" do
   block do
     mon_addresses = get_mon_addresses()

Try again:

# knife ssh name:ceph-0.example.com -x root chef-client
[2013-04-11T15:10:43+00:00] INFO: *** Chef 10.24.0 ***
[2013-04-11T15:10:44+00:00] INFO: Run List is [role[ceph-mon], role[ceph-osd], role[ceph-mds]]
[2013-04-11T15:10:44+00:00] INFO: Run List expands to [ceph::mon, ceph::osd, ceph::mds]
[2013-04-11T15:10:44+00:00] INFO: HTTP Request Returned 404 Not Found: No routes match the request: /reports/nodes/ceph-0.example.com/runs
[2013-04-11T15:10:44+00:00] INFO: Starting Chef Run for ceph-0.example.com
[2013-04-11T15:10:44+00:00] INFO: Running start handlers
[2013-04-11T15:10:44+00:00] INFO: Start handlers complete.
[2013-04-11T15:10:44+00:00] INFO: Loading cookbooks [apache2, apt, ceph]
[2013-04-11T15:10:44+00:00] INFO: Storing updated cookbooks/ceph/recipes/mon.rb in the cache.
No ceph-mon found.

[2013-04-11T15:10:44+00:00] INFO: Processing template[/etc/ceph/ceph.conf] action create (ceph::conf line 6)
[2013-04-11T15:10:44+00:00] INFO: Processing service[ceph_mon] action nothing (ceph::mon line 23)
[2013-04-11T15:10:44+00:00] INFO: Processing execute[ceph-mon mkfs] action run (ceph::mon line 40)
[2013-04-11T15:10:44+00:00] INFO: Processing execute[hack to force mon start] action run (ceph::mon line 65)
starting mon.ceph-0 rank 1 at 192.168.4.118:6789/0 mon_data /var/lib/ceph/mon/ceph-ceph-0 fsid f80aba97-26c5-4aa3-971e-09c5a3afa32f
[2013-04-11T15:10:44+00:00] INFO: execute[hack to force mon start] ran successfully
[2013-04-11T15:10:44+00:00] INFO: Processing ruby_block[tell ceph-mon about its peers] action create (ceph::mon line 69)
adding peer 192.168.4.118:6789/0 to list: 192.168.4.117:6789/0,192.168.4.118:6789/0

adding peer 192.168.4.117:6789/0 to list: 192.168.4.117:6789/0,192.168.4.118:6789/0

[2013-04-11T15:10:44+00:00] INFO: ruby_block[tell ceph-mon about its peers] called
[2013-04-11T15:10:44+00:00] INFO: Processing ruby_block[get osd-bootstrap keyring] action create (ceph::mon line 84)
2013-04-11 15:10:44.432266 7f8f9f8c0700  0
-- :/25965 >> 192.168.4.117:6789/0 pipe(0x16d9d30 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault

2013-04-11 15:10:50.433053 7f8f9f7bf700  0 -- 192.168.4.118:0/25965 >> 192.168.4.117:6789/0 pipe(0x7f8f94001d30 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault
2013-04-11 15:10:56.433268 7f8fa5e65700  0 -- 192.168.4.118:0/25965 >> 192.168.4.117:6789/0 pipe(0x7f8f94001d30 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault
2013-04-11 15:11:02.433987 7f8f9f8c0700  0 -- 192.168.4.118:0/25965 >> 192.168.4.117:6789/0 pipe(0x7f8f94002db0 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault
2013-04-11 15:11:08.434358 7f8f9f7bf700  0 -- 192.168.4.118:0/25965 >> 192.168.4.117:6789/0 pipe(0x7f8f94004fb0 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault

At this point it’s stalled presumably waiting to talk to the other mon, so in another terminal window had to kick off a chef-client run on ceph-1 to get it into the same state as ceph-0 (knife ssh name:ceph-1.example.com -x root chef-client). This allowed both nodes to progress to the next problem:

2013-04-11 15:11:28.563438 7f8fa5e67780 -1 monclient(hunting): authenticate NOTE: no keyring found; disabled cephx authentication
2013-04-11 15:11:28.563443 7f8fa5e67780 -1 unable to authenticate as client.admin
2013-04-11 15:11:28.563814 7f8fa5e67780 -1 ceph_tool_common_init failed.
2013-04-11 15:11:29.572208 7f2369130780 -1 monclient(hunting): authenticate NOTE: no keyring found; disabled cephx authentication
2013-04-11 15:11:29.572210 7f2369130780 -1 unable to authenticate as client.admin
2013-04-11 15:11:29.572527 7f2369130780 -1 ceph_tool_common_init failed.
2013-04-11 15:11:31.380073 7f1907d18780 -1 monclient(hunting): authenticate NOTE: no keyring found; disabled cephx authentication
2013-04-11 15:11:31.380078 7f1907d18780 -1 unable to authenticate as client.admin
2013-04-11 15:11:31.380720 7f1907d18780 -1 ceph_tool_common_init failed.
2013-04-11 15:11:32.392345 7fc2bc462780 -1 monclient(hunting): authenticate NOTE: no keyring found; disabled cephx authentication
[...]

And we’re spinning again.

But that’s enough for one day.

It just occurred to me that I haven’t blogged this yet. Back in January, I generated a new personal 4096-bit RSA PGP key. My old one was 1024-bit DSA, which I’m starting to worry about being not so secure in this day and age.

The new key fingerprint is as follows:

pub   4096R/A5326F5B 2013-01-29
      Key fingerprint = CFF4 E176 D294 DAED F603  3FE2 360F 8D38 A532 6F5B
uid                  Jeremy Visser <jeremy [at] visser [dot] name>
uid                  Jeremy Visser <jeremy [at] sunriseroad [dot] net>
uid                  Jeremy Visser <jeremy [dot] visser [at] gmail [dot] com>
sub   4096R/369D314A 2013-01-29

You may retrieve the latest copy of it from all major keyservers:

$ gpg --keyserver keys.gnupg.net --recv-keys A5326F5B

Being a new key, and only having had it signed at the LCA2013 Keysigning Party, it doesn’t have many signatures yet.

I’m really hoping that someone can provide me with some enlightenment.

I have a lot of ssh keys. 6 by today’s count. On my desktop I have my ssh configured with IdentitiesOnly yes and an IdentityFile for each host. This works great.

I then forward my agent to my dev VM. I can see the keys with ssh-add -l. So far so good. If I then ssh into a host, I can see it trying every key from the agent in sequence, which is sometimes going to fail with too many keys tried. However, if I try IdentitiesOnly yes in my dev VM config, it doesn’t offer any keys, if I add IdentityFile it doesn’t work because I don’t have those key files on my VM.

So what’s the solution? What I want is to specify identities by their identifier in the agent, e.g. danni@github, however I can’t see config to do that. Anyone got a nifty solution?

After I left the Bureau approximately a month ago I’ve taken up a new role with Infoxchange Australia. My first project here is working on a rewrite of an application using Django.

People here are really into behaviour driven testing, and we’re using Lettuce to do it (using a branch with better Django integration).

I sort of dislike this sort of testing, because it creates an annoying abstraction layer on top of the code, with a poorly defined, quasi-real language. It’s like a bad knock off of Applescript. Anyway, I got sick of defining steps per model, so I put together some generic steps for manipulating Django models (that I’ll have to contribute back).

Anyway they look like this (examples of the step in the docstrings):

# build a hash of model verbose names to models
# this is used by get_model()
def _models_generator():
    for model in get_models():
        yield (model._meta.verbose_name, model)
        yield (model._meta.verbose_name_plural, model)

MODELS = dict(_models_generator())


def get_model(model):
    """
    Convert a model's verbose name to the model class. This allows us to
    use the models verbose name in steps.
    """

    name = model.lower()
    model = MODELS.get(model, None)

    assert model, "Could not locate model by name '%s'" % name

    return model


def create_models(model, hashes):
    for hash_ in hashes:
        model.objects.create(**hash_)


def models_exist(model, hashes):
    for hash_ in hashes:
        assert \
            model.objects.filter(**hash_).exists(), \
            "Object does not exist"


@step(r'I have ([a-z][a-z0-9_ ]*) in the database:')
def create_models_generic(step, model):
    """
    And I have admin field values in the database:
    | name         | value   |
    | project_type | Twine   |

    The generic method can be overridden for a specific model by defining a
    function create_badgers(step), which creates the Badger model.
    """

    try:
        globals()['create_%s' % model](step)
    except KeyError:
        model = get_model(model)

        create_models(model, step.hashes)


@step(r'(?:Given|And|Then) ([A-Z][a-z0-9_ ]*) with ([a-z]+) "([^"]*)" has ([A-Z][a-z0-9_ ]*) in the database:')  # noqa
def create_models_for_relation(step, rel_model_name,
                               rel_key, rel_value, model):
    """
    And project with name "Ball Project" has goals in the database:
    | description                             |
    | To have fun playing with balls of twine |
    """

    lookup = {rel_key: rel_value}
    rel_model = get_model(rel_model_name).objects.get(**lookup)

    for hash_ in step.hashes:
        hash_['%s_id' % rel_model_name] = rel_model.id

    create_models_generic(step, model)


@step('(?:Given|And|Then) ([A-Z][a-z0-9_ ]*) should be present in the database')
def step_models_exist(step, model):
    """
    And objectives should be present in the database:
    | description      |
    | Make a mess      |
    """

    model = get_model(model)

    models_exist(model, step.hashes)


@step(r'There should be (\d+) ([a-z][a-z0-9_ ]*) in the database')
def model_count(step, count, model):
    """
    Then there should be 0 goals in the database
    """

    model = get_model(model)

    assert_equals(model.objects.count(), int(count))

A long time ago in a galaxy far, far away… Or, more accurately, thirteen years ago while living in a house maybe 600-800km away (depending on your mode of transport), I published a tutorial entitled JavaScript for E-Commerce, or JSEC for short.

From the web server logs, the JSEC tutorial (and, for that matter, the KeyGlove project, which happened at a similar time), are still pretty high on the list of search terms and entry pages that land random visitors on wirejunkie.com. And I still think the JSEC tutorial is a decent, well-structured tutorial, insofar as it leads one through a certain development process, but I did realise at some point that even though the code all still works in modern web browsers, the tutorial no longer necessarily instructs best practice. For example, it presents a makeEmptyArray() function and associated scaffolding necessary for running on Netscape 2.0.

You all remember Netscape 2.0, right?

But even though the content is outdated, I could never bring myself to take the JSEC tutorial down, because cool URIs don’t change. This is a positive restatement of the old saw ”I hope you know that this will go down on your permanent record”. Consequently I added a note at the top of the table of contents to say:

Please note that this tutorial was first published in the year 2000, and is based on work conducted between 1997 and 2000. While we trust that it still constitutes useful tutorial material, it should not necessarily be construed as to impart best practice in the year $THIS_YEAR.

That’s suitably cautionary, but I really want to take the opportunity to state clearly that if I were building an online shop today, I would probably not do so as outlined in the JSEC tutorial. There are several reasons for this, the two most obvious being:

1. The framework presented uses <frameset> heavily. This will almost certainly provide a poor experience for mobile web users, which is a huge number of people these days. I suspect it’s also problematic for people using screen readers.
2. At the time the tutorial was written, server-side (PHP/Python/Ruby etc.) ecommerce frameworks were either rare, immature, expensive, nonexistent, or perhaps all of the above. I rather hope think things have changed in the meantime, which makes a pure JavaScript solution less interesting than it used to be.

In summary, the JSEC tutorial will remain online for all to enjoy, but do please be aware of the historical context in which it was created.

(I also find it entertaining that I can believably use the term “historical context” after only a decade or so. Aaah, technology….)

This week is SUSE Hack Week 9. I wanted to spend some time working on a Ceph appliance image to make it easy to play with Ceph on openSUSE and/or SLES.

I tried making a SLES 11 SP2 appliance with SUSE Studio. I had to add the filesystems and devel:libraries:c_c++ repos from OBS to get reasonably up-to-date Ceph 0.56 and libboost_thread.so.1.49.0, but on boot when the appliance tried to expand its root filesystem, it died claiming it couldn’t load libe2p.so.2. Studio claims to be pulling in e2fsprogs from both the SP2 Updates and filesystems repo, so maybe that’s the problem. It seems impossible to choose one or the other, as they are the same version. (Update: it was just pointed out to me that you can click the little box next to the version number to choose which one is installed – must try again.)

So I left that alone and tried an openSUSE 12.3 appliance. The filesystems/ceph build for 12.3 is disabled, so I branched it and kicked off a build which failed with an exciting OOM error:

[ 3831s] [ 3803.167109] Out of memory: Kill process 16364 (cc1plus) score 254 or sacrifice child
[ 3831s] [ 3803.167959] Killed process 16364 (cc1plus) total-vm:825128kB, anon-rss:168760kB, file-rss:4kB
[ 3831s] g++: internal compiler error: Killed (program cc1plus)
[ 3831s] Please submit a full bug report,
[ 3831s] with preprocessed source if appropriate.
[ 3831s] See  for instructions.

Guess I should do what it says and file a bug. But I really did want something to play with immediately, so I added http://ceph.com/rpm/opensuse12/x86_64/ as a repo, and pulled in the upstream Ceph 0.56 RPMs. This seems to have worked and given me an openSUSE 12.3 image I can use to run through the Ceph 5-Minute Quick Start, Block Device Quick Start and CephFS Quick Start. So, here’s my extremely terse openSUSEified version of those quick start documents:

5-Minute Quick Start

Deploy the Appliance Image

I’m doing this with a couple of VMs, so in my case I make a couple of copies of the image:

# cp ~/openSUSE_12.3_Ceph_0.56.x86_64-0.0.3.qcow2 \
    /var/lib/libvirt/images/ceph-quickstart-server.qcow2
# cp ~/openSUSE_12.3_Ceph_0.56.x86_64-0.0.3.qcow2 \
    /var/lib/libvirt/images/ceph-quickstart-client.qcow2

Then I use virt-manager to create two VMs, backed by those images. Boot ‘em up, log in (root password is “linux”), run yast network and set sensible hostnames (“ceph-client” and “ceph-server” instead of “linux-kjqd”, although admittedly those names wouldn’t be very sensible in a real deployment with more than one node).

Edit the Configuration File

The appliance image includes the /etc/ceph/ceph.conf file from the original 5-minute quick start, so log in to ceph-server, edit that file and replace {hostname} and {ip-address} with their real values, then copy the configuration file to ceph-client:

# scp /etc/ceph/ceph.conf ceph-client:/etc/ceph/

Deploy the Configuration

On ceph-server, create directories for each daemon:

# mkdir -p /var/lib/ceph/osd/ceph-0
# mkdir -p /var/lib/ceph/osd/ceph-1
# mkdir -p /var/lib/ceph/mon/ceph-a
# mkdir -p /var/lib/ceph/mds/ceph-a

Still on ceph-server, run the following:

# cd /etc/ceph
# mkcephfs -a -c /etc/ceph/ceph.conf -k ceph.keyring

Start Ceph

On ceph-server:

# chkconfig ceph on
# rcceph start
# ceph health

This will initially show something like:

HEALTH_ERR 576 pgs stuck inactive; 576 pgs stuck unclean; no osds

Eventually it will say HEALTH_OK and you’re good to go.

Copy the Keyring to the Client

This is necessary for authentication:

# scp /etc/ceph/ceph.keyring ceph-client:/etc/ceph/

Block Device Quick Start

On ceph-client:

# rbd create foo --size 4096
# modprobe rbd
# rbd map foo --pool rbd --name client.admin
# mkfs.ext4 -m0 /dev/rbd1
# mkdir /mnt/myrbd
# mount /dev/rbd1 /mnt/myrbd

(Why is this /dev/rbd1, not /dev/rbd/rbd/foo as in the original quick start?)

CephFS Quick Start

On ceph-client (kernel driver, not FUSE):

# mkdir /mnt/mycephfs
# mount -t ceph -o name=admin,secret=$(ceph-authtool \
    --name client.admin /etc/ceph/ceph.keyring --print-key) \
    ceph-server:/ /mnt/mycephfs

Interestingly, this gives “mount: error writing /etc/mtab: Invalid argument”, but still seems to actually mount the filesystem.

Also note that it appears I have 32GB of space for Ceph to use, even though ceph-server only has a 16GB root partition. I rather think that’s because there’s two OSDs, but both are just running off the root filesystem, they’re not separate disks/filesystems. I assume this is one of those Don’t Try This At Home things.

As indicated, I had the opportunity to talk during the first Distro Recipes event organized in Paris last week, at the invitation of Hupstream. As Yoann Sculo posted, this was a very interesting day for me, and I really regret I was busy to also attend the first day and the opening.

After a nice welcome breakfast, Aurélien Bompard started by presenting the Fedora distribution.



He did a great job especially expalining how easy it was to become a Fedora maintainer, even if a comparison to Debian revealed that it’s much less different that what people may think (it also takes time to become a packager able to modify most distro packages) and I know by experience that the Fedora packagers are really picky (sometimes for not so good reasons) with new contributions.

After that I talked about HP and Linux distributions. I used in fact the standard HP marketing presentation of the company as a starter (modified of course to suit my needs and include more penguins !) in order to explain the span of our activities, our relationship with communities including distributions, announced that HP will even soon provide firmware for ProLiant servers under a package format (rpm and deb), the fact that HP doesn’t see Linux demand for desktop/laptop on the consumer market (no, it’s not just a price issue that would make Linux more appealing in that case as I justified) but that we do support Linux on some enterprise desktops/laptops. Hopefully this was useful and/or new to some of the audience.

Then Dodji Seketeli made the type of talk making you believe that you could contribute to gcc ! Of course, when he details how much time it took him to add some of the features of the next stable version, you know you can’t ! Well I at least Anyway lots of good news and features that make that future version 4.8 expected soon.

That conclude our morning sessions, and it was then time to eat !! Especially as we had a great buffet waiting for us as you can see:

In order to avoid a sleepy afternoon, we started right after by a round table with 7 people (!), that I had the pleasure to chair. With a representative of each distribution (Mageia, openSUSE, Fedora, Debian, Arch, Embedded) and a Microsoft representative, you could expect blood and swords fight ! Not at all, I was surprisingly happy that the elements were clearly exposed, each representative defending their own work rather than criticizing, and finding ways to propose more future joint work. Of course, some subjects such as LSB/FHS lead to more debate, but very constructive and I really enjoyed this time slot as a way to show that differences are an added value ! It was also the opportunity for me to meet with Colin Guthrie and Frédérc Crozat, which I had never met before. These distros should be happy to have such representatives defending them (and the others too of course ) Finally if you have ideas to share to improve cross-distribution work , consider joining the mailing listdedicated to his topic and start sharing your ideas.

Then it was time again for the remaining presentations. The first was Lucas Nussbaum. Long time Debian Developer, (he is even running for the Debian Project Leader now, vote for him !) he made a convincing picture of the Debian ecosystem, the numerous Web sites that contributors can create to enhance the distribution with stats, infos, Ubuntu correlations, … As usual, Debian appears as a very mature distribution, with a strong Governance, being perl friendly… If I had to change I may well become a debianers. But isn’t it because of the pres, as the morning I was a fedorian

The next speaker was a long time Linux enthousiast Pierre Ficheux. In fact back when it was Minitel time (not 2.0) I used his xtel program !! Pierre made a presentation (in english but with the accent ) around embedded Linux distributions, presenting various way to tailor one for your device (he was using a Raspberry Pi) depending whether you use an Ubuntu, a Yocto generated one or a pure OpenEmbedded linux one. Definitely a good idea to explore for my Pi !

And then we had the lightnings talks. Aurélien Bompard was there again for HyperKitty. Too bad it’s devoted to mailan, as I think Sympa would also benefit from such a work, as their archive management (at least on the latest versions I used) could be improved.

I came then again on stage for a project-builder.org presentation (building cross-distro packages for upstream projects) and made a short demo which I think is explaining much more than my slides, so I plan on using it more in the future !

After me, Eric Leblond explained how his upstream project (ulogd2) wasn’t picked up correclty by most distributions and asked for help to improce that.

And final speaker was Nicolas Vérité who made a panel on all mobile Linux distributions, recommending to follow closely Tizen for the future as the main force in this area.

Too bad it was already over. Anne closed the session and I’d like to thank her for the invitation and the perfect organization of this first cross-distributions vent as a real success. Well done and see you next year hopefully !

Today roidrage and I hosted the inaugural European edition of the popular Hangops podcast / hangout sessions.  Hangops.eu (as we’ve taking to calling it) is functionally the same its North American counterpart – with the exception that “our” 11:00 is that of Western Europe, not California. ;)

If you’re not familiar with the Hangops, now’s the time for you to get with the programme.  The format is simple : get a group of talkative Ops people together on Google Hangouts, and see what happens !  Sometimes there are moderated talks, other times there are special guests, but the basic idea never changes – it’s fundamentally a chance for nerds to present ideas, debate topics, learn a little something, and have a good time.

The sessions are simulcast to Youtube, so if you don’t want to join the hangout itself, you’re invited to listen and participate in the IRC channel (freenode/#hangops) at your leisure.

Hope to see you out next time!

Last week I used 2 BTC to support Jupiter Broadcasting’s Unfilter show (and their other shows, but only Unfilter takes BTC so far).  Just now I noticed that someone made a 0.5BTC donation to my blog (I’ve had a BTC donation address in the sidebar of my blog for a few years now).  Thanks!

As I promised to pass donations onwards, I googled for bitcoin donations, and chose the following places to give 0.05 BTC each:

1. Juice Rap News for making high-baud political commentary (Unfilter in rap form)
2. Freedom Box for actually doing something about Internet freedom.
3. Torservers.net (as recommended by torproject.org) for the same.
4. f-droid.org for keeping a healthy Open alternative.
5. Bitcoin Foundation to support and strengthen the infrastructure that made this possible.
6. The Free Software Foundation even though I don’t always agree with them.
7. Wikileaks for recognizing something society needs, even if they stumble at delivery.
8. The Internet Archive for something that only gets more useful over time.

There are two left to go, so I’ll keep an eye out for more opportunities to donate in the next few weeks…

-0.05

Recently I spoke at BarCamp Canberra about my tips and tricks to changing the world. I thought it might be useful to get people thinking about how they can best contribute to the world, according to their skills and passions.

Completely coincidentally, my most excellent boss did a talk a few sessions ahead of me which was the American Civil War version of the same thing I highly recommend it. John Sheridan – Lincoln, Lee and ICT: Lessons from the Civil War.

So you want to change the world?

Here are the tactics I use to some success. I heartily recommend you find what works for you. Then you will have no excuse but to join me in implementing Operation World Awesomeness.

The Short Version:

No wasted movement.

The Long Version:

1) Pick your battles: there are a million things you could do. What do you most care about? What can you maintain constructive and positive energy about even in the face of towering adverseries and significant challenges? What do you think you can make a difference in? There is a subtle difference between choosing to knock down a mountain with your forehead, and renting a bulldozer. If you find yourself expending enormous energy on something, but not making a difference, you need to be comfortable to change tactics.

2) Work to your strengths: everyone is good at something. If you choose to contribute to your battle in a way that doesn’t work to your strengths, whatever they are, then you are wasting energy. You are not contributing in the best way you can. You need to really know yourself, understand what you can and can’t do, then do what you can do well, and supplement your army with the skills of others. Everyone has a part to play and a meaningful way to contribute. FWIW, I work to know myself through my martial arts training, which provides a useful cognitive and physical toolkit to engage in the world with clarity. Find what works for you. As Sun Tzu said: know yourself.

3) Identify success: Figure out what success actually looks like, otherwise you don’t have either a measurement of progress, nor a measurement of completion. I’ve seen too many activists get caught up on a battle and continued fighting well beyond the battle being won, or indeed keep hitting their heads against a battle that can’t be won. It’s important to continually be monitoring and measuring, holding yourself to account, and ensuring you are making progress. If not, change tactics.

4) Reconnaissance: do your research. Whatever your area of interest there is likely a body of work that has come before you that you can build upon. Learn about the environment you are working in, the politics, the various motivations and interests at play, the history and structure of your particular battlefield. Find levers in the system that you can press for maximum effect, rather than just straining against the weight of a mountain. Identify the various moving parts of the system and you have the best chance to have a constructive and positive influence.

5) Networks & Mentors: identify all the players in your field. Who is involved, influential, constructive, destructive, effective, etc. It is important to understand the motivations at play so you can engage meaningfully, collaboratively and build a mutually beneficial network in the persuit of awesomeness. Strong mentors are a vital asset and they will teach you how to navigate the rapids and make things happen. A strong network of allies is also vital to keep you on track, and accountable, and true to your own purpose. People usually strive to meet the expectations of those around them, so surround yourself with high expectations. Knowing your network also helps you identify issues and opportunities early.

6) Sustainability: have you put in place a succession plan? How will your legacy continue on without you? It’s important if your work is to continue on that it not be utterly reliant upon one individual. You need to share your vision, passion and success. Glory shared is glory sustained, so bring others on board, encourage and support them to succeed. Always give recognition and thanks to people who do great stuff.

7) Patience: remember the long game. Nothing changes overnight. It always take a lot of work and persistence, and remembering the long game will help during those times when it doesn’t feel like you are making progress. Again, your network is vital as it will help you maintain your strength, confidence and patience Speaking of which, a huge thanks to Geoff Mason for reminding me of this one on the day.

8) Shifting power: it is worth noting that we are living in the most exciting of times. Truly. Individuals are more empowered than ever before to do great things. The Internet has created a mechanism for the mass distribution of power, but putting into the hands of all people (all those online anyway), the tools to:

1. publish and access knowledge;
2. communicate and collaborate with people all around the world;
3. monitor and hold others to account including companies, governments and individuals;
4. act as enforcers for whatever code or law they uphold. This is of course quite controversial but fascinating nonetheless; and
5. finally, with the advances in 3D printing and nanotechnology, we are on the cusp of all people having unprecedented access to property.

Side note: Poverty and hunger, we shall overcome you yet! Then we just urgently need to prioritise education of all the people. But that is a post for another day Check out my blog post on Unicorns and Doom, which goes into my thoughts on how online culture is fundamentally changing society.

This last aspect is particularly fascinating as it changes the game from one between the haves and the have nots, to one between those with and those without skills and knowledge. We are moving from a material wealth differentiation in society towards an intellectual wealth differentiation. Arguable we always had the latter, but the former has long been a bastion for law, structures, power and hierarchies. And it is all changing.

“What better place than here, what better time than now?” — RATM

Banner, during the opening march of the Forum.

The 2013 World Social Forum has been an interesting experience: as with past Forums I’ve attended, it’s been huge, chaotic, and layered with different uses of the space and counter-protests. I didn’t go to many of the talks and workshops this time, because I was primarily interested in how the space was used: the power dynamics of where activities were placed, who was included and how, the difference between intended uses of areas and how they actually ended up being used, and so on. This is part of the broader Mapping Movements project that Tim Highfield and I are working on. I have far too many thoughts to digest just yet (and many pages of notes, and a whole heap of pictures), but there are at least a few themes that have emerged which I can cover in brief.

It will come as no surprise for those who know much about the WSF that in many ways the space replicates structures of hierarchy. From the opening performance, which began with long speeches on a stage that the audience were expected to listen to dutifully, through to the organisation of individual rooms, often in lecture format with a microphone tightly controlled by the facilitator, the space of the Forum does not quite lived up to the claims of horizontality that are so frequently made. There are also plenty of critiques of the consumerism of the Forum: expensive (by local standards) food, stalls selling knick-knacks, WSF t-shirts on sale.

People used empty spaces to talk, make phonecalls, and rest.

On the other hand, there is a significant difference between what the organisers intend for the space of the Forum, and how it actually ends up being used. At one end of the spectrum are overtly oppositional uses of space, such as the creation of an autonomous camp within the Forum and anti-capitalist, anti-state graffiti. There were also many uses of space which were not as the organisers intended, but which were not oppositional: workshops which spilled out from lecture rooms into the more open space of the corridor, or people sitting in empty rooms or under trees to rest for a while or to meet. There were also ‘alternative’ uses of space which were not intentional, but rather the result of planning gone awry – workshops that didn’t run because people couldn’t find the room, or because the facilitators didn’t end up making it to the Forum.

For me, writing perhaps more as an activist than as a researcher (insofar as it’s possible to separate out those roles), what is notable is the difference between the Forum as it is and the Forum as it might be. I have my doubts about the goal of the Forum, and remain unclear as to what many of the attendees get out of this international gathering when so many struggles must be strongly linked with a particular locale. But insofar as an international (or global, although the many flags waved at the Forum emphasise the former characterisation) gathering is useful, there are many ways in which it could do more to embody the politics which those involved espouse. The Forum might have a safer spaces policy prominently displayed (and created by participants). Participants might have picked up rubbish ourselves, and invited the women who were employed as cleaners to attend any forums and workshops they liked. Food could be cooked communally, and served in ways that avoided too much waste (although most of the plates and cups were paper instead of plastic, at least). There could be clear child-friendly spaces assigned, and accessibility could be prioritised.

I’m aware, however, that doing much of this is not easy, and will always be a work in progress. Given how recently most autonomous organisations were banned in Tunisia and the challenges which Tunisia still faces, the effort which was put into the Forum was impressive. Many volunteers gave their time and energy, and I am particularly thankful to those Tunisians involved in the Forum (either formally, or in a more oppositional or informal capacity) who were patient with my lack of French or Arabic. The Forum was meant to be held in solidarity with Tunisia’s revolution, so perhaps the best measure of its success is how Tunisians evaluate it. How effective was it as a space for Tunisians to talk about what comes next, to make links with each other or with organisations around the world, or as a support for the revolution? What did Tunisians want from the Forum? Of course, there will be no single answer to this. As the chaos of the WSF subsides I look forward to seeing what Tunisians thought of it and what, if any, long-term effects it might have.

Russ Allbery wrote an informative post about how to determine which charities are worth donating to [1]. He has a link to another article about the charities to which he donates and concentrates on ways of analysing the effectiveness of charities. So someone who has different ideas about which types of charity are worthy of donation could still learn a lot from his post.

Adam Green wrote an interesting article for The New Yorker about Apollo Robbins who is one of the world’s best pick-pockets [2]. Apollo picks pockets as a magician to entertain people and always returns what he steals. Now he is working with neuroscientists who are devising experiments to determine why his tricks work.

Rick Falkvinge wrote an insightful article describing the way that the copyright monopoly is in direct opposition to the freedom to make contracts [3]. It’s a good rebuttal of a common argument in favor of copyright law.

Seth Godin gave an interesting TED talk about the problems with the education system, how and why it teaches conformity and little else [4]. One of his suggestions for improvement is to have students spend their evenings watching lectures by experts and class time asking questions. He also says that everything should be open book and that there is no value in memorising anything – it’s a bit of an overstatement but it’s essentially correct.

Cory Doctorow wrote an interesting article for The Guardian about positive externalities and copyright law [5]. I think that he didn’t choose the best way of framing this issue, but he makes some very interesting points anyway.

Andrew Norton wrote an interesting article about how to reduce corruption in the police force and other government agencies [6]. A large part of this is based on making them subject to the same laws as everyone else, which seems to be a radical idea.

Valerie Aurora wrote an insightful blog post about suicide [7].

Emily Oster gave an interesting TED talk about the factors that determine the spread of AIDS in Africa [8]. It’s quite different to what you probably expect.

• [1] http://www.eyrie.org/~eagle/journal/2012-12/006.html
• [2] http://www.newyorker.com/reporting/2013/01/07/130107fa_fact_green
• [3] http://tinyurl.com/apovrka
• [4] http://www.youtube.com/watch?v=sXpbONjV1Jc
• [5] http://tinyurl.com/a7rutft
• [6] http://tinyurl.com/a227fvg
• [7] http://tinyurl.com/a5zo6tn
• [8] http://tinyurl.com/3f7283s

I’ve been [comparatively] quiet for a while now, and I can finally announce the results of my endeavours: I’ve resigned from Internode (ie. iiNet) in order to accept a position at Google in Sydney.

This means that Missy and I and our kids are all packing our bags and getting ready to head eastwards, looking forward to new and exciting adventures.

FAQ:

1. When? My last week in Adelaide is the week of the 22nd of April.
2. What will you do at Google? I’ll be a Site Reliability Engineer (which is kinda like a sysadmin).
3. Why leave? Because it’s the right time, and I’m looking for new challenges.
4. Why Google? It seems like a good fit for interests and for me personally. The folk I know there are generally pretty awesome, and I intend to learn as much as I can from them.

As a final note, I’d like to thank Simon Hackett and Adam Fox for their support and encouragement over the past few years; without them, I don’t think I’d be where I am now.

(FWIW, I started coming up with a list of people I’d like to thank, but it was unmanageably long — “Thank you” to the rest of you too; you should know who you are).

My new Lenovo T430 arrived last week. After delighting in that satisfying new laptop smell, I made recovery DVDs I will presumably never need, then blew away Windows 7 and installed openSUSE 12.3 (full disclosure: I work for SUSE, so my choice of distro may not be entirely unbiased).

Some niceties:

• The textured touchpad is lovely. Much better feel than a pure flat surface.
• As I’d expect, the keyboard is excellent (even if PGUP/PGDN aren’t where I’m used to).
• The openSUSE installer is quick and easy. I’m pretty sure there’s less steps than last time I did a regular openSUSE install from scratch a couple of years ago.
• No problem setting up encrypted LVM, although on my ~500GB drive it defaults to a 20GB root and 25GB /home, with a whole lotta free space left over in the encrypted partition, so that might want some tweaking.
• Entering the passphrase on boot happens on a pretty graphical screen, you don’t get thrown back to a terminal window where random junk is appearing over the passphrase entry prompt.
• Moving my mail over from my old laptop was pretty much just an rsync of the Thunderbird profile directory (and maybe a tweak to ~/.thunderbird/profiles.ini)

Some oddities:

• The Novell GroupWise 8.0.2 client had a couple of problems:
• It claims to need libXm.so.3 (listed in RPM Requires), but works fine without it. This is fortunate, because openSUSE 12.3 doesn’t ship openmotif22-libs-32bit anymore.
• Unless you’ve installed libpangox-1_0-0-32bit, the GroupWise client will segfault somewhere in libwebrenderer.so. This is less than obvious.
• The YaST disk partitioner seems slightly confused adding new LVs inside my encrypted VG later on (it either locked up or crashed). I haven’t had time to investigate this properly, so I’ve ignored it for the moment and used lvcreate and mkfs in a terminal instead.
• You do need to reboot at least once after initial install for NetworkManager to work properly (this is mentioned in the release notes).
• I’m running GNOME 3.6, and I tried using the tweak tool to have it just blank the screen – not suspend – when closing the laptop lid. Turns out systemd is being too clever for me, so I had to fiddle with that a bit (set HandleLidSwitch=ignore in /etc/systemd/logind.conf, then run sudo systemctl restart systemd-logind).

Very little else to report so far. Aside from the oddities above everything else seems to Just Work^TM. OTOH, all I’ve really done is web browsing, email and assorted fiddling around in terminals. Maybe listened to a bit of music (the inbuilt speakers are well and truly loud enough, but a bit tinnier than real speakers – can’t say I’m terribly surprised by that though).

I’ve been asked to review the GNOME 3 Application Development Guide for Beginners; I went through the book in about half a day and wrote this somewhat short review afterwards, and published on G+; sadly, I used a limited distribution, and G+ does not allow changing that without resharing your own post. given that I wanted to push it on the blog, I took the chance to review some of the stuff I wrote, and expand it.

my initial impression of the GNOME 3 Application Development Guide for Beginners book is fairly positive: the topics covered are interesting, and the book never loses itself in them, so that beginners will not feel utterly stranded after the first three chapters, as it too often happens with “for beginners” books. I appreciated the “pop quiz” sections, as well as the small sections that recommended improvements to the example code.

obviously, writing a book enshrines a certain set of requirements and APIs, and that is problematic when there is high churn – like what happens in GNOME 3, especially in terms of development tools (libraries and applications) and overall experience. for instance, the section on Clutter (which is the one I can immediately give feedback on, given my position) still uses the deprecated “default stage”, and yet it uses the new ClutterActor easing state for animations; the default stage was deprecated at long last in Clutter 1.10, but its use was not recommended since the days of 1.0; the actor easing state API was introduced in the same release that deprecated the default stage. also, the example code published in the Clutter section does not use any of the layout managers provided by Clutter, preferring the fixed positioning of the actors, which is perfectly fine on its own; the book, though, then proceeds to mention the amount of code necessary to get something on the screen, compared to the equivalent code in GTK, that uses boxes and grids. in general, that’s an utterly fair thing to say: Clutter sits at a lower-level than GTK, and it doesn’t have complex constructs like GTK does; I’m pretty sure, though, there are better examples than a row of boxes that could have used a BoxLayout, or a FlowLayout, or a GridLayout, or a TableLayout; or better examples than using an explicit PangoFontDescription instance with a ClutterText to set a specific font name and size, instead of using the ClutterText:font-name property which wraps the whole thing for the developer’s convenience. in short: Clutter is more “raw” than GTK, but there are convenience APIs for developers.

it’s been a long time¹ since I started off as a beginner in developing with (and) GNOME, so all I can say about book for beginners is whether they are using what I write in the way I think it’s supposed to be used; as far as I’m concerned, apart from a couple of issues, this book is indeed providing a solid base for people that want to begin developing for GNOME and with GNOME.

the price is pretty accessible, compared to the cost of similar books: I’ve paid much, much more for an introductory book on Core Animation, or on Perl books; the ebook version, if you dislike the dead tree version, comes in various formats, including PDF and Kindle.

I’m not going to give votes or anything: it’d be a pointless number on an equally pointless scale; but if you’re a beginner, I think this book may be fairly interesting to you, if you want to start coding with GNOME technologies.

1. this year is actually my 10th bug-versary

I went and saw Peter Singer’s keynote for The Tasmanian Writers’ Festival last night. Perhaps unsurprisingly he spoke on ethics and three big problems affecting the world now (extreme poverty, animal welfare and climate change) and how these things relate to, and perhaps exacerbate each other.

Two things in particular stuck with me, and I thought it worth noting them here.

1) Professor Singer is in a field known as Applied Ethics. At some time in the past there was only Ethics. My inference is that the latter group are talking about – and thinking about – ethics, but not actually behaving any differently as a result of their cogitations. I find this notion simultaneously hilarious and horrifying.

2) At one point while speaking about living ethically, Professor Singer said that if you look back at the end of each day and say to yourself “well, I didn’t lie, cheat or steal, and I didn’t maim anybody”, you’re setting the bar too low. It would be better, he suggested, to look back and say “what did I do to improve the world today, or to help someone else in some way?” This seems like a pretty good approach to me.

With the imminent release of gcc 4.8, GCC has finally switched to C++ as the implementation language.  As usual, LWN has excellent coverage.  Those with long memories will remember Linux trying to use g++ back in 1992 and retreating in horror at the larger, slower code.  The main benefit was stricter typechecking, particularly for enums (a great idea: I had -Wstrict-enum patches for gcc about 12 years ago, which was a superset of the -Wenum-compare we have now, but never got it merged).

With this in mind, and Ian Taylor’s bold assertion that “The C subset of C++ is as efficient as C”, I wanted to test what had changed with some actual measurements.  So I grabbed gcc 4.7.2 (the last release which could do this), and built it with C and C++ compilers:

1. ../gcc-4.7.2/configure –prefix=/usr/local/gcc-c –disable-bootstrap –enable-languages=c,c++ –disable-multiarch –disable-multilib
2. ../gcc-4.7.2/configure –prefix=/usr/local/gcc-cxx –disable-bootstrap –enable-languages=c,c++ –disable-multiarch –disable-multilib –enable-build-with-cxx

The C++-compiled binaries are slightly larger, though that’s mostly debug info:

1. -rwxr-xr-x 3 rusty rusty 1886551 Mar 18 17:13 /usr/local/gcc-c/bin/gcc
   
   text       data        bss        dec        hex    filename
   
   552530       3752       6888     563170      897e2    /usr/local/gcc-c/bin/gcc
2. -rwxr-xr-x 3 rusty rusty 1956593 Mar 18 17:13 /usr/local/gcc-cxx/bin/gcc
   
   text       data        bss        dec        hex    filename
   
   552731       3760       7176     563667      899d3    /usr/local/gcc-cxx/bin/gcc

Then I used them both to compile a clean Linux kernel 10 times:

1. for i in `seq 10`; do time make -s CC=/usr/local/gcc-c/bin/gcc 2>/dev/null; make -s clean; done
2. for i in `seq 10`; do time make -s CC=/usr/local/gcc-cxx/bin/gcc 2>/dev/null; make -s clean; done

Using stats –trim-outliers, which throws away best and worse, and we have the times for the remaining 8:

1. real    14m24.359000-35.107000(25.1521+/-0.62)s
   
   user    12m50.468000-52.576000(50.912+/-0.23)s
   
   sys    1m24.921000-27.465000(25.795+/-0.31)s
2. real    14m27.148000-29.635000(27.8895+/-0.78)s
   
   user    12m50.428000-52.852000(51.956+/-0.7)s
   
   sys    1m26.597000-29.274000(27.863+/-0.66)s

So the C++-compiled binaries are measurably slower, though not noticably: it’s about 865 seconds vs 868 seconds, or about .3%.  Even if a kernel compile spends half its time linking, statting, etc, that’s under 1% slowdown.

And it’s perfectly explicable by the larger executable size.  If we strip all the gcc binaries, and do another 10 runs of each (… flash forward to the next day.. oops, powerfail, make that 2 days later):

1. real    14m24.659000-33.435000(26.1196+/-0.65)s
   
   user    12m50.032000-57.701000(50.9755+/-0.36)s
   
   sys    1m26.057000-28.406000(26.863+/-0.36)s
2. real    14m26.811000-29.284000(27.1308+/-0.17)s
   
   user    12m51.428000-52.696000(52.156+/-0.39)s
   
   sys    1m26.157000-27.973000(26.869+/-0.41)s

Now the difference is 0.1%, pretty much in the noise.

Summary: so whether you like C++ or not, the performance argument is moot.

(Obligatory clarification: this post is entirely to do with weird bus stop naming. I have nothing to do with the “Church of Scientology”, nor the “Church of Christ, Scientist”. Yeah, take that, Google Juice.)

So I live near and work in Bowral. I was looking at a map of Bowral on Google Maps recently when I moused over a bus stop and saw this:

That was surprising, given that I know the town well, and definitely would have noticed if a Church of Scientology had sprung up! Having driven past it many times, I know the sign on the building underneath that marker actually says “First Church of Christ, Scientist“. Big difference.

Clicking on it reveals it is a bus stop. What’s odd is that I’ve never noticed a bus stop before there — although there is another one I know about just around the corner, on Bowral St. So according to Google Maps, there is a “Church of Scientology, Bendooley St“.

Not quite related to the issue, but I noticed a Google Ad for Scientology on the Street View marker for the bus stop:

I forgot to click on the link (and thus cost the Church of Scientology money). But I digress. And yes, the actual sign on the building says “First Church of Christ, Scientist”. Blurry Street View pic, but I know what the sign says — I drive past it every day on the way to and from work. There is a White Pages entry and even a content-free website for the building.

I noticed the “Source” said “131500.com.au”, which is the NSW Government site for train, bus, and ferry info. I couldn’t find that “bus stop” on the 131500.com.au site, so I did a Google search for site:131500.com.au scientology. And look what I found:

That links to what appears to be an internal (Plone–based) page on the 131500.com.au backend with a listing of all the bus stops. And the so-called “Church of Scientology, Bendooley St, Bowral” appears to be listed, which confirms the dodgy data was indeed sourced from 131500.com.au, rather than it being a Google error, which I initially thought:

Even though that appears to be an internal site, there didn’t seem to be anything confidential exposed. But hey, go for your life and see if you can find anything. I’m just interested in correctly naming bus stops, myself.

I discovered this in late February and haven’t reported it yet. Why? Quite simply, because Google have disabled the “Report a problem” link for that place. Good going, Google!

And the 131500.com.au website didn’t seem to have a logical place where I could report the problem, either. As usual, these sorts of things go unfixed, while the wheels of bureaucracy keep turning.

I left the last post minutes before Jessica McKellar's keynote at PyCon 2013. Unfortunately the keynote was cancelled. Raymond Hettinger followed a series of lightning talk with a keynote the theme of which was "Python is amazing" (I don't rememeber the exact words, but that's the main idea). YAY!

Getting started with automated testing, by Carl Meyer, was a great introductory talk, a good way to remind me that not writing tests is not (just) being lazy, it's counterproductive (and therefore leads to more work later).

Then I moved on to a couple of much-needed talk on community building: Scaling community diversity outreach, by (take a breath) Asheesh Laroia, Jessica McKellar, Dana Bauer and Daniel Choi, and, after lunch, How (Not) To Build An OSS Community by Daniel Lindsley. So far my contacts in London for programming have been by mentor in the TechAbility programme (my gateway drug pusher, the way I really got into programming - thank you Ben!), a couple of friends and of course my husband (but I tend not to talk too much about computers with him). I probably never realized how much I miss a "real life" (as opposed to "just IRC/internet") community since I got here. There are the GNOME beer events, of course! But the only London Python User Group I see online is "professionals only", so it doesn't look very newcomers-friendly... and there is no chapter of the PyLadies in London. (I was told that I should set one up - but I cannot do it on my own. And yes, dear Londoners, this is another cry for help!) All of this probably just means that I need a job, I am tired of doing things on my own. End of the rant, sorry.

Later in the afternoon, yet another fantastic talk. Lynn Root presented Sink or swim: 5 life jackets to throw to New Coders. It was perfectly tailored for the stage I am in my programming education: I know the basics, now what? Well, there is her website, then there is... teaching others. I sense a pattern here...

And then I went to rest, because the day after I had to present my poster and I was already too nervous.

The day after - the last day - was basically "hey, here's my poster". I had presented a poster only once in my life, but it was a much smaller (although quite big!) conference, and it was in another field (game theory).

And you know what? I loved it! (Even if it meant missing the keynotes... and the other posters.)

I talked about what I did last summer, why it's a good idea to do something like that, how you do it (and what you shouldn't do); most important, I spread the word about the Outreach Program for Women - all the leaflets but one or two were gone by the end of the session, and many people went away with the link to the website scribbled on a piece of paper!

I wasn't able to stand on my feet, afterwards. So: nap! And then, introduction to sprints...

...so that yesterday I managed to find something for me at the OpenHatch sprint. I helped with a webpage that was incomplete (a webpage that was... teaching - how to contribute to an OpenSource project. Mmmmh, more of that pattern), and I cleaned up the wiki a bit. I had never heard of OpenHatch before this conference, but I hope this is the beginning of a beautiful frienship (to use the words of Claude Rains in Casablanca).

And now I should be packing (how will I stuff all the t-shirts I got into my suitcase?) and go on my way...

So: a big thank you to PyCon, to the PSF (who generously sponsored my travel and my hotel!), to all the wonderful people I met, too many to fit in a post here (also: I am terrible with names, I would probably forget someone). And... maybe see you in Florence later this year!

PS: I wrote the post this morning (PST). It's now almost night (also PST, I don't want to know what time it is in London), and I am at the airport - flight delayed by approximately 3 hours. I use the airport connection to give you... the poster! With a huge thank you to Marina Zhurakhinskaya and Meg Ford who sent me so many useful suggestions.

I’ve been kindly invited for the first Distro Recipes event in Paris the 4th and 5th of April.

As I have an internal HP meeting on the 4th, I’ll be only available at the end of that day, but will present on the 5th how Hardware manufacturers work with Linux distributions, giving the example of HP. I’ll also monitor a round table aound “Linux distributions: differences and commonalities” where we will try to have polite discussions about what makes a distribution unique, and what is instead worth sharing by collaborating. Finally I’ll also present during the lightning talks “Project-Builder.org: packaging for multi-OS Open Source Projects“

So won’t have that too much time outside of presentations, as you can see, but would be happy anyway to meet with MondoRescue or Project-Builder.org or HP/Linux users and talk with them.

Anyway a great event to be in, as the list of speakers is really interesting, all majors distros being represented, and for sure very interesting new contacts to make, and hopefully the curiosity to discover these other distros that you don’t use Come for the same reasons, and see you there !

As part of a big Logstash project at Mozilla (more on that to come), I was looking for an HAProxy module for Puppet, stumbling across the official Puppetlabs module in the process.  I’m told that this module works fairly well, with the caveat that it sometimes outputs poorly-formatted configuration files (due to a manifestly buggy implementation of concat).  Furthermore, the module more or less requires storeconfigs, which we do not use in our primary Puppet system.

Long story short, while I never ended up using HAProxy as part of the project, I did remix the official module to solve both of the aforementioned issues.  From the README :

This module is based on Puppetlabs’ official HAProxy module; however, it has been “remixed” for use at Mozilla. There are two major areas where the original module has been changed :

• Storeconfigs, while present, are no longer required.
• The “listen” stanza format has been abandoned in favour of a frontend / backend style configuration.

A very simple configuration to proxy unrelated Redis nodes :

  class { 'haproxy': }

  haproxy::frontend { 'in_redis':
    ipaddress       => $::ipaddress,
    ports           => '6379',
    default_backend => 'out_redis',
    options         => { 'balance' => 'roundrobin' }
  }

  haproxy::backend { 'out_redis':
    listening_service => 'redis',
    server_names      => ['node01', 'node02'],
    ipaddresses       => ['node01.redis.server.foo', 'node02.redis.server.foo'],
    ports             => '6379',
    options           => 'check'
  }

If that sounds interesting to you, the module is available on my puppetlabs-haproxy repo on Github. Pull requests welcome !

I know. It’s been a long process. And I’m not even fully satisfied with the results, as we do have unresolved bugs in this version. But as it also fixes some critical others that were expecting for a long time (3+ months), I didn’t want to delay more the release of this version.

So here it is. mondo 3.0.3, mindi 2.1.4 and mindi-busybox 1.18.5-3 are now availble from the master ftp server. And as given in the announce, most of the distributions (nearly 100 of them) also have packages ready for use. Once more couldn’t do it without project-builder.org

The detailed change logs are available for mindi, mindi-busybox and mondo.

Among the main fixes brought by this version, a much better support of SLES 10, SLES 11 SP1, SLES11 SP2, especially for LVM and device exclusion, grub, keyboard.

I also received very, very valuable contributions from users, which helps producing a better version of course, and reduce the time of the resolution of the problem in the upstream version. Thanks again for helping ! Including on some very old bugs. And Victor being one of the most prolific.

There are even some new features such as the support of swaplabel or the inclusion of all tools part of minimal.conf now in the first boot part.

As said earlier, some bugs ar not fixed yet. I passed most of the week trying to fix the automatic mode where CTRL-ALT-DEL doesn’t work. I added support in 3.0.3 of an inittab file, but whatever the conf I use fo now, it doesn’t trigger a reboot when using the magic key or sending it from the KVM monitor. So be warned and use with care. I’ll report on the busybox ML as I still have the issue with 1.20.2, and try to get help from this community. Could well trigger the release of a new mindi-busybox (and maybe mindi) soon after this one.

In the serie of un-fixed bugs for lack of time for this time are some that a user reported as more important to fix soon:

• Ticket #628: 3.0.2-1 Fail to Restore from External Hard Disk – This is a very big issue here.
• Ticket #641: Specifying the Same Backup to NFS via CLI=Success via GUI=Fail – Although it is possible to work around this problem it is preventing the creation of exceptional backups by users with little knowledge of Mondo. A fix will be most welcome and will present a more appropriate impression of Mondo.
• Ticket #640: mondo.tmp.xxxxx & mondo.scratch.xxxxx Issues – This is not a critical issue. A fix will help with backup management and housekeeping tasks.

I globally agree these are also important to fix (even if less than Ticket 627). I also think we need to work on Ticket 656 as well. I’d like to get your comments as well in order to prioritize the fixes in the next version, as I can’t work full time on MondoRescue, so need to choose where to put my efforts.

I should have probably written something in the last days, but I was, well... overwhelmed.

I arrived in Santa Clara (after about 12 hours on a plane, and that was just to San Francisco) on the 12th - in the afternoon, local time. A few hours to rest, to get used to the jet lag, and then...

TUTORIALS!

A hands-on introduction to Python for beginning programmers, by Jessica McKellar, was the perfect start: a smooth (re)introduction to the basics - it made me feel well. Hands-on intermediate Python, by Matt Harrison, followed. Fantastic. I learned so much, in such a short time: the balance of theory and practice was the right one. And the handouts are going to provide a very interesting (re)read for the future.

And on Wednesday night, some of the OPW women of the area had a small outing... (A special thank you to Aleta Dunne for the organization!)

Then, Thursday. Scripting: from Hard-drive to Github to PyPI, also by Matt Harrison, was advertised as "Novice" - but unfortunately I found it a bit too fast-paced for me. More precisely: the Python in it was not too difficult, but the "software engineering practice" level was above me. I look forward to re-reading the handouts in a less hectic environment. In the afternoon, there was Jessica McKellar again with Contribute with me! Getting started with open source development. Some much needed git practice - I will just say that.

And then... PARTY! I met nice people, I browsed the booths of the job fair getting contacts (and merchandise)... great time.

The conference started on Friday. Moving remarks by Jesse Noller on "changing the future" were followed by a keynote by Eben Upton... and the announcement that all of us (all 2500, and yes, that is two-thousand-five-hundred!) were going to get a Raspberry Pi. I will let you imagine the enthusiasm in the room.

I had to pick a talk. I was undecided between Jessica McKellar's How the Internet works and Esther Nam's How to Except When You're Excepting: I went for the first, and it was a fantastic introductory talk. I have to find a good book on the subject... (yes, dear reader, that is a cry for help!)

Then I spent some time browsing around, and after lunch I crashed in the Ada Initiative booth. I have rarely seen such a welcoming crowd, so great at making you feel at ease... Talking about the Imposter Syndrome mixed with nail painting (don't ask), and I also got a lot of good advice for my career.

I was really looking forward to a couple of talks on documentation - but I was feeling terrible (one of my dizziness attacks, a particularly nasty one) and I moved back to the hotel for a quiet night. I hope that the recording will be put online soon.

Now I am here, completing this post... But wait! It's almost time for the first keynote of the day! I'd better hurry! Talk to you soon! Bye!

The folk at Packt Publishing sent me an e-copy of GNOME 3 Application Development Beginners Guide the other day. Since I find myself with a couple of weeks off (more on that another time) I’m going to be reading it and writing a review.

The book weighs in at 366 pages and purports to cover GLib, GTK+, GStreamer, E-D-S, WebKit, desktop D-Bus APIs, i18n and unit testing in both Javascript (via Seed) and Vala.

Hopefully I will get it read in the next couple of weeks and get my thoughts jotted down. I am not getting anything except an e-copy of the book for my trouble so you can trust me to be brutally honest

I must confess to reading this book rather quickly over two nights, so I’m sure some nuances were missed.

Like a lot of Australians, and presumably Queenslanders, I do have a bit of an unhealthy interest in our criminal past. I think I was in grade six when some of the Fitzgerald inquiry was finishing up (or recommendations being handed down, or some such) and getting interested then. Personally, I make a lot of use of our Joh given right to mix metaphors.

Overall I was disappointed by the book. There’s clearly a lot of research gone into it, which is great, and the narrative ties it together reasonably well. There are editing issues and some ham fisted attempts at pop psychology. A glossary for all the colloquialisms would have been useful. The worst thing for me though, is that the main interviewee, Lewis, comes out with nary a red cross against his name. Time and time again Lewis is implicated in the skulduggery of the time, but he denies the worst of it at every turn.

I feel I can’t quite call the book a whitewashing of Lewis’s history yet, as there’s still another half to go, Lewis may well let it all out then and redeem himself. But I also feel the author has let us down by not digging deeper on Lewis. Maybe that was part of the interview deal; or maybe Lewis still has powerful friends.

On a matter that I really should disclose, it appears that family members are named in the book, in none too good a light.

Linux Conf Australia 2013, (alias linux.conf.au, alias LCA) was an amazing experience, and it’s difficult to summarise it succinctly, but there are definitely some highlights and important take-aways that I’d like to make special note of.

My fundamental reason for attending was that Ben Kero and I were invited to do a talk regarding Puppet (an important infrastructural tool that we use at Mozilla).  The talk, like all of the presentations given by Mozillians at LCA this year (and there were more than a few), was very well-received, and delivered to a packed auditorium – in fact, people had to be turned away at the door beforehand !

Since our talk bumped up against the lunch period, we had occasion to stay in the space and launch an informal panel discussion that included other Mozillians as well as representatives from a number of other Open Source companies and organisation.  We engaged on a variety of topics, ranging from IT-centric to questions about the future of the web, and the importance of open standards and market competition.  It was only when the organisers forced us out that the auditorium was finally cleared.

Interestingly, from the moment I arrived in Australia, and through until the very last day, I found myself acting as an ambassador for Mozilla.

For example, a day before the conference even started, I was part of an informal debate concerning the future of mobile and the importance of FirefoxOS within it.  Other participants included a Ubuntu employee and a number of hardware hackers who – as it turned out – were already trying to port FirefoxOS to other types of phones.

Another example even occurred at a local cricket match.  I was wearing my Firefox T-shirt (which usually generates interest), and ended up doing a some impromptu demos of Firefox for Android to a couple of different groups – the highly satisfying result was that a handful of people downloaded and installed it on the spot !

These, and more, are all opportunities to engage people about Mozilla, and in many cases, to introduce our values and mission to entirely new audiences.  During this and other conferences, I’ve found that even long-time users of Firefox and Thunderbird (for example) are often not even aware of who and what Mozilla is, and what we’re about.  I try my best to be a good ambassador, and I’m proud to represent the organisation wherever, and whenever I can.

From a personal perspective, the conference had two major benefits: it was an excellent learning experience, and a fantastic opportunity to spend time in-person with a number of my co-workers in the IT/Ops team.

Of note, I had the pleasure of attending a number of talks and presentations about technology that we use in our environment that have had a direct impact in my work-flow already.  Beyond the directly-applicable benefits, I was particularly inspired by a presentation about programming for embedded devices (shout-out to Bunnie Huang) which, due to peculiar technical constraints, requires a very precise and measured approach to development.  These ideals could – and should – be carried through outside of the embedded world.

In summary, LCA 2013 was an excellent opportunity to learn, teach, and engage the public concerning technology in specific, Mozilla in particular, and open source in general.  I’m already looking forward to next year !

I finished up at Google last week and am now working at NICTA, an Australian ICT research institute.

My work with Google was exciting and I learned a lot. I like to think that Google also got a lot out of me – I coded and contributed to some YouTube caption features, I worked on Chrome captions and video controls, and above all I worked on video accessibility for HTML at the W3C.

I was one of the key authors of the W3C Media Accessibility Requirements document that we created in the Media Accessibility Task Force of the W3C HTML WG. I then went on to help make video accessibility a reality. We created WebVTT and the <track> element and applied it to captions, subtitles, chapters (navigation), video descriptions, and metadata. To satisfy the need for synchronisation of video with other media resources such as sign language video or audio descriptions, we got the MediaController object and the @mediagroup attribute.

I must say it was a most rewarding time. I learned a lot about being productive at Google, collaborate successfully over the distance, about how the WebKit community works, and about the new way of writing W3C standard (which is more like pseudo-code). As one consequence, I am now a co-editor of the W3C HTML spec and it seems I am also about to become the editor of the WebVTT spec.

At NICTA my new focus of work is WebRTC. There is both a bit of research and a whole bunch of application development involved. I may even get to do some WebKit development, if we identify any issues with the current implementation. I started a week ago and am already amazed by the amount of work going on in the WebRTC space and the amazing number of open source projects playing around with it. Video conferencing is a new challenge and I look forward to it.

While preparing the new version of MondoRescue (test packages on their way for final remarks before having next stable version), I was trying to make a missing version of mindi-busybox for a colleague still using RHEL 3. I quickly realized that my promise to make it (hoping to just type pb -p mondorescue -m rhel-3-i386,rhel-3-x86_64 -r tags/3.0.2 cms2vm) was not so easy to perform

It happens that the version we use (1.18.5) doesn’t compile out of the box for such an old beast as a RHEL 3 with it’s 2.4 kernel. So I had to make some adaptations to the code, in order to have it compile for RHEL3, but still working for newest versions. At least, I have something which seems to be working for RHEL 3, RHEL 6 and Fedora 18, so now I’m building for all my supported distros which now are 120+ and will probably take up to mid day tomorrow.

For those of you interested, had issues with the main Makefile, statfs.h (HAVE_SYS_STATFS_H shouldn’t be set in RHEL 3 case) and a missing BLKGETSIZE64 macro, and the corresponding patch is at: http://trac.mondorescue.org/changeset/3085

With that I’m also trying to fix 2 remaining bugs (http://trac.mondorescue.org/ticket/627 and

http://trac.mondorescue.org/ticket/651) which should have no other impact. The first one is still not fully tested as corrected.

My goal is to publish the next stable version 3.0.3 of MondoRescue this week-end, for my birthday

I had to put together a small web app the other day, using SQLAlchemy and Flask. Because I hate writing code multiple times, when I can do things using a better way, I wanted to be able to serialise SQLAlchemy ORM objects straight to JSON.

I decided on an approach where taking a leaf out of Javascript, I would optionally implement a tojson() method on a class, which I would attempt to call from my JSONEncoder¹.

It turns out to be relatively simple to extend SQLAlchemy’s declarative base class to add additional methods (we can also use this as an excuse to implement a general __repr__().

from sqlalchemy.ext.declarative import declarative_base as real_declarative_base

# Let's make this a class decorator
declarative_base = lambda cls: real_declarative_base(cls=cls)

@declarative_base
class Base(object):
    """
    Add some default properties and methods to the SQLAlchemy declarative base.
    """

    @property
    def columns(self):
        return [ c.name for c in self.__table__.columns ]

    @property
    def columnitems(self):
        return dict([ (c, getattr(self, c)) for c in self.columns ])

    def __repr__(self):
        return '{}({})'.format(self.__class__.__name__, self.columnitems)

    def tojson(self):
        return self.columnitems

We can then define our tables in the usual way:

class Client(Base):
    __tablename__ = 'client'

    ...

You can obviously replace any of the methods in your subclass, if you don’t want to serialise the whole thing. Bonus points for anyone who wants to extend this to serialise one-to-many relationships.

And what about calling the tojson() method? That’s easy, we can just provide our own JSONEncoder.

import json

class JSONEncoder(json.JSONEncoder):
    """
    Wrapper class to try calling an object's tojson() method. This allows
    us to JSONify objects coming from the ORM. Also handles dates and datetimes.
    """

    def default(self, obj):
        if isinstance(obj, datetime.date):
            return obj.isoformat()

        try:
            return obj.tojson()
        except AttributeError:
            return json.JSONEncoder.default(self, obj)

Cutting edge Flask provides a way to replace the default JSON encoder, but the version I got out of pip does not. This is relatively easy to work around though by replacing jsonify with our own version.

from flask import Flask

app = Flask(__name__)

def jsonify(*args, **kwargs):
    """
    Workaround for Flask's jsonify not allowing replacement of the JSONEncoder
    in my version of Flask.
    """

    return app.response_class(json.dumps(dict(*args, **kwargs),
                                         cls=JSONEncoder),
                              mimetype='application/json')

If you do have a newer Flask, where you don’t have to replace jsonify, you can also inherit from Flask’s JSONEncoder, which already handles things like datetimes for you.

1. The tojson() method actually returns a Python dict understandable by JSONEncoder

Composite and Swap — Getting it Right

Where the author tries to make sure DRI3000 is going to do what we want now and in the future

DRI3000

The basic DRI3000 plan seems pretty straightforward:

1. Have applications allocate buffers full of new window contents, attach pixmap IDs to those buffers and pass them to the X server to get them onto the screen.

2. Provide a mechanism to let applications know when those pixmaps are idle so that they can reuse them instead of creating new ones for every frame.

3. Finally, allow the actual presentation of the contents to be scheduled for a suitable time in the future, generally synchronized with the monitor. Let the client know when this has happened in case they want to synchronize themselves to vblank.

The DRI3 extension provides a way to associate pixmap IDs and buffers, and given the MIT-SHM prototype I’ve already implemented, I think we can safely mark this part as demonstrably implementable.

That leaves us with a smaller problem, that of taking pixmap contents and presenting them on the screen at a suitable time and telling applications about the progress of that activity.

In the absence of compositing, I’m pretty sure the initial Swap extension design would do this job just fine, and should resolve some of the known DRI2 limitations related to buffer management. And, I think that goal is sufficient motivation to go and implement that. However, I wanted to write up some further ideas to see if the DRI3000 plan can be made to do precisely what we want in a composited world.

The Composited Goal

To make sure we’re all on the same page, here’s what I expect from the Swap extension in a composited world:

1. Application calls Swap with new window pixmap

2. Compositor hears about the new pixmap and uses that to construct a new screen pixmap

3. Compositor calls Swap with new screen pixmap

4. Vertical retrace happens, executing the pending swap operation

5. Compositor hears about the swap completion for the screen

6. Application hears about the swap completion for its window

In particular, applications should not hear that their swap operations are complete until the contents appear on the screen. This allows for applications to throttle themselves to the screen rate, either doing double or triple buffering as they choose.

I didn’t add steps here indicating buffers going idle or being allocated, because I think that should all happen ‘behind the scenes’ from the application’s perspective. Many applications won’t care about the swap completion notification either, but some will and so that needs to be visible.

Redirected Swaps?

Owen Taylor suggested that one way of getting the compositor involved would be to have it somehow ‘redirect’ Swap operations, much like we do with window management operations today. I think that idea may be a good direction to try:

1. Application calls Swap with new window pixmap

2. Swap is redirected to compositor, passing along the new window pixmap

3. Compositor constructs a new screen pixmap using the new window pixmap

4. Compositor calls Swap on the screen and the window, passing the new screen pixmap and the new window pixmap. When the screen update occurs, the screen and the window both receive swap completion events.

This has the added benefit that the X server knows when the compositor is expecting window pixmaps to change like this — the compositor has to explicitly request Swap redirection.

Window Pixmap Names and GEM Buffer Handles

One issue that swapping window pixmaps around like this brings up is how to manage existing names for the window pixmap. Right now, applications expect that window pixmaps will only change when the window is resized. If the Swap extension is going to actually replace the window pixmap when running with a suitable compositor, then we need to figure out what the old names will reference.

Are there non-compositor applications using NameWindowPixmap that matter to us? How about non-compositor applications using TextureFromPixmap to get a GEM handle for a window pixmap? For now, I’m very tempted to just break stuff and see who complains, but knowing what we’re breaking might be nice beforehand.

Idling Pixmaps

When an application is done drawing to a window pixmap and has passed it off to the X server for presentation, we’d like for that pixmap to be automatically marked as discardable as soon as possible. This way, when memory is tight, the kernel can come steal those pages for something critical. Of course, applications may not want to let the server mark the pixmap as idle after being used, so a flag to the Swap call would be needed.

Ideally, the pixmap would become idle immediately after the pixmap contents have been extracted. In the absence of a compositor, that would probably be when the Swap operation completes. With a compositor running, we’d need explicit instruction from the compositor telling us that the window pixmap was now ‘idle’:

┌───
    SwapIdle
    drawable: Drawable
    pixmap: Pixmap
      ▶
└───

Furthermore, the application needs to know that the pixmap is in fact idle. I think that we’ll need a synchronous X request that marks a buffer as ‘no longer idle’ and have that return whether the buffer was discarded while idle. It doesn’t seem sufficient to use events here as the application will need to completely reconstruct the pixmap contents in this case. This reply could also contain information about precisely what contents the pixmap does contain.

┌───
    SwapReuse
    drawable: Drawable
    pixmap: Pixmap
      ▶
    valid: BOOL
    swap-hi: CARD32
    swap-lo: CARD32
└───

Pixmap Lifetimes and Triple Buffered Applications

If we redirect the Swap operation and send the original application window pixmap ID to the compositor, what happens when the application frees that pixmap before the compositor gets around to using the contents?

Surely the Compositor must handle such cases, and not just crash. However, I’m fine with requiring that the application not free the pixmap until told by the compositor.

Kogan Mobile is the newest virtual telco in Australia [1]. They resell Telstra 3G (not NextG or LTE) and while their coverage isn’t as good as the full Telstra service it’s more than adequate for my needs as they provide 3G coverage to 97% of the population and 2G+3G coverage to 98.5%. Their coverage is probably a lot better than Three who had the worst record of network coverage in Australia yet managed to always provide coverage where I wanted it – I was a happy Three customer for more than 6 years.

Kogan’s main selling point is that they offer unlimited calls to Australian mobile phones and land-lines and unlimited SMS for a pre-paid fee of only $300 per annum ($25 per month on average). My parents have been getting unpleasantly large phone bills which have considerably more than $25 of calls to mobile phones every month so an obvious solution for them is to sign up for a Kogan mobile phone and use it for all such calls. There are other ways my parents could save money on calls (such as VOIP) but a mobile phone is easiest and offers other benefits such as running Android apps (when compared to using a non-smart phone).

6G of Data!

Kogan also offers 6G of data per month, the down-side to this is that they bill in 1MB increments per “session”. I was worried that this might be per TCP connection or something else silly but I decided to sign my parents up for it as they aren’t going to use a lot of data (they claim that they don’t want to use the Internet on their phone but I know better). I’ve done some tests on the SIM I got for my parents. For testing purposes I installed the Kogan SIM in my wife’s new Nexus 4 and had it provide Wifi net access to my phone while we were playing Ingress.

So far after 2 days which involved a reasonable amount of Ingress (I reached level 7) as well as all the usual stuff that happens in the background for two phones (checking email, news, weather, etc) Kogan considers that 177MB have been used out of the 6144MB for the month, which means that even with what is an unusual amount of traffic for us the account in question still isn’t going to use half the quota for the month. Now that 10 days have elapsed with less intense usage Kogan considers that a total of 1373MB have been used.

Kogan also don’t seem to mention whether they bill for transmitted data. I used the 3G Watchdog app to measure the amount data transferred, the above table has the amounts of data that 3G Watchdog considers were sent and received along with the amount that is listed by the Kogan Android app. My past experience with 3G Watchdog and Virgin Mobile is that it’s usually quite accurate but has been over-reporting the data transfers recently (I think that Virgin is only billing me for downloads while 3G Watchdog counts uploads). So the relatively small difference between the 3G Watchdog report and what Kogan thinks I’ve done means that either rounding the “session” up to the nearest meg doesn’t make any significant difference (which would imply that a “session” can be a long time) or that Kogan isn’t counting uploaded data and the session rounding up only adds about 40% to the total recorded transfer.

My current plan with Virgin Mobile gives me 1.5G per month of quota, so as long as Kogan’s rounding doesn’t increase the recorded data transfer by a factor of 4 I will still be able to transfer more data with Kogan while paying less. One disadvantage of using Kogan is that I might have to tweak programs like my email program to poll less frequently to avoid excessive session charges (a program polling every 5 minutes would use up the 6G quota in 21 days if each poll counted as a session) – although current tests indicate that this won’t be necessary. But the up-side is that there are no extra fees with Kogan, they merely restrict data access – for my use and that of most people I know it’s better to have data access cut off than to receive a large bill.

The Kogan Android App

Kogan has an Android app that will give the status of your account and allow you to change the plan etc. This is quite nice but one major disadvantage is that it’s also a sales app for the Kogan online store. This is bad for the user as some aspects of what I consider the core functionality are limited (for example there’s no way to force a poll of the data usage count or determine how current the data is). But there’s an obvious advantage to Kogan in providing a way to sell their goods that is going to be used by every customer of Kogan Mobile.

The sales part of the app isn’t very functional IMHO, it doesn’t seem to have basic functionality such as sorting a list of items by price.

Benefits of Kogan

6G of data is a lot!

$300 per annum is quite cheap, anyone who makes any serious use of phones will be paying more than that in Australia.

Lack of extra fees means that there is little need to restrict net access. I can risk getting cut off near the end of the month but I can’t risk the potential for hundreds of dollars in excess fees.

The Kogan app shows me the data used so I will probably uninstall 3G Watchdog, having one less program running is a good thing.

You get a free SIM (value $5) when you buy a phone from Kogan.

Disadvantages of Kogan

They are new to the Telco business and admit that their customer service is lacking due to unexpected demand.

If you order a SIM now they state that it will arrive in April. Apparently they are deliberately delaying orders because they can’t cope with demand.

The included call quota doesn’t include international calls. While unlimited free calls in Australia is great if you make many international calls then this could end up costing you more. Other mobile telcos such as Lebara offer good deals for International calls, it could be an option to use a Lebara SIM with an old non-smart phone while using Kogan for your smart phone.

I am concerned about the lack of detail about how data is accounted. If the definition of a “session” changes then 6G could turn out not to be enough. As Kogan is reselling a Telstra service it is possible that Telstra could change the deal without Kogan being able to stop them.

Conclusion

I will move my phone and my wife’s phone to Kogan ASAP. My general idea is to sign us up for Kogan about 2 weeks apart, so if one phone runs out of the 6G data quota then the other phone can be used as a Wifi access point for 2 weeks. If the phones don’t both have their quota end at the same time then there is less chance of both phones running out during a high traffic month.

• [1] https://www.koganmobile.com.au/

This is a short post to announce the release of efitools version 1.4. The packages are available here:

http://download.opensuse.org/repositories/home:/jejb1:/UEFI/

Just select your distribution (various versions of Debian, Ubuntu, Fedora and openSUSE).

The main additions over version 1.3 is that there are two new utilities: efi-readvar and efi-updatevar that allow you to read and manipulate the UEFI signatures database from linux userspace.  The disadvantage of these tools is that they require the new efivarfs filesystem to be mounted somewhere in the system (efivarfs was added in kernel 3.8, so you either have to be running a very recent distribution, like openSUSE Tumbleweed, or you have to build your own kernels to use it).    To mount the efivarfs filesystem, just do

modprobe efivars

mount -o efivarfs none /mnt (or some other useful mount point)

As long as this is successful, you can now use the efi- tools to read and write the secure variables.  In order to write the variables in User Mode, you must have the private part of the Platform and Key Exchange Keys available.  So, assuming your private part of the platform key is PK.key, you can add your own KEK with

efi-updatevar -a -c KEK.crt -k PK.key KEK

And then add your own signature database key with

efi-updatevar -a -c DB.crt -k KEK.key db

Where KEK.crt and DB.crt are X509 certificates.

Note that by default, files in the efivarfs filesystem are owned by root and have permission 0644, so efi-readvar can be executed by any user, but efi-updatevar needs to be able to write to the variable files (i.e. would have to run as root).

There’s also extensive man pages documenting all the options for both efi-readvar and efi-updatevar.

As you can see with the few messages I had time to post on this blog, the end of 2012 start of 2013 has been pretty busy, and I’m late in delivering the 2 projects I’m leading. So this week, I decided it was time to make a 0.12.2 version of pb, and make it available. Was asked by my colleagues of FOSSology, specially to add Fedora 18 support, so I also build my VMs to make packages on this distro.

BTW, a bit of ranting for Fedora once more: no perl by default , and no ifconfig nor route command either which is breaking MondoRescue of course, and I guess tons of other software around. These guys don’t care about past, but don’t provide compatibility tools either !! So I’ll have to make new patches, just to do the same as what was done, but with another command again. Not to speak of systemd which I still have issues to deal with That doesn’t make stuff go faster !

Anyway, the version is now out, no official bugs fixed, but a lot of small stuff here and there which were desrving a release. No time to test Fedora 17/18 VE yet, so you’ll have to do that yourself if you want. I also had a look at virsh usage in combination with pb, and it’s again not as easy as it could seem to be. Especially port redirection I’m easily using by launching qemu-kvm manually with the -redir or hostfwd option doesn’t seem to be possible with the user mode network through virsh (neither manager nor CLI). Will have to post on their ML to see how they do that, if they can !

And MondoRescue has been very late. I really need to publish a verion, but I still have some blocking bugs I really would like to get rid of: CTRL+ALT+DEL not working anymore during restore, some LVM issues on RHEL, some grub issues on SLES… Hopefully at the end of next week I’ll have made progresses.

Of course patches are much easy to integrate, but I receive more bug reports than them

And also back to preparing the HP internal TES event, Solutions Linux confs, submitting to LinuxCon, working on an FLOSS ITIL stack, learning more OpenStack, looking at Intel’s TXT…. so many things I’d like to do or learn and will never have time to ! Maybe at least I’ll talk about that another time.

x-on-resize: a simple display configuration daemon

I like things to be automated as much as possible, and having abandoned Gnome to their own fate and switched to xfce, I missed the automatic display reconfiguration stuff. I decided to write something as simple as possible that did just what I needed. I did this a few months ago, and when Carl Worth asked what I was using, I decided to pack it up and make it available.

Automatic configuration with a shell script

I’ve had a shell script around that I used to bind to a key press which I’d hit when I plugged or unplugged a monitor. So, all I really need to do is get this script run when something happens.

The missing tool here was something to wait for a change to happen and automatically invoke the script I’d already written.

Resize vs Configure

The first version of x-on-resize just listened for ConfigureNotify events on the root window. These get sent every time anything happens with the screen configuration, from hot-plug to notification when someone runs xrandr. That was as simple as possible; the application was a few lines of code to select for ConfigureNotify events, and invoke a program provided on the command line.

However, it was a bit too simple as it would also respond to manual invocations of xrandr and call the script then as well. So, as long as I was content to accept whatever the script did, things were fine. And, with a laptop that had a DisplayPort connector for my external desktop monitor, and a separate VGA connector for projectors at conferences, the script always did something useful.

Then I got this silly laptop that has only DisplayPort, and for which a dongle is required to get to VGA for projectors. I probably could write something fancy to figure out the difference between a desktop DisplayPort monitor and DisplayPort to VGA dongle, but I decided that solving the simpler problem of only invoking the script on actual hotplug events would be better.

So, I left the current invoke-on-resize behavior intact and added new code that watched the list of available outputs and invoked a new ‘config’ script when that set changed.

The final program, x-on-resize, is available via git at

git://people.freedesktop.org/~keithp/x-on-resize

I even wrote a manual page. Enjoy!

So, how did the Documentation sprint go? Not bad, not bad at all. I wish I would have been feeling better (I was quite under the weather, I don't know why) - and that I could have been a bit more productive (and that I could have gone for a few more beers with the others), that's all I can complain about.

Anyway!

• I finally wrote the "generic widget" example for the Python GTK+ tutorial for beginners. I also learned a bit of Cairo in the process.
• I reviewed (with a precious help of Kat Gerasimova) the above mentioned tutorial as a whole.
• Last but not least, with Allan Day and Dave King we planned a complete rewriting of the presentation of the developer docs.

The last point is the reason for the title of this post. The idea is to separate the tutorials for beginners (that may be either a 10 minutes tutorial on "how to do something" or a more extensive introduction to a library) and the documentation directed at more experienced programmers. The former would end in a separate area in the website, whereas the latter would be under "tutorials" in the main website. Allan Day has designed a wonderful mock-up:

So, what's next?

My idea is to start by splitting the current Python GTK+ tutorial in two versions. The first one should be a tutorial for beginners, structured as the way the Tutorial for beginners is now: a gradual path through the widgets with some theory; the pages being a gradual building of a nice example, on the model of Taryn Fox's tutorial for JavaScript. The second one should be something for more experienced programmers, structured on the model of the "GTK+ widgets sample code" page: the pages being almost an API, but with a medium-difficulty example to illustrate it.

I would then revise the Guitar Tuner and Image Viewer tutorials to make them a first example of "10 minutes tutorials" (for GStreamer and GTK+, respectively).

And then... I could do the same for the JavaScript tutorials; I am quite new to the language, so I would love some help - but on the other hand I was quite new to Python last year and that helped me a lot in walking in the shoes of the beginner for which the tutorial was intended! And/or I could continue with Python and do something like what we have for GTK+ for some other library (there is a nice Cairo tutorial that could be a great starting point).

So: many plans for the future have come out of this Documentation sprint. Many pleasant meetings, a lot of new ideas, a better understanding of little (but not less relevant!) things (I never knew that there was a difference between the capitalization of the titles in UK and US English - now I know)...

So: a "see you soon" to everyone who was there, a GIGANTIC thank you to our hosts in Brno, especially Florian Nadge, who woke up at 4am to pick me up at the airport on the first day! and that accompanied us almost everywhere.

And, of course: thanks to the GNOME Foundation who sponsored my travel!

And... see you soon for GUADEC, Brno!

So I found a bit of time to hack on my project today. Today’s task was to load and validate data coming from Geoscience Australia’s SRTM digital elevation model data, which I downloaded from their elevation data portal last week.¹ The data is Creative Commons, so I might just upload it somewhere, if I can find a place for 2GB of elevation data.

This let me load elevation data in a 3 arcsecond (about 100m) grid, which I did using the ubiquitous GDAL via its Python API. Initial code is here. It doesn’t do anything super clever yet, like check and normalise the projection, because I don’t need to.²

Looking at plots of values can give you a gist of what’s what (oh look, it goes out to sea, and then the data is masked out) but it doesn’t really validate anything. I could do validation runs against my GPS tracks, but for a first pass, I decided it would be easier to validate using Google’s Elevation API. This is a pretty neat web service that you make a request to, and it gives you back some JSON (or XML). There are undoubtedly Python APIs to access this, but it’s pretty easy to do a simple call with urllib2 or httplib. I chose to reuse my httplib Client wrapper from my RunKeeper/HealthGraph API. I wrote it directly in the test.

For a real unit test, I would have probably calculated the residuals, and ensured they sat within some acceptable range, but I’m lazy, so instead I just plotted them together. Google’s data, you will notice, includes bathymetry, which is actually pretty neat.

1. Note to the unwary, seems to be buggy in Chrome?
2. I did write a skeleton context manager for gdal.Open. I say skeleton because it doesn’t actually do anything smart like turning errors from GDAL into Exceptions, because I didn’t have any errors to handle.

This evening I was driving through one of the inner suburbs of Melbourne when a man flagged me down. He said that his mother was dying and he needed a taxi ride to some hospital far away and needed to borrow $200. He was saying something about his phone, I wasn’t sure if he was planning to give me his phone number so I could call him to ask for repayment or offering his phone as collateral on the loan (incidentally a well known scam is to offer a stolen phone as collateral for a loan, it’s a way of selling a locked phone that doesn’t have cables).

I’ve encountered many beggars over the years, but he was by far the most serious about it – he demonstrated the level of desperation that I’ve only previously seen documented in history books and reports from travelers who visited developing countries. I will never know if his mother was dying, there are lots of other reasons why someone might urgently need cash (most of which won’t get much sympathy).

I gave him $20 as a gift. If his story was legitimate then I gave him 10% of what he needed so he only had to find another 9 people willing to do the same. If he was lying then I can afford to lose $20. In any case I definitely wasn’t going to do what he asked and withdraw hundreds of dollars from an ATM for him. Also regardless of whether he was telling the truth I didn’t want to have him repay me, if he’s telling the truth then I’m happy to give money to him and if he’s not then I’m better off avoiding him in future. If I had $50 I would probably have given it to him, but $200 is too much.

As I drove off I looked in my rear-vision mirror and saw him running between cars on the road trying to flag someone else down. Running through moving traffic on a Saturday night is another indication of how serious he was, generally someone who’s in a good state of mind and wants a long and healthy life won’t do that.

I got a lot of complaints about the lack of Debian or Ubuntu packages.  Apparently alien doesn’t work all that well, and some of the packages (like libc) are too new anyway on openSUSE.  So I finally figured out how to get OBS to build debian packages.  You can find them here:

http://download.opensuse.org/repositories/home:/jejb1:/UEFI/

I should note that don’t have an Ubuntu system, so the Ubuntu debs are untested.  The Debian packages work on my wheezy and testing systems.  In spite of the directory name, I severely doubt they’ll work properly on Debian 6.0: the openssl on that platform is too old, so I had to construct a special one just to get it to build.

If you want to keep them up to date, add this line to /etc/apt/sources

deb http://download.opensuse.org/repositories/home:/jejb1:/UEFI/Debian_6.0/ ./

And also add the repository key with apt-key add

A friend who’s a long-time iPhone user just asked for my advice about whether to get a Samsung Galaxy S3, a Samsung Galaxy Note 2, or a iPhone 5.

Advantages for Android

I think that liberty should be the first consideration, I’ve previously written about how Android phones won’t necessarily give you as much freedom as you desire if you buy on the basis of price and features [1]. But even the least free Android options are way better than the iOS (iPhone and iPad) environment. This isn’t necessarily a big deal for my friend, like most of the population he usually just wants things to work – being able to hack them isn’t such an issue. However unlike most of the population he does make a reasonable portion of his income from software development and it could be that he will have a contract for developing an app on a mobile device – in which case the freedom to tinker on Android will help him. He could use an iPhone for his personal use and develop on an Android platform for his clients, but generally it’s more efficient if your personal use of technology is similar to that of your clients. The Nexus devices are very good for liberty and they also have nice hardware at a low price, I’ve just got a Nexus 4 for my wife and it’s very nice.

The next issue is that of hardware standards, I’ve previously written about the potential for developing a standard form factor for Android phones [2] although this doesn’t seem likely to be implemented in the near future. The wide range of Android hardware means that the range of cases etc on the market is rather small. But the advantage of the wide range is that with an Android phone you can have a device that’s bigger, smaller, cheaper, cuter, or faster than an iPhone. There are Android devices which have a higher resolution, more RAM, more storage (if you include SD storage), or has other benefits over an iPhone. For whatever reasonable range of specs appeal to you you can probably find a device to match. I’ve previously written about the way the ideal size for devices depends on your hand size and your preferred manner of gripping the device [3], so the lack of size range in Apple devices is not just a limitation on personal choice but also a failure to properly support people with different size hands. Depending on the preferred manner of gripping a phone the iPhone 5 is either too big for an average woman or too small for a tall man.

The Google Play store apparently has more applications than the iPhone/iPad App Market. This difference can be expected to increase now that the Samsung Galaxy S3 is outselling the iPhone 5. Comparing the number of unit sales of the iPhone vs Android phones is no longer interesting, comparing Samsung to Apple is the interesting thing.

Advantages for the iPhone

By all accounts it’s quite an easy process to backup and restore all iPhone settings. You can expect that after losing an iPhone you can just connect the new one to your PC and have it work in exactly the same way after all the data is transferred. Trying to do such things on Android is merely difficult if you have root access to your phone and the source and destination phones are of exactly the same make and model. But if you have different versions of the phone or if you don’t have root access then it may be impossible. I welcome comments from anyone who knows of good solutions to this problem.

The iPhone achieved a reasonable share of the smart-phone market before Android really started going well so there are a lot of people who are used to the iPhone. Simply by being unfamiliar Android will be a more difficult option for people who have used the iPhone – such as my friend. But it is possible to learn other systems. Generally I think that this may be a big issue for people who use Macs for all their other computing. But if the only Apple product you use is an iPhone then switching to Android shouldn’t be a big deal.

Update:

One feature of the iPhone that is very important to my friend is the ability to add arbitrary tags in the contacts. In addition to name, address, phone number, etc he wants to add arbitrary notes related to his business. While he could put that sort of thing into the “Notes” field in Android he would rather have several fields with his own names. Android 4.1.x definitely doesn’t have this and I can’t test Android 4.2.x at this time. Is there any way of doing such things on Android?

Conclusion

It seems to me that Android devices are better in every way apart from backup, restore, and general management. If I was about to buy 100 phones then I’d probably consider the iPhone (not necessarily buy but definitely consider). But for a single user I definitely recommend Android devices.

The Android devices which seem good at the moment are the Galaxy S3 (which I’m using now), the Nexus 4 (which is really good apart from being unable to change the battery or add more storage), and the Galaxy Note 2 (which is about the biggest phone available).

One of the things that my friend wants to do is to use a phone instead of a tablet or laptop. I think that the Galaxy Note 2 is the only option for him.

• [1] http://etbe.coker.com.au/2012/05/06/liberty-mobile-phones/
• [2] http://etbe.coker.com.au/2012/01/04/standardising-android/
• [3] http://etbe.coker.com.au/2013/02/11/phone-tablet-sizes/

In the last week I revised as much Python 3 as I could, and I tried to teach myself the basics of JavaScript. And let's not forget Mallard!

The (Not So Evil) Plan:

• To give a closure to that Python tutorial! If it is possible to give a closure to something... ;-)
• To help as far I can, particularly with the other developer tutorials (thus particularly with the JavaScript tutorial).
• To tackle Izidor Matušov (again, as far as I can: I remember that he is quite tall) and start working on Getting Things GNOME, finally.
• To have fun! Well, that's always part of the plan...

Will I be able to follow my plan? In everything? At least in part? Will I freeze in the -11°C I read about on BBC Weather?

More news to follow...

DRI3000 — Even Better Direct Rendering

This all started with the presentation that Eric Anholt and I did at the 2012 X developers conference, and subsequently wrote about in my DRI-Next posting. That discussion sketched out the goals of changing the existing DRI2-based direct rendering infrastructure.

Last month, I gave a more detailed presentation at Linux.conf.au 2013 (the best free software conference in the world). That presentation was recorded, so you can watch it online. Or, you can read Nathan Willis’ summary at lwn.net. That presentation contained a lot more details about the specific techniques that will be used to implement the new system, in particular it included some initial indications of what kind of performance benefits the overall system might be able to produce.

I sat down today and wrote down an initial protocol definition for two new extensions (because two extensions are always better than one). Together, these are designed to provide complete support for direct rendering APIs like OpenGL and offer a better alternative to DRI2.

The DRI3 extension

Dave Airlie and Eric Anholt refused to let me call either actual extension DRI3000, so the new direct rendering extension is called DRI3. It uses POSIX file descriptor passing to share kernel objects between the X server and the application. DRI3 is a very small extension in three requests:

1. Open. Returns a file descriptor for a direct rendering device along with the name of the driver for a particular API (OpenGL, Video, etc).

2. PixmapFromBuffer. Takes a kernel buffer object (Linux uses DMA-BUF) and creates a pixmap that references it. Any place a Pixmap can be used in the X protocol, you can now talk about a DMA-BUF object. This allows an application to do direct rendering, and then pass a reference to those results directly to the X server.

3. BufferFromPixmap. This takes an existing pixmap and returns a file descriptor for the underlying kernel buffer object. This is needed for the GL Texture from Pixmap extension.

For OpenGL, the plan is to create all of the buffer objects on the client side, then pass the back buffer to the X server for display on the screen. By creating pixmaps, we avoid needing new object types in the X server and can use existing X apis that take pixmaps for these objects.

The Swap extension

Once you’ve got direct rendered content in a Pixmap, you’ll want to display it on the screen. You could simply use CopyArea from the pixmap to a window, but that isn’t synchronzied to the vertical retrace signal. And, the semantics of the CopyArea operation precludes us from swapping the underlying buffers around, making it more expensive than strictly necessary.

The Swap extension fills those needs. Because the DRI3 extension provides an X pixmap reference to the direct rendered content, the Swap extension doesn’t need any new object types for its operation. Instead, it talks strictly about core X objects, using X pixmaps as the source of the new data and X drawables as the destination.

The core of the Swap extension is one request — SwapRegion. This request moves pixels from a pixmap to a drawable. It uses an X fixes Region object to specify the area of the destination being painted, and an offset within the source pixmap to align the two areas.

A bunch of data are included in the reply from the SwapRegion request. First, you get a 64-bit sequence number identifying the swap itself. Then, you get a suggested geometry for the next source pixmap. Using the suggested geometry may result in performance improvements from the techniques described in the LCA talk above.

The last bit of data included in the SwapRegion reply is a list of pixmaps which were used as source operands to earlier SwapRegion requests to the same drawable. Each pixmap is listed along with the 64-bit sequence number associated with an earlier SwapRegion operation which resulted in the contents which the pixmap now contains. Ok, so that sounds really confusing. Some examples are probably necessary.

• If the SwapRegion operation was implemented by copying data out of the source pixmap into the destination drawable, then the idle swap count will be equal to the swap count from this SwapRegion operation.

• If the SwapRegion operation was implemented by swapping the destination contents with the source contents, then the idle swap count will be equal to the previous swap count on the destination drawable.

I’m hoping you’ll be able to tell that in both cases, the idle swap count tries to name the swap sequence at which time the destination drawable contained the contents currently in the pixmap.

Note that even if the SwapRegion is implemented as a Copy operation, the provided source pixmap may not be included in the idle list as the copy may be delayed to meet the synchronization requirements specfied by the client.

Finally, if you want to throttle rendering based upon when frames appear on the screen, Swap offers an event that can be delivered to the drawable after the operation actually takes place.

Because the Swap extension needs to supply all of the OpenGL SwapBuffers semantics (including a multiplicity of OpenGL extensions related to that), I’ve stolen a handful of DRI2 requests to provide the necessary bits for that:

1. SwapGetMSC
2. SwapWaitMSC
3. SwapWaitSBC

These work just like the DRI2 requests of the same names.

Current State of the Extensions

Both of these extensions have an initial protocol specification written down and stored in git:

1. DRI3 protocol

2. Swap protocol

Camp JS was a weekend conference held at Koonjewarre down at Springbrook.

The area the accommodation in was lovely, if the weather had been better I would have definitely done a little bush walking. The accommodation itself was pretty basic, but along with your own sleeping bag and pillow was reasonable for a weekend, but would start getting a bit painful for longer stays. The food on offer was very good for the most part, except for one meal, but that turned out to be a backup meal.

The conference did have planned workshops all weekend, and some of these were very good. However the main point of the conference was not attending the workshops, it was interacting with other developers, both on a technical and social level.

On the upside were some impressive international guests and sponsors. On the downside for me were the smokers who always smoked just outside whatever building I was in. On the interesting side was the open bar, free beer and wine for all, at any time, without any problems whatsoever; something to think about given lca’s recent moves to restrict alcohol access.

So I didn’t end up posting that much about LCA…

The main conference was fantastic:

• Clojure is awesome for concurrency
• Unix party tricks are extremely fun, and rather scary, and now I just want to disappear off teh interwebz
• Schwern taught me lots about Git – I now have a Github account and I feel a lot more confident with it!
• Repent, for the end of the Unix epoch is nigh!
• Pia Waugh can talk very fast, particularly given some sleep deprivation
• bunnie’s keynote provided some very interesting insights into the world of consumer manufacturing, particularly pricing
• TBL can speak even faster than Pia – this was actually problematic…
• Asheesh Laroia and OpenHatch are pretty awesome
• Paul Fenwick can talk very, very fast given a 90 second timeslot…

Anyway, it was awesome. I highly recommend it, and I’m already planning to make my way to Perth for 2014.