Introspection Porting Guide — PyGObject

Skip to main content
`Ctrl`+`K`

# Introspection Porting Guide

Attention

The pygtkcompat package has been deprecated since PyGObject 3.48. The functionality has been removed since PyGObject 3.52.

## Beginning

Starting with PyGObject 3.1.92 there’s a module called pygtkcompat which will make it easier to port an application from PyGTK.

There are two different approaches to porting your application from PyGTK to gobject-introspection:

1. Using a PyGTK compatibility layer (pygtkcompat)

2. Using a shell script which converts the callsites (`pygi-convert.sh`)

There are advantages and disadvantages with both:

1. Using pygtkcompat makes it possible to focus on porting to Gtk 3.x while not changing anything else, users and developers can keep on using the software as before.

2. Using pygtkcompat allows you to support both Gtk 2.x and Gtk 3.x with the same code base

3. Using `pygi-convert.sh` you can avoid using any layers and use the supported API directly

### Using pygtkcompat

The pygtkcompat module provides a PyGTK compatible API on top of gobject-introspection which makes it possible to run your application on top of both PyGTK and gobject-introspection at the same time.

Before using it you should port your application to using latest API, available 2.24, for instance:

    widget.window should be widget.get_window()
    container.child should be container.get_child()
    widget.flags() & gtk.REALIZED should be container.get_realized()

and so on.

Once an application has been updated to the latest PyGTK API, you can then import pygtkcompat and enable the parts you need. For instance, to enable PyGTK compatible API on top of the Gtk 3.0 typelib, use the following:

    from gi import pygtkcompat

    pygtkcompat.enable()
    pygtkcompat.enable_gtk(version='3.0')

That’s it, if you’re lucky enough and you’re not using any strange/weird apis you should be able to run your application. If you want an app to be compatible with both PyGTK and PyGI, you can use the following technique:

    try:
        from gi import pygtkcompat
    except ImportError:
        pygtkcompat = None

    if pygtkcompat is not None:
        pygtkcompat.enable()
        pygtkcompat.enable_gtk(version='3.0')

    import gtk

Note

Porting the application from 2 to 3 is best covered in https://docs.gtk.org/gtk3/migrating-2to3.html. That usually includes things such as change expose-event to draw and do drawing with cairo.

## How does PyGI work?

Initial GI support was added to pygobject in version 2.19.0 (August 2009), but the entire GI/pygobject/annotations stack really only stabilized in version 2.28, so that in practice you will need at least this version and the corresponding latest upstream releases of GTK and other libraries you want to use.

pygobject provides a `gi.repository` module namespace which generates virtual Python modules from installed typelibs on the fly. For example, if you have the GIR for GTK 3 installed, you can do:

    $ python -c 'from gi.repository import Gtk; print Gtk'
    <gi.module.DynamicModule 'Gtk' from '/usr/lib/girepository-1.0/Gtk-3.0.typelib'>

and use it just like any other Python module.

Absolutely unexpected first example:

    $ python -c 'from gi.repository import Gtk; Gtk.MessageDialog(None, 0, Gtk.MessageType.INFO, Gtk.ButtonsType.CLOSE, "Hello World").run()'

Let’s look at the corresponding C declaration:

    GtkWidget* gtk_message_dialog_new (GtkWindow *parent, GtkDialogFlags flags, GtkMessageType type, GtkButtonsType buttons, const gchar *message_format, ...);

and the C call:

    GtkMessageDialog* msg = gtk_message_dialog_new (NULL, 0, GTK_MESSAGE_INFO, GTK_BUTTONS_CLOSE, "Hello World");
    msg.run()

So what do we see here?

1. The C API by and large remains valid in Python (and other languages using the GI bindings), in particular the structure, order, and data types of arguments. There are a few exceptions which are mostly due to the different way Python works, and in some cases to make it easier to write code in Python; see below for details. But this means that you can (and should) use the normal API documentation for the C API of the library. Devhelp is your friend!

2. As Python is a proper object oriented language, pygobject (and in fact the GI typelib already) expose a GObject API as proper classes, objects, methods, and attributes. I. e. in Python you write

       button = Gtk.Button()
       button.set_label("foo")

   instead of the C gobject syntax

       GtkWidget* button = gtk_button_new(...);
       gtk_button_set_label(button, "foo");

   The class names in the typelib (and thus in Python) are derived from the actual class names stated in the C library (like `GtkButton`), except that the common namespace prefix (`Gtk` here) is stripped, as it becomes the name of the module.

3. Global constants would be a heavy namespace clutter in Python, and thus pygobject exposes them in a namespaced fashion as well. I. e. if the `MessageDialog` constructor expects a constant of type `GtkMessageType`, then by above namespace split this becomes a Python class `Gtk.MessageType` with the individual constants as attributes, e. g. `Gtk.MessageType.INFO`.

4. Data types are converted in a rather obvious fashion. E. g. when the C API expects an `int*` array pointer, you can supply a normal Python array `[0, 1, 2]`. A Python string `"foo"` will match a `gchar*`, Pythons `None` matches `NULL`, etc. So the GObject API actually translates quite naturally into a real OO language like Python, and after some time of getting used to above transformation rules, you should have no trouble translating the C API documentation into their Python equivalents. When in doubt, you can always look for the precise names, data types, etc. in the .gir instead, which shows the API broken by class, method, enum, etc, with the exact names and namespaces as they are exposed in Python.

As I mentioned above, this is in no way restricted to GTK, GNOME, or UI. For example, if you handle any kind of hardware and hotplugging, you almost certainly want to query udev, which provides a nice glib integration (with signals) through the gudev library. This example lists all block devices (i. e. hard drives, USB sticks, etc.):

    >>> from gi.repository import GUdev
    >>> c = GUdev.Client()
    >>> for dev in c.query_by_subsystem("block"):
    ...     print dev.get_device_file()
    ...
    /dev/sda
    /dev/sda1
    /dev/sda2
    [...]

See the GUDevClient documentation for the corresponding C API. GI is not even restricted to GObject, you can annotate any non-OO function based API with it. E. g. there is already a `/usr/share/gir-1.0/xlib-2.0.gir` (although it’s horribly incomplete). These will behave as normal functions in Python (or other languages) as well.

## Differences to the C API

The structure of method arguments is by and large the same in C and in GI/Python. There are some notable exceptions which you must be aware of:

### Constructors

The biggest one is constructors. There is actually two ways of calling one:

- Use the real constructor implementation from the library. Unlike in normal Python you need to explicitly specify the constructor name:

      Gtk.Button.new()
      Gtk.Button.new_with_label("foo")

- Use the standard GObject constructor and pass in the initial property values as named arguments:

      Gtk.Button(label="foo", use_underline=True)

The second is actually the recommended one, as it makes the meaning of the arguments more explicit, and also underlines the GObject best practice that a constructor should do nothing more than to initialize properties. But otherwise it’s pretty much a matter of taste which one you use.

### Passing arrays

Unlike C, higher level languages know how long an array is, while in the C API you need to specify that explicitly, either by terminating them with `NULL` or explicitly giving the length of the array in a separate argument. Which one is used is already specified in the annotations and thus in the typelib, so Python can automatically provide the right format without the developer needing to append an extra `None` or a separate len(my_array) argument.

For example, in C you have

    gtk_icon_theme_set_search_path (GtkIconTheme *icon_theme, const gchar *path[], gint n_elements)

In Python you can just call this as

    my_icon_theme.set_search_path(['/foo', '/bar'])

and don’t need to worry about the array size.

### Output arguments

C functions can’t return more than one argument, so they often use pointers which the function then fills out. Conversely, Python doesn’t know about pointers, but can easily return more than one value as a tuple. The annotations already describe which arguments are “out” arguments, so in Python they become part of the return tuple: first one is the “real” return value, and then all out arguments in the same order as they appear in the declaration. For example:

    GdkWindow* gdk_window_get_pointer (GdkWindow *window, gint *x, gint *y, GdkModifierType *mask)

In Python you would call this like

    x, y, mask = mywindow.get_pointer()

The Python help system shows the correct input, output and return argument expectations:

    >>> from gi.repository import Gdk
    >>> help(Gdk.Window.get_pointer)
    Help on function get_pointer:

    get_pointer(*args, **kwargs)
        get_pointer(self) -> x:int, y:int, mask:Gdk.ModifierType

### GDestroyNotify

Some GLib/GTK functions take a callback method and an extra `user_data` argument that is passed to the callback. In C they often also take a `GDestroyNotify` function which is run once all callbacks are done, in order to free the memory of user_data. As Python has automatic memory management, pygobject will take care of all this by itself, so you simply don’t specify the GDestroyNotify argument. For example:

    void gtk_enumerate_printers (GtkPrinterFunc func, gpointer user_data, GDestroyNotify destroy, gboolean wait)

In Python you call this as

    Gtk.enumerate_printers(my_callback, my_user_data, True)

### Non-introspectable functions/methods

When you work with PyGI for a longer time, you’ll inevitably stumble over a method that simply doesn’t exist in the bindings. These usually are marked with `introspectable="0"` in the GIR.

In the best case this is because there are some missing annotations in the library which don’t have a safe default, so GI disables these to prevent crashes. They usually come along with a corresponding warning message from g-ir-scanner, and it’s usually quite easy to fix these.

Another common case are functions which take a variable number of arguments, such as `gtk_cell_area_add_with_properties()`. Varargs cannot be handled safely by libgirepository. In these cases there are often alternatives available (such as `gtk_cell_area_cell_set_property()`). For other cases libraries now often have a `..._v()` counterpart which takes a list instead of variable arguments.

## Threads

For using threads with PyGObject, please see: Threads/Concurrency with Python and the GNOME Platform

## Overrides

A specialty of pygobject is the possibility of replacing functions, methods, or classes of the introspected library with custom code, called “overrides”. As the goal is to stay very close to the original API, they should be used and written sparsely, though. One major use case is to provide replacements for unintrospectable methods. For example, `Gtk.Menu.popup()` is not introspectable in GTK, but the GTK override implements this method in terms of `Gtk.Menu.popup_for_device()`, so in this case the override actually helps to get closer to the original API again. Another important case is automatic data type conversion, most prominently to allow passing unicode objects to methods which expect an UTF-8 encoded `gchar*`. This also actually helps to prevent workarounds in application code and maintain a clean API. Thirdly, overrides are useful if they help to massively ease development. For example it is quite laborious to do GDBus calls or GVariant constructions with the native Gio/GLib API. pygobject offers convenience overrides to make these a lot simpler and more Pythonic, but of course without actually breaking the original API.

Overrides should be quite easy to understand. In general you should not even be required to know about it, as most of them really just fix stuff to work as expected. :-)

## Porting from PyGTK 2 to PyGI GTK 3

Note that this is really two migrations in one step, but is recommended as GTK2 still has a lot of breakage with PyGI. It is recommended to port applications to PyGI/GTK+ 3 first, then port to Python 3 as an additional step if Python 3 support is desired. Otherwise you can end up with a conflated and hard to track set of changes. See the GTK2 → GTK3 migration documentation.

If we compare the PyGTK vs. PyGI code for a “Hello” message box, we see that it’s actually very similar in structure:

    $ python -c 'import gtk; gtk.MessageDialog(None, 0, gtk.MESSAGE_INFO, gtk.BUTTONS_CLOSE, "Hello World").run()'

vs.

    $ python -c 'from gi.repository import Gtk; Gtk.MessageDialog(None, 0, Gtk.MessageType.INFO, Gtk.ButtonsType.CLOSE, "Hello World").run()'

So PyGTK also does the representation of the C functions as proper classes and methods, thus if you port from PyGTK to PyGI, the structure by and large remains the same.

### Step 1: The Great Renaming

The biggest part in terms of volume of code changed is basically just a renaming exercise. E. g. `gtk.*` now becomes `Gtk.*`, and `gtk.MESSAGE_INFO` becomes `Gtk.MessageType.INFO`. Likewise, the imports need to be updated: `import gtk` becomes `from gi.repository import Gtk`.

Fortunately this is is a mechanical task which can be automated. The pygobject git tree has a script pygi-convert.sh which is a long list of perl -pe ‘s/old/new/’ string replacements.

It’s really blunt, but surprisingly effective, and for small applications chances are that it will already produce something which actually runs. Note that this script is in no way finished, and should be considered a collaborative effort amongst porters. So if you have something which should be added there, please don’t hesitate to open a bug or ping on IRC (#python on irc.gnome.org)). We will be happy to improve the script.

When you just run `pygi-convert.sh` in your project tree, it will work on all `*.py` files. If you have other Python code there which is named differently (such as `bin/myprogram`), you should run it once more with all these file names as argument.

- Make sure you don’t keep using the static bindings for a library that we are using through introspection. That would cause wrappers from both bindings be mixed and compatibility issues would arise.

### Step 2: Wash, rinse, repeat

Once the mechanical renamings are out of the way, the tedious and laborious part starts. As Python does not have a concept of “compile-time check” and can’t even check that called methods exist or that you pass the right number of parameters, you now have to enter a loop of “start your program”, “click around until it breaks”, “fix it”, “goto 1”.

The necessary changes here are really hard to generalize, as they highly depend on what your program actually does, and this will also involve the GTK 2 → 3 parts. One thing that comes up a lot are `pack_start()`/`pack_end()` calls. In PyGTK they have default values for `expand`, `start`, and `padding` attributes, but as GTK does not have them, you won’t have them in PyGI either (see bgo#558620 - Add default values).

Warning

Note that you can’t do a migration halfway: If you try to import both `gtk` and `gi.repository.Gtk`, you’ll get nothing but program hangs and crashes, as you are trying to work with the same library in two different ways. You can mix static and GI bindings of different libraries though, such as `dbus-python` and `gi.repository.Gtk`.

If your application uses plugins, you can use libpeas. It is a GObject plugins library that support C, Python and Javascript languages though introspection.

### Step 3: Packaging changes

After you have your code running with PyGI and committed it to your branch and released it, you need to update the dependencies of your distro package for PyGI. You should grep your code for “gi.repository” and collect a list of all imported typelibs, and then translate them into the appropriate package name. For example, if you import “Gtk, Notify, Gudev” you need to add dependencies to the packages which ship them:

- Debian/Ubuntu ship them in separate packages named `gir<GI_ABI_version>-<libraryname>-<library_ABI_version>`, so in this example `gir1.2-gtk-3.0`, `gir1.2-notify-0.7`, and `gir1.2-gudev-1.0`. You can find out with e. g. `dpkg -S /usr/lib/girepository-1.0/Gtk-3.0.typelib`.

- Fedora ships the typelibs together with the shared libraries, so in this example `gtk3`, `libgudev1`, `libnotify`. You can find out with e. g. `rpm -qf /usr/lib/girepository-1.0/Gtk-3.0.typelib`.

At the same time you should drop the old static bindings, like python-gtk2, python-notify, etc.

Finally you should also bump the version of the pygobject dependency to (>= 2.28) to ensure that you run with a reasonably bug free PyGI.

## Examples

- pygobject’s git tree has a very comprehensive gtk-demo showing off pretty much all available GTK widgets in PyGI

- Examples of previously done pygtk → pyGI ports:

  - Apport: http://bazaar.launchpad.net/~apport-hackers/apport/trunk/revision/1801

  - Jockey: http://bazaar.launchpad.net/~jockey-hackers/jockey/trunk/revision/679

  - system-config-printer: https://git.fedorahosted.org/cgit/system-config-printer.git/log/?h=pygi

  - gtimelog: http://bazaar.launchpad.net/~gtimelog-dev/gtimelog/trunk/revision/181 (this is interesting because it makes the code work with both PyGTK and PyGI, whichever is available)

## Comments

- What versions of the python packages do we need, and how do we get the for different distributions (gentoo, deb-based, rpm-based…?)

- Can we just fallback to import gtk, gdk…. if we detect old enough python packages, or do we need to perform other workarounds?

One of the biggest challenges in porting is that all of the constants have changed names. The above `pygi-convert.sh` script gives a good idea about what the new names are, but if you have any problems, refer to the relevant .gir file directly. For example, if your old code says “gtk.TREE_VIEW_COLUMN_AUTOSIZE” and you’re not sure what the new code should be, search in /usr/share/gir-1.0/Gtk-2.0.gir for “COLUMN_AUTOSIZE”, and you’ll find some code that looks like this:

    <enumeration name="TreeViewColumnSizing"
                 glib:type-name="GtkTreeViewColumnSizing"
                 glib:get-type="gtk_tree_view_column_sizing_get_type"
                 c:type="GtkTreeViewColumnSizing">
      <member name="grow_only"
              value="0"
              c:identifier="GTK_TREE_VIEW_COLUMN_GROW_ONLY"
              glib:nick="grow-only"/>
      <member name="autosize"
              value="1"
              c:identifier="GTK_TREE_VIEW_COLUMN_AUTOSIZE"
              glib:nick="autosize"/>
      <member name="fixed"
              value="2"
              c:identifier="GTK_TREE_VIEW_COLUMN_FIXED"
              glib:nick="fixed"/>
    </enumeration>

Looking at that, you can determine the new constant name. It starts with “Gtk”, then a period, then the enumeration name (“TreeViewColumnSizing”), then another period, then the member name in ALL CAPS (“AUTOSIZE”). So, this tells you to change “gtk.TREE_VIEW_COLUMN_AUTOSIZE” into “Gtk.TreeViewColumnSizing.AUTOSIZE”

## Problems

- Listening to signals: if you are listening a signal like “size-allocate” and when you print the allocation you get as the type GdkRectangle instead of CairoRectangleInt, the way to fix it is by overriding the signal instead of listening to it.