EntriesAbout

Automaking Prolog, Part Two

The Gory Details

Last time, I talked about the process that led me to building my “automake” tool for Prolog. Today, I wanted to go into the details of how I put it together.

Note that the code below is as of 23 February 2022 – things may have changed since then. You can find the full code here

prolog/automake.pl

Let’s start with the simple part: automake.pl. This is the little bit of Prolog code that interfaces with the “foreign” library to be notified of file changes and call the built-in make predicate.

It starts in the normal way a Prolog file does; defining a module and importing the libraries we’ll use.

:- module(automake, [automake/0,
                     noautomake/0]).

:- use_module(library(debug), [debug/3]).
:- use_module(library(make), [make/0]).
:- use_module(library(time), [alarm/3, remove_alarm/1]).

All three of the libraries we pull in can also be autoloaded, but I like to be explicit with what’s being used.1

Next, we get into the interesting stuff.

:- use_foreign_library(foreign(watchdir4pl)).

We load the foreign library, which contains the platform-specific code to watch for file changes. More on that to come.

:- dynamic watches/2.

We define a dynamic predicate for state that the module will keep.

Each watches/2 fact associates a file being watched with a handle to the native-coder watcher. It turns out we don’t actually need that…but more on that later.

automake :-
    noautomake,
    dir_monitor_init(Monitor),
    add_new_source_files(Monitor),
    thread_create(handle_file_changed(Monitor), _, [alias(automake)]).

noautomake :-
    catch(thread_send_message(automake, done, [timeout(0)]),
          error(existence_error(message_queue, automake), _),
          fail), !,
    thread_join(automake, _).
noautomake.

Next, we define the entry-point predicates that we export from the module. automake starts the watcher, noautomake stops it.

We do this by creating a thread and giving it an alias, which we can use to stop, by sending a message. As we’ll see in a moment, the “watcher” thread periodically checks for received messages; upon receiving one, it will stop. I use this pattern a lot in Prolog when making tools with multiple threads, but I think I first came across it in Clojurescript, using core.async.

Usually, I do this by explicitly creating a message queue, passing it to the predicate the new thread is running, and keeping a reference to the queue in a dynamic predicate as well. In this case though, since I was already giving the thread an alias (to get nicer error and debug messages), I realized I could simplify things by just using the thread’s “name” as the queue (which then sends/receives to/from the thread’s built-in queue). Here’s what changed, if you’re curious:

-:- dynamic watcher_queue/1.
-
 automake :-
     noautomake,
     dir_monitor_init(Monitor),
     add_new_source_files(Monitor),
-    message_queue_create(DoneQ),
-    assertz(watcher_queue(DoneQ)),
-    thread_create(handle_file_changed(Monitor, DoneQ), _, [alias(automake)]).
+    thread_create(handle_file_changed(Monitor), _, [alias(automake)]).

 noautomake :-
-    watcher_queue(Q), !,
-    thread_send_message(Q, done),
-    thread_join(automake, _),
-    retractall(watcher_queue(Q)).
+    catch(thread_send_message(automake, done, [timeout(0)]),
+          error(existence_error(message_queue, automake), _),
+          fail), !,
+    thread_join(automake, _).
 noautomake.

Let’s break down exactly what the automake predicate is doing.

First, it calls noautomake, to stop an existing thread if there is one. I do this to prevent multiple conflicting watching threads.

Then, we invoke one of the predicates from our foreign library, dir_monitor_init/1 to create a new monitor object. This will be the thing we use to interact with the platform-appropriate library.

Next, we call add_new_source_files(Monitor). That predicate looks like this:

add_new_source_files(Monitor) :-
    forall(source_file(File),
           ( maybe_add_watch(Monitor, File) )).

This uses the built-in source_file non-deterministic predicate to iterate over all the source files that are currently in use and start watching them if we weren’t already.

We’ll invoke this predicate multiple times, so if new files get loaded, we pick them up.

maybe_add_watch/2 uses the dynamic predicate we defined up top to see if we’re already monitoring the file and, if we aren’t and it’s a real file, uses another of our foreign predicates to add the file to the monitor.

maybe_add_watch(Monitor, Path) :-
    watches(Path, _)
    -> true
    ; ( debug(automake, "Adding watch for ~w", [Path]),
        exists_file(Path)
      -> ( dir_monitor_add(Monitor, Path, W),
           assertz(watches(Path, W)) )
      ; true ).

Backing up to automake, the next step is calling thread_create/3 and starting it with the predicate handle_file_changed/1. This predicate, running in a background thread, will be the thing that checks for changes from the monitor.

handle_file_changed(Monitor) :-
    dir_monitor_read(Monitor, Event, 0.5), !,
    catch(handle_event(Event, Monitor),
         Err,
         debug(automake, "Error handling event ~w: ~w", [Event, Err])),
    handle_file_changed(Monitor).
handle_file_changed(Monitor) :-
    thread_get_message(automake, _, [timeout(0)]), !,
    dir_monitor_stop(Monitor).
handle_file_changed(Monitor) :-
    add_new_source_files(Monitor),
    handle_file_changed(Monitor).

The predicate is essentially a polling loop, implemented as a tail-recursive function.

The first “case” checks for changes via dir_monitor_read/3, yet another of our foreign predicates. It takes the monitor to check, an Event variable which will be filled with the event that happened, and a timeout. Here, we tell it to time out after half a second; if that time elapses without an incoming event, the call fails and we go on to other cases.

If an event does arrive, then we call the handle_event/2 predicate, which (as we’ll see) mostly just causes make. It’s wrapped in a catch/3 so if something goes wrong there it doesn’t take down the thread.

The second cases checks the message queue for an incoming message. If it receives one, it stops the monitor and completes without a recursive call, thereby finishing the thread.

Finally, the third case adds new source files, if any, and loops again.

We’re almost finished with the Prolog side. The last thing to see is the handle_event/2 predicate.

This used to be more complicated – based on other code I had using inotify – but in the process of building this, I realized that I actually only needed to care about modifications and what the modification was doesn’t matter at all.

handle_event/2 doesn’t call make directly because we want to “debounce” the changes slightly, so if a bunch of files are changed in quick succession (e.g. after checking out a branch so a bunch of files change), we don’t try to run make a bunch of times at once.

:- dynamic make_alarm/1.

handle_event(watchdir(modified, _, _), _) :-
    maybe_cancel_make,
    alarm(0.2, do_make, Alarm),
    assertz(make_alarm(Alarm)).
handle_event(Event, _) :-
    debug(automake(debug), "Unknown event ~w", [Event]).

This uses library(alarm) to delay 0.2 seconds before calling do_make/0; it stores the “alarm” in a dynamic predicate, so maybe_cancel_make/0 can stop the alarm if it’s called again before firing:

maybe_cancel_make :-
    make_alarm(Alarm), !,
    debug(automake(debug), "Cancelling alarm ~w", [Alarm]),
    remove_alarm(Alarm),
    retractall(make_alarm(Alarm)).
maybe_cancel_make.

Now, finally, we get to do_make/0:

do_make :-
    retractall(make_alarm(_)),
    make.

Done!

But…how do we actually get those file change notifications? To find that, we need to dive into some platform-specific C.

c/watchdir4pl.c

Much of this code is based on pack(inotify).

We start, as usual with C, by pulling in our header files.

#include <SWI-Stream.h>
#include <SWI-Prolog.h>

#include <assert.h>
#include <errno.h>
#ifndef __WINDOWS__
#include <poll.h>
#endif
#include <stdlib.h>
#include <string.h>
#ifdef __linux__
#include <sys/inotify.h>
#elif defined(__APPLE__)
#include <fcntl.h>
#include <sys/types.h>
#include <sys/event.h>
#include <sys/time.h>
#elif defined(__WINDOWS__)
#include <windows.h>
#endif
#include <unistd.h>

We have a couple #ifdef’s here, to pull in the different libraries we need on different platforms.

Next, we create the struct that will hold the “monitor” state.

#define WATCH_BUF_SIZE 4096

typedef struct watchref {
  atom_t symbol;
#ifndef __WINDOWS__
  int fd;
#endif
  size_t len;
#ifdef __linux__
  struct inotify_event *ev;
  char buf[WATCH_BUF_SIZE];
#elif defined(__APPLE__)
  size_t nevents;
  size_t max_events;
  struct kevent *watching;
#elif defined(__WINDOWS__)
  size_t nHandles;
  size_t maxHandles;
  HANDLE *dwChangeHandles;
#endif
} watchref;

Again, we have some platform-specific differences

On Linux, we store an inotify_event and buffer for the data on Linux. On macOS and Windows, we have a dynamic list of handles for each file/directory that we’re watching.

The next bit is defining basic predicates to make a blob object that can be passed around and handled properly by the Prolog foreign library machinery.

static int write_watchref(IOSTREAM *s, atom_t eref, int flags) {
  watchref **refp = PL_blob_data(eref, NULL, NULL);
  watchref *ref = *refp;
  (void)flags;

  Sfprintf(s, "<watchdir>(%p)", ref);
  return TRUE;
}

static int release_watchref(atom_t aref) {
  watchref **refp = PL_blob_data(aref, NULL, NULL);
  watchref *ref = *refp;

#ifdef __linux__
  if ( ref->fd >= 0 ) {
    close(ref->fd);
  }
#elif defined(__APPLE__)
  for (int i = 0; i < ref->nevents; i++) {
    close(ref->watching[i].ident);
  }
  PL_free(ref->watching);
#elif defined(__WINDOWS__)
  for (int i = 0; i < ref->nHandles; i++) {
    FindCloseChangeNotification(ref->dwChangeHandles[i]);
  }
  PL_free(ref->dwChangeHandles);
#endif

  PL_free(ref);

  return TRUE;
}

static void acquire_watchref(atom_t aref) {
  watchref **refp = PL_blob_data(aref, NULL, NULL);
  watchref *ref = *refp;

  ref->symbol = aref;
}

static int save_watchdir(atom_t aref, IOSTREAM *fd) {
  watchref **refp = PL_blob_data(aref, NULL, NULL);
  watchref *ref = *refp;
  (void)fd;

  return PL_warning("Cannot save reference to <watchdir>(%p)", ref);
}

static atom_t load_watchdir(IOSTREAM *fd) {
  (void)fd;

  return PL_new_atom("<saved-watchdir>");
}

static PL_blob_t watchdir_blob =
{ PL_BLOB_MAGIC,
  PL_BLOB_UNIQUE,
  "watchdir",
  release_watchref,
  NULL,
  write_watchref,
  acquire_watchref,
  save_watchdir,
  load_watchdir
};

Not too much to say about that – just Prology stuff. Only thing of interesting is the release_watchref function, which frees the platform-specific resources.

Next, some more helper-type functions for communicating the watchdir struct to and from Prolog.

static int unify_watchdir(term_t t, watchref *er) {
  if (er->symbol) {
    return PL_unify_atom(t, er->symbol);
  } else {
    return PL_unify_blob(t, &er, sizeof(er), &watchdir_blob);
  }
}

static int get_watchdir(term_t t, watchref **erp, int warn) {
  void *data;
  size_t len;
  PL_blob_t *type;

  if (PL_get_blob(t, &data, &len, &type) && type == &watchdir_blob) {
    watchref **erd = data;
    watchref *er = *erd;
#ifdef __WINDOWS__
    if (1) {
#else
    if (er->fd >= 0) {
#endif
      *erp = er;
      return TRUE;
    } else if (warn) {
      PL_existence_error("watchdir", t);
    }
  }

  if (warn) {
    PL_type_error("watchdir", t);
  }

  return FALSE;
}

Then we define the atoms and functors we’ll be creating & “exporting”:

static atom_t ATOM_created;
static atom_t ATOM_deleted;
static atom_t ATOM_modified;

static functor_t FUNCTOR_error2;
static functor_t FUNCTOR_watchdir_error2;
static functor_t FUNCTOR_watchdir3;

Helper function for throwing an error term that will look like error(watchdir_error(<watchdir>(12345678), "The error message", _)).

static int watchdir_error(watchref *ref) {
  term_t ex, in;

  return ( (ex = PL_new_term_ref()) &&
           (in = PL_new_term_ref()) &&
           unify_watchdir(in, ref) &&
           PL_unify_term(ex, PL_FUNCTOR, FUNCTOR_error2,
                                PL_FUNCTOR, FUNCTOR_watchdir_error2,
                                    PL_TERM, in,
                                    PL_CHARS, strerror(errno),
                                PL_VARIABLE) &&
           PL_raise_exception(ex)
           );
}

Now, we finally get into the platform-specific functions; first, creating the appropriate monitor object:

static foreign_t pl_watchdir_create_dir_monitor(term_t watchdir) {
  int fd = 0;
#ifdef __linux__
  fd = inotify_init1(IN_CLOEXEC);
#elif defined(__APPLE__)
  fd = kqueue();
#endif
  if (fd < 0) {
    return watchdir_error(NULL);
  }

  watchref *ref = PL_malloc(sizeof(*ref));
  if (!ref) {
#ifndef __WINDOWS__
    close(fd);
#endif
    return PL_resource_error("memory");
  }
  memset(ref, 0, sizeof(*ref));
#ifndef __WINDOWS__
  ref->fd = fd;
#endif
#ifdef __APPLE__
  ref->nevents = 0;
  ref->watching = PL_malloc(10 * sizeof(struct kevent));
  if (!ref->watching) {
    close(fd);
    PL_free(ref);
    return FALSE;
  }
  ref->max_events = 10;
#elif defined(__WINDOWS__)
  ref->nHandles = 0;
  ref->dwChangeHandles = PL_malloc(10 * sizeof(HANDLE));
  if (!ref->dwChangeHandles) {
    PL_free(ref);
    return FALSE;
  }
  ref->maxHandles = 10;
#endif
  if ( !unify_watchdir(watchdir, ref) ) {
#ifndef __WINDOWS__
    close(fd);
#endif
    PL_free(ref);
    return FALSE;
  }

  PL_succeed;
}

On Linux, all we need to do is create the inotify file-descriptor with inotify_init1, create the watchref struct, and unify it with the output variable.

Under macOS, we similarly create a kqueue, but then also create a buffer (ref->watching) that we’ll use to store the kevents we’ll be checking.

With the Windows API, there’s no over-arching structure, just the buffer of “HANDLE”s, which will be the change notification things we’ll check.

static foreign_t pl_watchdir_stop(term_t watchdir) {
  watchref *ref;
  if (get_watchdir(watchdir, &ref, TRUE)) {
#ifdef __WINDOWS__
  for (int i = 0; i < ref->nHandles; i++) {
    FindCloseChangeNotification(ref->dwChangeHandles[i]);
  }
#else
    close(ref->fd);
    ref->fd = -1;
#endif
    return TRUE;
  }
  return FALSE;
}

To stop watching, on Windows we call FindCloseChangeNotification on each of the handles we’ve created; on Linux and macOS, it suffices to just close the inotify/kqueue descriptor (at least, I think it does).

Now that we can create a monitor, we get more into the real differences between the various systems, as we add something to watch:

static foreign_t pl_watchdir_add_monitor(term_t watchdir, term_t file_path, term_t watch) {
  watchref *ref;

  if( !get_watchdir(watchdir, &ref, TRUE) ) {
    return FALSE;
  }

  char *file_path_name;
  if (!PL_get_file_name(file_path, &file_path_name, PL_FILE_OSPATH)) {
    return FALSE;
  }

#ifdef __linux__
  int wd = inotify_add_watch(ref->fd, file_path_name,
                             IN_CREATE | IN_DELETE | IN_DELETE_SELF
                             | IN_MOVE | IN_MODIFY | IN_MOVE_SELF | IN_DELETE);
  if (wd == -1) {
    return PL_resource_error("inotify_watch");
  }
  return PL_unify_integer(watch, wd);
#elif defined(__APPLE__)
  if (ref->nevents == ref->max_events) {
    struct kevent *new_watching = PL_realloc(ref->watching,
                                             sizeof(struct kevent) * (ref->max_events + 10));
    if (!new_watching) {
      return PL_resource_error("memory");
    }
    ref->watching = new_watching;
    ref->max_events += 10;
  }
  int wd = open(file_path_name, O_EVTONLY);
  if (wd < 0) {
    return PL_resource_error("kqueue_open");
  }
  int idx = ref->nevents;
  EV_SET(&ref->watching[idx], wd, EVFILT_VNODE, EV_ADD | EV_CLEAR,
         NOTE_WRITE, 0, NULL);
  ref->nevents += 1;
  return PL_unify_integer(watch, idx);
#elif defined(__WINDOWS__)
  if (ref->nHandles == ref->maxHandles) {
    HANDLE* new_change_handles = PL_realloc(ref->dwChangeHandles,
                                            sizeof(HANDLE) * (ref->maxHandles + 10));
    if (!new_change_handles) {
      return PL_resource_error("memory");
    }
    ref->dwChangeHandles = new_change_handles;
    ref->maxHandles += 10;
  }
  int idx = ref->nHandles;
  char dirpath[255];
  char dirname[255];
  _splitpath_s(file_path_name, dirpath, 4, dirname, 255, NULL, 0, NULL, 0);
  dirpath[2] = '\\';
  dirpath[3] = '\0';
  strncat(dirpath, dirname, 255 - 4);
  ref->dwChangeHandles[idx] = FindFirstChangeNotification(dirpath, 0,
                                                          FILE_NOTIFY_CHANGE_LAST_WRITE);
  ref->nHandles += 1;
  return PL_unify_integer(watch, idx);
#endif
}

Again, Linux is the simple case. We just call inotify_add_watch, specifying the inotify file descriptor we created, the path of the file to watch, and the appropriate flags.

For kqueue on macOS, we first have to check if we have enough space to store the new struct, realloc-ing as needed, then open the file (with the O_EVTONLY flag to indicate we’re just opening to get events, not to actually read or write) and create the kevent struct, using the EV_SET macro provided by sys/event.h.

Windows is a bit more fiddly.

We have to do things a little differently here, because the API only allows checking for changes in a directory, not to a specific file. We use the _splitpath_s standard library function to get the directory name and the device path from the file name, then use FindFirstChangeNotification to create a handle to watch that directory.

Readers will note that this means we’re creating excessive watches on Windows if there are multiple files in the same directory. If (when?) I decide to make this more robust and general-purpose, I’d presumably have to store some additional information of which files we wanted to see in the particular directory, filter the events, and avoid creating extra directory watches as needed. That seemed somewhat annoying – maintaining a dynamic map data structure in C does not sound particularly fun – and for this use-case excessive notifications aren’t that big a problem. I probably will eventually nerd-snipe myself into doing that though…

Anyway, next we have a function for removing watches. I’m not actually using this now though. I previously was also watching for file deletions and removing the watch, but since Prolog still will have that file in its collection of source files, it would get added again.

Additionally, handling that same logic with kqueue and Windows events looked complex – having to actually read out the notifications and determine which file was deleted – so I just took that code out of the Prolog layer. This function lives on though; at some point, I’ll want to burn a bunch of time generalizing and fixing these things.

static foreign_t pl_watchdir_remove_monitor(term_t watchdir, term_t watch) {
  watchref *ref;
  int w;

  if (get_watchdir(watchdir, &ref, TRUE) && PL_get_integer_ex(watch, &w)) {
#ifdef __linux__
    if (inotify_rm_watch(ref->fd, w) == 0) {
      return TRUE;
    }
#elif defined(__APPLE__)
    close(ref->watching[w].ident);
    memmove(ref->watching + w, ref->watching + w + 1,
            ref->nevents - w - 1);
    ref->nevents -= 1;
    return TRUE;
#elif defined(__WINDOWS__)
    FindCloseChangeNotification(ref->dwChangeHandles[w]);
    memmove(ref->dwChangeHandles + w, ref->dwChangeHandles + w + 1,
            ref->nHandles - w - 1);
    ref->nHandles -= 1;
    return TRUE;
#endif
  }
  return FALSE;
}

Nothing particularly interesting here anyway; calling the platform-specific functions to remove the watch and, on macOS and Windows, shifting down the arrays of kevents/handles.

Finally, we’ve reached the real meat, the most complex part of the system: Actually reading events.

We’ll start by looking at some linux-specific helper functions:

#ifdef __linux__
static int put_in_event(term_t t, const struct inotify_event *ev) {
  if (ev->mask & (IN_ISDIR | IN_CREATE) ) {
    term_t name = PL_new_term_ref();
    return ( PL_unify_term(name, PL_MBCHARS, ev->name) &&
             PL_unify_term(t, PL_FUNCTOR, FUNCTOR_watchdir3,
                         PL_ATOM, ATOM_created,
                           PL_TERM, name,
                           PL_INT, ev->wd)
             );
  }

  if (ev->mask & (IN_ISDIR | IN_DELETE) ) {
    term_t name = PL_new_term_ref();
    return ( PL_unify_term(name, PL_MBCHARS, ev->name) &&
             PL_unify_term(t, PL_FUNCTOR, FUNCTOR_watchdir3,
                         PL_ATOM, ATOM_deleted,
                           PL_TERM, name,
                           PL_INT, ev->wd)
             );
  }
  return PL_unify_term(t, PL_FUNCTOR, FUNCTOR_watchdir3,
                       PL_ATOM, ATOM_modified,
                       PL_VARIABLE, PL_VARIABLE);
}

#define addPointer(p, n) (void*)((char*)(p)+(n))
#define nextEv(ev)   addPointer((ev), sizeof(*(ev))+(ev)->len)
#endif

This function and macros came from Jan’s inotify pack. The put_in_event function is a helper for unifying a new watchdir/3 term with the appropriate information from the inotify_event struct. The macros are used below, for reading out the events.

For the code here, we’ll first look at the common prelude, then examine each platform-specific implementation individually.

static foreign_t pl_watchdir_read_event(term_t watchdir, term_t event, term_t timeout_term) {

  watchref *ref;
  if (!get_watchdir(watchdir, &ref, TRUE)) {
    return FALSE;
  }

  int has_timeout = FALSE;
  int timeout = 0;
  {
    double t;
    if (PL_get_float_ex(timeout_term, &t)) {
      has_timeout = TRUE;
      timeout = (int)(t * 1000.0);
    }
  }

Basic FFI things here; get the watchdir struct from the term, get the requested reading timeout. After this, we get into the platform-specifc stuff:

#ifdef __linux__
  if (ref->ev == NULL) {
    if (has_timeout) {
      struct pollfd fds[1];
      for (;;) {
        fds[0].fd = ref->fd;
        fds[0].events = POLLIN;

        int rc = poll(fds, 1, timeout);
        if (rc < 0 && errno == EINTR) {
          if (PL_handle_signals() < 0) {
            return FALSE;
          }
          continue;
        }
        if (rc == 0) {
          return FALSE;
        }
        break;
      }
    }

    ssize_t len = read(ref->fd, ref->buf, sizeof(ref->buf));
    if (len < 0) {
      return watchdir_error(ref);
    }
    ref->len = len;
    ref->ev = (struct inotify_event*)ref->buf;
    assert((char*)ref->ev < &ref->buf[ref->len]);
  }

  term_t ev = PL_new_term_ref();
  if (put_in_event(ev, ref->ev)) {
    ref->ev = nextEv(ref->ev);
    if ((char*)ref->ev >= &ref->buf[ref->len]) {
      ref->ev = NULL;
    }
    return PL_unify(event, ev);
  }

  return FALSE;

On Linux, with inotify, we use poll to listen to the inotify file descriptor (re-trying if the read was interrupted). If we get time-out, we return false. Otherwise, we read the available data into the buffer in our watchref struct, create an inotify_event struct out of that, then use our helper function to unify that struct with our “output” value and return. Note that we also potentially read more than one event at once, so if we have a bunch of events, we don’t need to poll next time.

Next, we look at the kqueue implementation.

#elif defined(__APPLE__)
  {
    struct timespec ts;
    ts.tv_sec = 0;
    ts.tv_nsec = timeout * 1000000;
    for (;;) {
      struct kevent evt;
      int events_count = kevent(ref->fd, ref->watching, ref->nevents, &evt, 1, &ts);
      if (events_count < 0 && errno == EINTR) {
        if (PL_handle_signals() < 0) {
          return FALSE;
        }
        continue;
      }
      if (events_count < 0 || evt.flags == EV_ERROR) {
        return watchdir_error(ref);
      }
      if (events_count == 0) {
        return FALSE;
      }
      return PL_unify_term(event, PL_FUNCTOR, FUNCTOR_watchdir3,
                           PL_ATOM, ATOM_modified,
                           PL_VARIABLE, PL_VARIABLE);
    }

It looks pretty similar to the Linux version; instead of poll-ing, we use kevent, passing it the list of structs we’ve built. The handling code is simpler here, because – at this point – we’re only listening for change events and don’t actually care which file was changed, so we just unify the return value with watchdir(modified, _, _). That is, we leave the contextual information unbound, since (once again!) in this particular use-case, it doesn’t matter which file changed.

Finally, our Windows version:

#elif defined(__WINDOWS__)
  {
    DWORD res = WaitForMultipleObjects(ref->nHandles, ref->dwChangeHandles, 0,
                                       timeout);
    if (res == WAIT_FAILED) {
      return watchdir_error(ref);
    }
    if (res == WAIT_TIMEOUT) {
      return FALSE;
    }
    do {
      size_t idx = res - WAIT_OBJECT_0;
      FindNextChangeNotification(ref->dwChangeHandles[idx]);
      res = WaitForMultipleObjects(ref->nHandles, ref->dwChangeHandles, 0, 0);
    } while (res >= WAIT_OBJECT_0 && res < WAIT_OBJECT_0 + ref->nHandles);
    return PL_unify_term(event, PL_FUNCTOR, FUNCTOR_watchdir3,
                         PL_ATOM, ATOM_modified,
                         PL_VARIABLE, PL_VARIABLE);
  }
#endif
}

Again, a structure vaguely resembling the other cases.

We use the Windows-specific WaitForMultipleObjects as the equivalent of poll and kevent. The strange do-while loop construct is here because WaitForMultipleObjects just tells us at least one thing is ready to read. We consume the notification with FindNextChangeNotification, then loop as long as there are still things readable – this is my attempt to work around the issue of having potentially multiple, redundant watches.

Finally, as in the kqueue case, we end by making a “fake” watchdir/3 functor, just saying something was modified. This is another bit that will probably eventually become more complicated when I want to make the watcher more general-purpose.

The file ends with the standard foreign module bit to register all the atoms, functors, and predicates we’ve used:

install_t install_watchdir4pl() {

  ATOM_created  = PL_new_atom("created");
  ATOM_deleted  = PL_new_atom("deleted");
  ATOM_modified = PL_new_atom("modified");

  FUNCTOR_error2          = PL_new_functor(PL_new_atom("error"),          2);
  FUNCTOR_watchdir_error2 = PL_new_functor(PL_new_atom("watchdir_error"), 2);
  FUNCTOR_watchdir3       = PL_new_functor(PL_new_atom("watchdir"),       3);

  PL_register_foreign("dir_monitor_init",   1, pl_watchdir_create_dir_monitor, 0);
  PL_register_foreign("dir_monitor_stop",   1, pl_watchdir_stop,               0);
  PL_register_foreign("dir_monitor_add",    3, pl_watchdir_add_monitor,        0);
  PL_register_foreign("dir_monitor_remove", 2, pl_watchdir_remove_monitor,     0);
  PL_register_foreign("dir_monitor_read",   3, pl_watchdir_read_event,         0);
}

We made it!

.build.yml

Speaking of making it…how can we make this pack easily installable?

For my previous packs, I’d put them on Github, which the pack_install infrastructure has particular support for. It can also work with arbitrary other pack urls, of course, but one annoying little wrinkle is that it requires the pack archive it downloads have a single extension – so, mypack.zip is okay, mypack.tgz is okay, but mypack.tar.gz won’t work (according to the documentation, at least…but looking at the source, I’m not actually sure if that’s the case…). To install it then, I had to get prospective users to run pack_install(automake, [url('https://git.sr.ht/~jamesnvc/automake'), pack(automake), git(true)]). A bit of a mouthful and not nearly as nice as the normal pack_install(automake).

More importantly though, since the pack has a foreign library, building it requires a C compiler. That is, I think, something I can somewhat assume is the case for SWI users on Linux and macOS, but on Windows that’s more of a pain.

Since I had already figured out cross-compilation in the process of testing the Windows implementation and I was eager to try out Sourcehut’s much-touted build system, I decided to make a build file that would generate a zip archive, with a pre-built Windows DLL included. As I mentioned in my previous post, it took a while to get this all working and figured out, but rope.pl’s .build.yml was a very helpful example to get me started.

Let’s break down what this does:

image: fedora/34
packages:
  - mingw64-gcc
  - wget
  - tar
  - zip
  - p7zip
  - p7zip-plugins
  - pl # swi-prolog
sources:
  - https://git.sr.ht/~jamesnvc/automake
secrets:
  - a7bc4f48-af19-42d5-8487-e65930f81706

This is the standard sort of beginning for the build file. We use fedora as the base, mainly just because that’s what Jan uses as a Docker image for his Windows cross-compilation.

We include our source, of course, as well as a “secret”. That secret is a file, which will be put at ~/.oauth2_token; more on that in the final step.

We also install a number of packages; we’ll see why we need these ones below, as we look at the tasks:

tasks:
  - getwinswipl: |
      mkdir -p swiplwin
      cd swiplwin
      wget -q https://www.swi-prolog.org/download/devel/bin/swipl-8.5.7-1.x64.exe
      7z x -y swipl-8.5.7-1.x64.exe bin/libswipl.dll include

The first task downloads a Windows build of SWI (using wget) and extracts the libswipl DLL and the header files from it (p7zip & p7zip-plugins). Those will be needed for the cross-compilation step, coming up next:

- package: |
    cd automake
    LD=x86_64-w64-mingw32-gcc CC=x86_64-w64-mingw32-gcc \
     PACKSODIR="lib/x64-win64" SOEXT=dll \
     LDSOFLAGS=" -shared -L${HOME}/swiplwin/bin " \
     CFLAGS=" -I${HOME}/swiplwin/include " \
     SWISOLIB=" -lswipl " SWILIB=" -lswipl " \
     make
    make clean
    echo "export _PACK_NAME=$(swipl -s pack.pl -g "name(N), writeln(N)" -t halt)" \
    | sudo tee /etc/profile.d/pack.sh
    echo "export _PACK_VERS=$(swipl -s pack.pl -g "version(V), writeln(V)" -t halt)" \
    | sudo tee -a /etc/profile.d/pack.sh
    source /etc/profile.d/pack.sh
    zip -r "${_PACK_NAME}-${_PACK_VERS}.zip" pack.pl prolog c lib README.md Makefile LICENSE

Here, we set a bunch of environment variables to set the compiler to be the mingw cross-compiler, reference the libswipl dll and header files we downloaded in the previous step, and run make to generate the DLL. We run make clean after that (to get rid of the intermediate object file) and finally generate a .zip file with all the necessary assets.

We also use swipl (installed from the package manager) to read out the pack name and version from the pack.pl file and use that to name the zip archive. Obviously, we could have hard-coded the name part, but I wanted to make this as general as I could, so the recipe can be re-used for other Prolog packs.

Finally, we upload the zip:

- upload: |
    source /etc/profile.d/pack.sh
    set +x
    source ~/.oauth2_token
    set -x
    cd automake
    if git describe --exact-match; then
      ZIP_NAME="${_PACK_NAME}-${_PACK_VERS}.zip"
      TAG=$(git describe --tags)
      set +H
      set +x
      curl --oauth2-bearer "${OAUTH2_TOKEN}" https://git.sr.ht/query \
       -F operations="{\"query\": \"mutation(\$file: Upload!) { uploadArtifact(repoId: 73902, revspec: \\\"${TAG}\\\", file: \$file) { id } }\", \"variables\": {\"file\": null}}" \
       -F map='{"0": ["variables.file"]}' -F 0=@$ZIP_NAME
      set -x
      set -H
      # swipl from repo doesn't include headers?
      sudo cp ~/swiplwin/include/*.h /usr/local/include
      swipl -q -g "pack_remove('${_PACK_NAME}')" \
       -g "pack_install('https://git.sr.ht/~jamesnvc/automake/refs/download/$(git describe --tags)/${ZIP_NAME}', [interactive(false)])" -t halt
    fi

There are a few things worth pointing out here:

First, I set +x before sourcing the ~/.oauth2_token file, which sets OAUTH2_TOKEN to contain the token we use to authenticate the upload, so that token doesn’t get included in the build output (which is publically visible).

Next, we use git describe --exact-match to check if the current git HEAD has an (annotated) tag on it, meaning it’s a release.

If it does, we use curl to add the zip file to the release. It uses the Sourcehut GraphQL API for this, which is a little annoying to do with curl, but it works. I had previously tried to use the hut tool instead, but that required installing go, had somewhat unclear documentation, and didn’t seem to be working properly for me. I’d rather just use curl and understand exactly what’s happening anyway.

We not only set +x around this (again, so the OAUTH2_TOKEN doesn’t show up), but also set +H to disable bash history expansion. This needs to be done so the exclaimation mark in the GraphQL “operations” doesn’t get interpreted. That could have been worked around by using single-quotes instead of double-quotes around that JSON parameter, but I also need to include the tag as the revspec argument. A little annoying; I think I could probably do something with jq to create the json document without needing all the extra escaping…but this works for now.

Anyway, after that’s done, we run swipl to remove and then re-install the new version of the pack. This both validates that the install works and also has the side effect of registering the new version on the pack page.

Fin

There you have it, a detailed walk-through of the automake pack. Hopefully this provides some value to someone out there trying to make their own cool thing with Prolog. If it does, let me know!

Footnotes:

1

For a while, I had :- set_prolog_flag(autoload, false). in my ~/.config/swi-prolog/init.pl to force me to always be clear about exactly which modules and predicates I was using (which broke a fair few libraries).