Simple Queue Package for R

I recently coded up a simple package that implements a file-based queue abstract data type. This package was needed for a different package that I’m working on involving parallel processing (more on that in the near future). Actually, this othe package is a project that I started almost nine years ago but was never able to get off the ground. I tried to implement a queue interface in the filehash package but it never served the purpose that I needed.

Recently, I came across Rich FitzJohn’s thor package, which provides an interface to the LMDB database, which is a light-weight key-value style database. When I saw this, I realized that it was exactly what I needed because it supports transactions with blocking. So if multiple processes attempt to write to the database at once, it will block the other processes untill the currently writing one is finished, and then move on to the next one.

The code for the queue package is available on GitHub and can be installed in the usual way via devtools:

devtools::install_github("rdpeng/queue")

The package just involves four functions:

• enqueue(): add an element to the queue
• dequeue(): remove an element from the queue (and return it)
• peek(): return the next element of the queue (but don’t remove it)
• is_empty(): is the queue empty:

There are also functions

• create_queue: create the queue and associated files on the disk
• init_queue: initialize a queue that has been previously created

Everything is implemented as S3 classes and methods. Queues are just sub-directories on the filesystem that contain some metadata for the underlying LMDB database.

You can create a queue with the create_Q() function.

library(queue)
x <- create_queue("testq")
is_empty(x)

[1] TRUE

You can then add arbitrary objects to the queue with enqueue(). Behind the scenes, objects are serialize()-ed. 

enqueue(x, "hello")
enqueue(x, 1)
peek(x)

[1] "hello"

Finally, you can remove an object from the front of the queue with dequeue().

obj <- dequeue(x)
obj

[1] "hello"

peek(x)

[1] 1

The implementation is pretty basic right now but the guts of it are there. I don’t plan to add any more features, but hopefully will round out the documentation and add some tests.

~/myDocuments/git/simplystats_distill/_posts/2017-08-28-simple-queue-package-for-r/index.Rmd