Queue for Kirby

Plugins
1

I was in need for a Queue, so I made one. :slight_smile:

This plugin adds a basic queue to the Kirby CMS, using Cron and Kirby’s flat file system.

It’s mostly designed to be used in other plugins, such as the new Webmentions plugin I’m writing.

Github: https://github.com/sebsel/queue-for-kirby

queue::define('job_name', function($job) {
    $job->get('param');
    // contains 'some data' in the job added below
});

queue::add('job_name', [
    'param' => 'some data'
]);

Same post at Seblog.nl and Jens’ Kirby-Plugins repo

4 Likes
2

Already at version 2.0.0: I added a widget to it today.

queue-widget
queue-widget.jpg960×668 79 KB

3 Likes
3

I hope it works as well as it looks. :slight_smile:

It makes me want to code a plugin for conjobs. We can have the cronjob on a server, but it’s also possible to trigger a job from the site and the panel.

When adding, saving, deleting, renaming or updating a page, it could check the time when the last run was and if it was more than x minutes ago, run again.

It could do the same thing when a page loads for a user.

A textfile for each job could be used. Maybe it does not even need to read the file, it just need to look at the modified timestamp of the file and update it accordingly.

4

I hope so too! My testing works fine, but I haven’t completed the Webmention plugin, so I am not using it live yet :slight_smile:

I don’t really understand what you mean by ‘a plugin for cronjobs’. What you describe is more or less what this plugin does, or am I missing something?

Note that the ‘X jobs waiting’ button in the widget links to /panel/queue/work at the moment, which will try and work all the jobs from the queue. Probably not the best, but it’s triggering (all the) jobs from the panel, kind of as you described.

I was also thinking about adding an option for a web-bug, to use for people without Cron on their server. That would be an image, like <img src="/worker.php">, and then in worker.php, send a one-pixel image, drop connection, and then do the work, described here.

And then there is a style of worker I can’t use, but what is really a worker.php: a worker that you just run with $ php worker.php in your terminal, which has something like this:

// load all of Kirby once!

while (true) {
    while (queue::hasJobs()) {
        queue::work();
    }
    sleep(10);
}
// And then just keep checking for work, every 10 seconds, 
// never ending the php script.

Oh, and it might be nice to schedule jobs for the future. In my case, “when I posted a post… check back in 12 minutes and see if my mobile phone sent my location from that time to the server”. (I have an app that tracks that, and submits it every 10 minutes.) So that’s a function I might add.

Enough to improve! (But I’ll be working on Webmentions first.)

5

Alright then. I thought we would need a server cron service to work as trigger for your plugin? My plugin idea was to fake such a trigger, to make it trigger on what is happening in the panel or the site, that then for example could trigger your plugin to run. But maybe you have it built in?

Anyway, I don’t like the look of the image thing, looks too hacky for my taste. To trigger it from the site I think a better approach would be to use something similar to https://github.com/jenstornell/kirby-starthook or just on load by the custom plugin.

6

Ahh, I see! Yeah, I am on shared hosting, but still have Cron (in my DirectAdmin panel, under advanced, but it’s there), and most of the shared host I have seen here in The Netherlands have it enabled too. So that’s why I took this cronjob approach first.

I totally agree the image looks hacky, but your starthook runs on pageload, right? Then you’re just moving the work to the next request, which is even worse than doing it inline. (A person sending a webmention knows what it’s waiting for, but the person who visits the homepage right after a webmention was requested just thinks the site is slow.)

The whole point about a queue is to get work out of the pageload, into a separate process. Any solution that triggers with a panel hook, within a plugin, etc, will defeat the purpose of the queue.

7

For anyone who’s reading along: if you just want to define a job and do it / trigger it somewhere else in your code, you can use Kirby’s triggers:

kirby()->hook('my.action', function($name, $message) {
  echo 'hi ' . $name . ', ' . $message;
});

// and elsewhere:

kirby()->trigger('my.action', ['Seb', 'how are you?']);

Or take a look at Kirby 3’s new Events:

use Kirby\Events\Events;

// the class with the events trait
class MyClass {
    use Events;
}

// create the event
MyClass::on('say', function($message) {
    echo $message;
});

// trigger the event
MyClass::trigger('say', 'hello');
8

I totally agree the image looks hacky, but your starthook runs on pageload, right?

Yes it does. The big difference to just use a plugin in this case is that it’s aware of the $page object, which you don’t have when loading a plugin.

The whole point about a queue is to get work out of the pageload, into a separate process. Any solution that triggers with a panel hook, within a plugin, etc, will defeat the purpose of the queue.

Alright. I tend to agree. It will take some pageload depending on how slow the job is. I agree with that.

I have a project http://modehallen.se/ where I populate a database. Often it does not do everything in one job. It’s split it up into a smaller jobs. For example, it insert 100 products each time, instead of 10000 in one go.

which is even worse than doing it inline

Instead of doing what I suggested with starterhook or the image hack, it could probably be solved with an ajax call.

By this discussion it’s clear that a server cron is less hacky and zero overhead, so it should probably be used when ever possible.

9

Often it does not do everything in one job. It’s split it up into a smaller jobs. For example, it insert 100 products each time, instead of 10000 in one go.

The plugin does not split up jobs at the moment, the worker will just run for all 10.000 jobs, but if you’re using the Cron setup, it will do it behind the scenes. (It will still take a while, but at least people are not waiting for it, and the work has to be done anyway, or else you shouldn’t queue it in the first place.)

Instead of doing what I suggested with starterhook or the image hack, it could probably be solved with an ajax call.

Sure, that’s the same thing. The important part is that you cut the connection to the client as soon as possible, and ignore that connection loss. You just want to trigger a work script, you are not loading something for the user. What if your job takes one minute, the user closes the page after 30 seconds, and your script terminates mid-work? If you’re using an image or an AJAX call: return something, disconnect, then work.

Edit: note that the cron job is always preferred above the AJAX or img, because if you don’t have visitors, you’re job will not be done with the AJAX or img route.

1 Like