Hi,

I am using the RSS Connector module with the External Content module to import items from a remote RSS feed into my SilverStripe database as Blog Entries. This works great.

However, I would really, REALLY like to make this an automagic process. This is where the Queued Jobs module comes in, I am told..... I don't know where to begin. Queued Jobs comes with an example job called "PublishItemsJob," and I understand that I run it with the command dev/tasks/ProcessJobQueueTask, but I don't understand how to actually PUT a job in the queue. Where am I telling the system to RUN the PublishItemsJob?? Where/how am I defining it?

I know this is vague but I'm having a hard time wrapping my mind around this.

Thanks in advance,
Garrett

There's a couple of ways to make this happen - one of which should actually be possible now that I've committed the scheduled import job (oops!). If you can, grab the latest source from https://github.com/nyeholt/silverstripe-external-content, which should now have a ScheduledExternalImportJob class in it. Once you have the latest, when you go to the 'Import' tab of the external content source/item, you should have the option to choose an import schedule; this will automatically create the queued job for you

An alternative is to create your own QueuedJob which read data from the external connector and trigger importing appropriately. When I do these types of recurring jobs, I have something at the end of the 'process()' method that does something like

$delay = 3600; // how many seconds before executing the job the next time
$job = new MyJob();
singleton('QueuedJobService')->queueJob($job, date('Y-m-d H:i:s', time() + $delay));

You can create jobs by using a task; dev/tasks/CreateQueuedJobTask?name=(job name)

Hope that helps!

Hi Marcus,

My very sincere thanks for your reply. This is *exactly* what I need -- the ability to schedule a job! I cloned the git project and popped the new code into my external-content module and voila, the dropdown to choose the schedule is there.

So I tried it, and here's the deal.

If I do NOT choose to repeat the task, it works. If I set any value in the "Repeat import each" field, nothing is imported. It does show up on Jobs tab under "Queued," but nothing ever updates. It just sits there.

Can you, first, think of anything that could be causing it not to execute on its own, and second, think of a good way to debug this?

Thanks again,
Garrett

What does your crontab look like?

You should have a cronjob doing something like

*/1 * * * * php sapphire/cli-script.php dev/tasks/ProcessJobQueueTask queue=2

to actually execute the job.

To test that the job actually works, you can manually execute the job by clicking the gear icon

This is my crontab file:

*/1 * * * * php /Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask 

This is the correct full path. I am on Mac OSX. I'm not sure what the "queue=2" bit in your example is. Does this crontab file need to be in any certain place in the directory structure?? I issued:

$ crontab -e

And I'm not sure where the file got created..... Ok, so after I executed the job using the gear icon, it ran. Yay! Then the next time to run showed up in the queue, exactly 15 minutes later, which was the interval I chose. Yay! And then when that time rolled around, again nothing happened. Boo! What am I doing wrong or not understanding here?

Thanks for your continued attention on this, Marcus.

//Garrett

The queuedjobs module has multiple separate queues that jobs can be created in; shorthand for these at the moment is simply 1, 2 or 3. The idea being that jobs that just need running asap are put in 1, some that can be deferred a little bit put in 2, or those that it doesn't matter when they're run are put in 3. By default it should actually process 2, but to make the crontab clearer, I normally explicitly add queue=2

(if you're going to use the queuedjobs module, you're probably best adding in another entry like

*/10 * * * * php /Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask queue=3

to trigger any long jobs that get created)

From the sounds of things either the cronjob isn't firing, or it is and something is failing before picking up the queue. A couple of things you can do

1) Make sure your PHP binary is the cli (it should be, but some systems can be a bit weird with it)

$ php -v

should show something like the following - the (cli) bit is the important part

PHP 5.3.5-1ubuntu7 with Suhosin-Patch (cli)

2) Try executing the task from the commandline manually - just run

$ php /Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask queue=2

That should produce output like

Running task 'ProcessJobQueueTask'...

[2012-05-04 11:25:39] Processing queue 2
[2012-05-04 11:25:39] No new jobs

If that works, then the crontab isn't executing properly. You could try something simple like

*/1 * * * * echo "test" >> ~/test-cron

which should output stuff to a test-cron file in your homedir

Thanks for all the help.

My crontab is working, as the test-cron file is being created and written.

The job also worked when I executed the command from the terminal.

I guess I just don't get the relationship between what I see in queued jobs and what is in my crontab file. Do they work with each other? What is the point of the queued jobs module/UI if it doesn't actually write the job/command to the crontab file? If I say I want it to run every 15 mins in the UI and then I have */60 in my crontab, which will happen?

I am assuming that the command in the crontab simply runs WHATEVER is queued and the the queued jobs module is simply a place to create the jobs. Yes? But the interval time conflict confuses me. And then we can get back to why this isn't executing.

This is what I've got now:

*/15 * * * * php /Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask queue=3

//Garrett

The relationship between the Jobs list and the crontab entry is that each job in the list will be executed in the order they are added to the queue, except if they're given a date in the future to execute at. So the crontab entry is basically saying 'every minute, check if there are any jobs on the queue that can be executed now'.

What time is the job set to run at (the Start After field in the table)? If this is still in the future, the ProcessJobQueueTask will ignore it until that date/time is reached.

Also, for your cron you should have it set to execute both queue=2 and queue=3, like the following. The "Job Type" column in the Jobs tab will indicate which queue the job has been placed on.

*/1 * * * * php /Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask queue=2
*/15 * * * * php /Users/gfisher/Sites/horn/horn/sapphire/cli-script.php dev/tasks/ProcessJobQueueTask queue=3