The queue system allows placing items in a queue and processing them later. The system tries to ensure that only one consumer can process an item.
Before a queue can be used it needs to be created by DrupalQueueInterface::createQueue().
Items can be added to the queue by passing an arbitrary data object to DrupalQueueInterface::createItem().
To process an item, call DrupalQueueInterface::claimItem() and specify how long you want to have a lease for working on that item. When finished processing, the item needs to be deleted by calling DrupalQueueInterface::deleteItem(). If the consumer dies, the item will be made available again by the DrapalQueueInterface implementation once the lease expires. Another consumer will then be able to receive it when calling DrupalQueueInterface::claimItem().
The $item object used by the DrupalQueueInterface can contain arbitrary metadata depending on the implementation. Systems using the interface should only rely on the data property which will contain the information passed to DrupalQueueInterface::createItem(). The full queue item returned by DrupalQueueInterface::createItem() needs to be passed to DrupalQueueInterface::deleteItem() once processing is completed.
While the queue system makes a best effort to preserve order in messages, due to the pluggable nature of the queue, there is no guarantee that items will be delivered on claim in the order they were sent. For example, some implementations like beanstalkd or others with distributed back-ends like Amazon SQS will be managing jobs for a large set of producers and consumers where a strict FIFO ordering will likely not be preserved.
The system also makes no guarantees about a task only being executed once: callers that have non-idempotent tasks either need to live with the possiblity of the task being invoked multiple times in cases where a claim lease expires, or need to implement their own transactions to make their tasks idempotent.
Drupal core! This is an important milestone for the project so enable
the module and check it out ... :)
Thanks to Rok Žlender, Károly Négyesi, Jimmy Berry, Kevin Bridges, Charlie
Gordon, Douglas Hubler, Miglius Alaburda, Andy Kirkham, Dimitri13, Kieran
Lal, Moshe Weitzman, and the many other people that helped with testing
over the past years and that drove this home.
It all works but it is still rough around the edges (i.e. documentation
is still being written, the coding style is not 100% yet, a number of
tests still fail) but we spent the entire weekend working on it in Paris
and made a ton of progress. The best way to help and to get up to speed,
is to start writing and contributing some tests ... as well as fixing
some of the failures.
For those willing to help with improving the test framework, here are
some next steps and issues to resolve:
- How to best approach unit tests and mock functions?
- How to test drupal_mail() and drupal_http_request()?
- How to improve the admin UI so we have a nice progress bar?
- How best to do code coverage?
- See http://g.d.o/node/10099 for more ...
Dries Buytaert
Angie Byron