Symfony2 and Gearman: Parallel Processing and Background Processing
On a few of our projects we have a few different needs to either queue items to be processed in the background or we need a single request to be able to process something in parallel. Generally we use Gearman and the GearmanBundle. Let me explain a few different situations where we’ve found it handy to have Gearman around.
Background Processing
Often we’ll need to do something which takes a bit more time to process such as sending out a couple thousand push notifications to resizing several images. For this example lets use sending push notifications. You could have a person sit around as each notification is sent out and hope the page doesn’t timeout, however after a certain number of notifications, not to mention a terrible user experience, this approach will fail. Enter Gearman. With Gearman you are able to basically queue the event that a user has triggered a bunch of notifications that need to be processed and sent.
<?php
// In controller
public function triggerSendNotificationsAction(Request $request)
{
// Code to get that user triggered a bunch of notifications to an array of user ids.
$userIds = array(.......);
$message = 'some message from the frontend user';
//Get Gearman and tell it to run in the background a 'job'
$this->get('gearman')->doBackgroundJob('MyDemoBundleWorker~sendNotificationsToUsers',
json_encode(array('user_ids'=>$userIds,'message'=>$message)));
return $this->renderText('Your notifications are being sent now!');
}What we’ve done above is sent to the Gearman server a job to be processed in the background which means we don’t have to wait for it to finish. At this point all we’ve done is queued a job on the Gearman server, Gearman itself doesn’t know how to run the actual job. For that we create a ‘worker’ which reads jobs and processes them:
<?php
<?php
namespace My\DemoBundle\Gearman\Worker;
use Mmoreram\GearmanBundle\Driver\Gearman;
use Symfony\Component\DependencyInjection\Container;
/**
* @Gearman\Work(
* iterations = 30,
* description = "Test worker",
* service = "my.test_worker"
* )
*/
class TestWorker
{
private $container;
public function __construct(Container $container)
{
$this->container = $container;
}
/**
* Test method to run as a job
*
* @param \GearmanJob $job Object with job parameters
*
* @return boolean
*
* @Gearman\Job(
* name = "sendNotificationsToUsers",
* description = "Sends a list users an awesome notification"
* )
*/
public function sendNotificationsToUsers(\GearmanJob $job)
{
// Get the data we put in for the job previously.
$data = json_decode($job->workload(),true);
foreach($data['user_ids'] as $userId)
{
// Some task that takes a long, long time to do.
$notification->send();
}
}
}The worker will consume the job and then process it as it sees fit. In this case we just loop over each user ID and send them a notification.
Parallel Processing
One one of our applications users can associate their account with multiple databases. From there we go through each database and create different reports. On some of the application screens we let users poll each of their databases and we aggregate the data and create a real time report. The problem with doing this synchronously is that you have to go to each database one by one, meaning if you have 10 databases and each one takes 1 seconds to get the data from, you have at least ten seconds the user is waiting around; this doesn’t go well when you have 20 databases and so on. Instead, we use Gearman to farm out the task of going to each database and pull the data. From there, we have the request process total up all the aggregated data and display it. Now instead of waiting 10 seconds for each database, we farm out the work to 10 workers, wait 1 second and then can do any final processing and show it to the user. In the example below for brevity we’ve just done the totaling in a controller.
<?php
// In controller...
public function getReportAcrossDatabasesAction(Request $request)
{
// Listen for Gearman to notify that it has data from a completed job.
$this->get('event_dispatcher')
->addListener('gearman.client.callback.complete',array($this,'gearmanCallback'));
$gearman = $this->get('gearman');
//Assume $request->request->get('connectionIds') is list of connections.
foreach($request->request->get('connectionIds') as $connID)
{
$gearman->addTask('MyDemoBundleWorker~gatherVastAmountsOfData',array(json_encode('connection_id'=>$connId)));
}
$gearman->runTasks();
$totalRowCount = 0;
foreach($this->callbackData as $data)
{
$totalRowCount+= $data['total'];
}
return array('totalRowCount'=>$totalRowCount);
}
public function gearmanCallback(GearmanClientCallbackCompleteEvent $e)
{
$this->callbackData[] = json_decode($e->getGearmanTask()->data(),true);
}What we’ve done here is created a job for each connection. This time we add them as tasks, which means we’ll wait until they’ve completed. On the worker side it is similar to except you return some data, ie return json_encode(array('total'=>50000)); at the end of the the function.
What this allows us to do is to farm out the work in parallel to all the databases. Each worker runs queries on the database, computes some local data and passes it back. From there you can add it all together (if you want) and then display it to the user. With the job running in parallel the number of databases you can process is no longer limited on your request, but more on how many workers you have running in the background. The beauty with Gearman is that the workers don’t need to live on the same machine, so you could have a cluster of machines acting as ‘workers’ and be able to process more database connections in this scenario.
Anyways, Gearman has really made parallel processing and farming out work much easier. As the workers are also written in PHP, it is very easy to reuse code between the frontend and the workers. Often, we’ll start a new report without Gearman; getting logic/fixing bugs in a single request without the worker is easier. After we’re happy with how the code works, we’ll move the code we wrote into the worker and have it just return the final result.
Good luck! Feel free to drop us a line if you need any help.