Symfony2: Making impersonating a user more friendly

With Symfony2 the firewall comes with a built in feature: impersonate a user. We’ve been using impersonation as an admin tool for about 5 years as it is very effective for troubleshooting. When a user files a support ticket saying something isn’t showing properly to them or they are getting random errors it is very easy to just quickly switch to that user and see what they are seeing. As with all features, this one may not be appropriate for your application if your user expects no administrative staff to have access to his or her account.

While Symfony’s built in impersonation feature is a great step up from having to build it by hand, it still can be a bit more friendly. We’ve seen two additional functions we wanted the impersonation to handle. First, we wanted it to on exit from impersonating the user returns the user to where the user first started to impersonating. Currently it just brings you back to wherever you link the user. Second, if already impersonating a user and trying to start to impersonate another, we didn’t want it to throw an error but to quietly switch you. This functionality could lead to unwanted circumstances if an impersonating user believes they can impersonate another user, and then slowly just keep exiting impersonation of each user and go back up the chain they went down. However, in our situation the time admins hit this was when they’d impersonate one user, realize they clicked the wrong one, click back and try to impersonate a different user. As the browser uses it’s cached page when the user hits back they see the list of users as if they were an admin and can click on the correct user. If they do this they are hit with a 500 error, “You are already switched to X user”.

For both of our goals we overrode the built in switch user class. It is really easy to override, as all you need to do is specify in your parameters.yml “security.authentication.switchuser_listener.class: My\AppBundle\Listener\SwitchUser”. We used the built in class as our starting template: https://github.com/symfony/symfony/blob/2.5/src/Symfony/Component/Security/Http/Firewall/SwitchUserListener.php Our final class ended looking like:

<?php
namespace My\AppBundle\Listener;

use Symfony\Component\Security\Core\Exception\AccessDeniedException;
use Symfony\Component\Security\Core\SecurityContextInterface;
use Symfony\Component\Security\Core\User\UserProviderInterface;
use Symfony\Component\Security\Core\User\UserCheckerInterface;
use Symfony\Component\Security\Core\Authorization\AccessDecisionManagerInterface;
use Psr\Log\LoggerInterface;
use Symfony\Component\HttpKernel\Event\GetResponseEvent;
use Symfony\Component\Security\Core\Exception\AuthenticationException;
use Symfony\Component\HttpFoundation\RedirectResponse;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\Security\Core\Role\SwitchUserRole;
use Symfony\Component\Security\Core\Authentication\Token\UsernamePasswordToken;
use Symfony\Component\Security\Core\Exception\AuthenticationCredentialsNotFoundException;
use Symfony\Component\Security\Core\Authentication\Token\TokenInterface;
use Symfony\Component\Security\Http\Event\SwitchUserEvent;
use Symfony\Component\Security\Http\Firewall\ListenerInterface;
use Symfony\Component\Security\Http\SecurityEvents;
use Symfony\Component\EventDispatcher\EventDispatcherInterface;

/**
 * Based on built in Symfony 2 SwitchUserListener class.  Modified to allow:
 * 1. Redirect user on exit back to original impersonation url if one exists
 * 2. Allow user to impersonate different user if they are already impersonating a user.
 */
class SwitchUserListener implements ListenerInterface
{
    private $securityContext;
    private $provider;
    private $userChecker;
    private $providerKey;
    private $accessDecisionManager;
    private $usernameParameter;
    private $role;
    private $logger;
    private $dispatcher;

    // Used to disable the URI redirect in case user is already impersonating one user and trying to switch to another.
    private $useOverrideUri;

    /**
     * Constructor.
     */
    public function __construct(SecurityContextInterface $securityContext, UserProviderInterface $provider, UserCheckerInterface $userChecker, $providerKey, AccessDecisionManagerInterface $accessDecisionManager, LoggerInterface $logger = null, $usernameParameter = '_switch_user', $role = 'ROLE_ALLOWED_TO_SWITCH', EventDispatcherInterface $dispatcher = null)
    {
        if (empty($providerKey)) {
            throw new \InvalidArgumentException('$providerKey must not be empty.');
        }

        $this->securityContext = $securityContext;
        $this->provider = $provider;
        $this->userChecker = $userChecker;
        $this->providerKey = $providerKey;
        $this->accessDecisionManager = $accessDecisionManager;
        $this->usernameParameter = $usernameParameter;
        $this->role = $role;
        $this->logger = $logger;
        $this->dispatcher = $dispatcher;
        $this->useOverrideUri = true;
    }

    /**
     * Handles the switch to another user.
     *
     * @param GetResponseEvent $event A GetResponseEvent instance
     *
     * @throws \LogicException if switching to a user failed
     */
    public function handle(GetResponseEvent $event)
    {
        $request = $event->getRequest();

        if (!$request->get($this->usernameParameter)) {
            return;
        }

        if ('_exit' === $request->get($this->usernameParameter)) {
            $this->securityContext->setToken($this->attemptExitUser($request));
        } else {
            try {
                $this->securityContext->setToken($this->attemptSwitchUser($request));
            } catch (AuthenticationException $e) {
                throw new \LogicException(sprintf('Switch User failed: "%s"', $e->getMessage()));
            }
        }

        $session = $request->getSession();

        $request->query->remove($this->usernameParameter);

        $overrideUri = $session->get('onSwitchURI',null);
        if($request->get('returnTo'))
        {

            $session->set('onSwitchURI',$request->get('returnTo'));
            $request->query->remove('returnTo');

        }
        else
            $session->remove('onSwitchURI');

        $request->server->set('QUERY_STRING', http_build_query($request->query->all()));


        $response = new RedirectResponse($this->useOverrideUri && $overrideUri ? $overrideUri : $request->getUri(), 302);

        $event->setResponse($response);
    }


    /**
     * Attempts to switch to another user.
     *
     * @param Request $request A Request instance
     *
     * @return TokenInterface|null The new TokenInterface if successfully switched, null otherwise
     *
     * @throws \LogicException
     * @throws AccessDeniedException
     */
    private function attemptSwitchUser(Request $request)
    {
        $token = $this->securityContext->getToken();
        $originalToken = $this->getOriginalToken($token);


        if (false !== $originalToken) {
            if ($token->getUsername() === $request->get($this->usernameParameter)) {
                return $token;
            } else {
                // User is impersonating someone, they are trying to switch directly to another user, make sure original user has access.
                if (false === $this->accessDecisionManager->decide($originalToken, [$this->role])){
                    throw new AccessDeniedException();
                }

                // User has a return url, most likely to admin area, as they are just trying to reimpersonate non-admin redirect to default.
                $this->useOverrideUri = false;

            }
        }
        else if (false === $this->accessDecisionManager->decide($token, [$this->role])) {
            throw new AccessDeniedException();
        }

        $username = $request->get($this->usernameParameter);

        if (null !== $this->logger) {
            $this->logger->info(sprintf('Attempt to switch to user "%s"', $username));
        }

        $user = $this->provider->loadUserByUsername($username);
        $this->userChecker->checkPostAuth($user);

        $roles = $user->getRoles();

        // If there is an original token, only let them switch back to that user.
        if($originalToken)
            $roles[] = new SwitchUserRole('ROLE_PREVIOUS_ADMIN', $originalToken);
        else
            $roles[] = new SwitchUserRole('ROLE_PREVIOUS_ADMIN', $this->securityContext->getToken());

        $token = new UsernamePasswordToken($user, $user->getPassword(), $this->providerKey, $roles);

        if (null !== $this->dispatcher) {
            $switchEvent = new SwitchUserEvent($request, $token->getUser());
            $this->dispatcher->dispatch(SecurityEvents::SWITCH_USER, $switchEvent);
        }

        return $token;
    }

    /**
     * Attempts to exit from an already switched user.
     *
     * @param Request $request A Request instance
     *
     * @return TokenInterface The original TokenInterface instance
     *
     * @throws AuthenticationCredentialsNotFoundException
     */
    private function attemptExitUser(Request $request)
    {
        if (false === $original = $this->getOriginalToken($this->securityContext->getToken())) {
            throw new AuthenticationCredentialsNotFoundException('Could not find original Token object.');
        }

        if (null !== $this->dispatcher) {
            $switchEvent = new SwitchUserEvent($request, $original->getUser());
            $this->dispatcher->dispatch(SecurityEvents::SWITCH_USER, $switchEvent);
        }

        return $original;
    }

    /**
     * Gets the original Token from a switched one.
     *
     * @param TokenInterface $token A switched TokenInterface instance
     *
     * @return TokenInterface|false The original TokenInterface instance, false if the current TokenInterface is not switched
     */
    private function getOriginalToken(TokenInterface $token)
    {
        foreach ($token->getRoles() as $role) {
            if ($role instanceof SwitchUserRole) {
                return $role->getSource();
            }
        }

        return false;
    }
}

Here are the specifics on what everything we did and why.

First feature: Redirecting the user on exiting impersonating a user to where they originally started impersonating them. As we didn’t want to go around our entire application updating logic for the exit impersonation links if we decided to later change the behavior, we decided to build the redirect into the class itself. We didn’t want to rely on the user’s browser referrer header, so instead we decided to on the links to impersonate a user to include a “returnTo” parameter. This parameter is set to the current URI (app.request.uri). At line 97 we save the returnTo parameter to the session, for later use. On line 93, as a user is switching (in this case exiting) a user, if the session has a stored “returnTo” URL, we assign it to the “$overrideURI” variable. On line 107 we have a bit of logic on if we redirect them to the default route or the “returnTo” URL. The reason for the additional “$this->useOverrideURI” variable on this line is for our second feature of switching between users when you are already impersonating one. As the logic all runs through the same routine, if you are simply switching to a new user from an already impersonated one, we don’t want to redirect you back to your original URL when you started all the impersonating, so we disregard the redirect in this case and redirect to the default route. An example of this is admin impersonates user A, then wants to impersonate user B. Upon impersonating user B, the admin does not want to be redirected back to the admin dashboard (the sessions returnTo URL), but to where the impersonate user link is pointing to (User B homepage).

Second feature: Allow users to impersonate a different user while already impersonating another. One Line 134 is where the original SwitchUserListener would usually throw a 500 error as you are already impersonating a user. Instead, we make sure that the original token has the appropriate permissions, if so it will not throw an exception. Line 159 is the other main update for this feature. If you are already impersonating a user and try to impersonate another user, upon exiting you want to go back to your original user. Now if a original impersonation token (user) exists, we keep that as the user you’ll be switched to when you exit the impersonation.

Display Ad Preview Tool

Looking for a way to preview online display ads and automatically save a screenshot/grab/capture?

Based on conversations we’ve had with one of our consulting clients, Datapoint Media, who are very familiar with the online advertising industry, it became quickly apparent that there really isn’t a good automated solution currently out there. When a buyer asks for “proofs” of their banner ads on the main sites that they will appear in, Ad Operations personnel are faced with two less than thrilling (and quite time intensive) options:

  • Grab screenshots of the sites that the client would like to preview and download the standalone display ad images the client is buying. Then open up Photoshop or other photo editor and copy and paste those ad images over the existing banner ads on the screenshot of the target websites.
  • Wait until the campaign is in flight and hope to catch lightning in a bottle by loading up the website the ad is likely to rotate into, refreshing the page continuously until the ads the clients bought appear, and finally taking a screenshot of the site.

Imagine having to do this every day week in and week out for hundreds of client orders.

Given the strong demand for a tool and a lack of automated solutions, we worked with Datapoint Media to build a tool as part of their existing Audience Extension platform .

Here’s how the Banner Ad preview tool works:

  1. Simple web based UI allows users to enter a website URL for which they’d like to preview the ads on. Once selected, the website is displayed in an iframe “preview window” to allow the user to get the lay of the land and see the current ad layout of the website.
  2. Users can choose from 3 options on how they want to input the banner ad/creative images they want to display on the selected site. The 3 options are:

    • Upload the actual image file(s)
    • Enter the url(s) of the creative images
    • Enter the Ad Server (such as Google’s DoubleClick For Publishers DFP) line item/ campaign ID that contains the creative ad imageimg1
  3. At this point, users submit the preview request. If they chose the Ad Server ID entry method, the Ad Servers API is pinged for a listing of all the associated creative images. After that, users select which creatives they want to include in the screenshot.img2
  4. The request is placed in a queue to be automatically processed. Next, users are presented with a confirmation that they will receive an email with the screenshot file attached within a few minutes. No need for any more work to be done by humans, it’s time for the robots to do the heavy lifting.
  5. Behind the scenes the tool loads up an “invisible” browser window on the server which points to the target website. Next it executes a series of commands to inspect the website determining where the valid ad slots are located. Once the slots are defined, it matches up the open slots with the dimensions of the banner ads that the user has selected. If the dimensions match, it replaces the existing ads on the website with the user entered banner ads and takes a screenshot.image03
  6. The resulting screenshot file is saved on the application server and automatically emailed to the user.
image01 image04

Want more ad tech gadgets? Sign up!

* indicates required

If you have any questions or are interested in gaining access to the tool, feel free to contact the guys over at http://www.datapointmedia.com.

Net Neutrality: A recap and some cliffnotes

Net Neutrality has been all over the news lately and I’ve been fielding a couple of questions related to it. At Setfive, we think it’s a critically important issue, both to startups and the technology infrastructure of the United States as a whole. Because of that, we decided to pull together an overview, some history, and key outcomes surrounding the Net Neutrality debate. As always, questions or comments welcome!

What is Net Neutrality? First coined by Columbia Law professor Tim Wu, network neutrality, or net neutrality for short, states that internet service providers (such as Verizon and Comcast) and governments should provide you with access to content and data regardless of where it came from equally. Internet service providers (ISPs) are not allowed to discriminate and slow speeds for one company in favor of its competitor.

Essentially, net neutrality maintains a free, open, and fair internet.

The Lead Up To January 14, 2014
  • In 2002, the FCC had the opportunity to regulate ISPs as it had done for the phone companies. Ultimately though, the FCC chose not to at all citing that ISPs are “information services”, completely different than the telecommunication services phone companies provide.
  • However a few years later, the FCC began to notice the enormous power and strength that ISPs had accumulated over the years. In an attempt to curb and regulate them, the FCC created the Open Internet Rules in 2010
The Open Internet Rules established:
  • Enforced transparency of ISPs operations and management of their networks
  • Prohibited ISPs from obstructing access to legal content and applications
  • Maintained an equal and fair playing field online by preventing ISPs from giving preference to one company over another. Essentially becoming the core of net neutrality

In response to these rules, Verizon brought the FCC to court in 2013 on the charge that the agency had no authority to use the Open Internet rules to regulate ISPs.

Fast forward to January 14, 2014
  • On this day, a DC circuit court determined in the Verizon Communications Inc. vs FCC case that portions of the Open Internet Rules especially the ones pertaining to an equal and fair internet could not be applied to ISPs.
  • The reasoning was that portions of the rules apply only to common carriers, which provide telecommunication services. But since ISPs are classified by the FCC as providers of information services, they’re not considered under the law as common carriers.

What does this ruling mean? It eliminated the only existing rules protecting net neutrality. As a result, ISPs can now:

  • Charge companies fees for “premium” access to their consumers. Think Verizon charging Netflix to stream to their customers at better rates.
  • Selectively prioritize one source of traffic over another. Think Comcast prioritizing delivering its Xfinity onDemand service over HBO Go.
  • And of course, create “slow lanes” and “fast lanes” paving the way to charging for ala carte Internet packages, just like TV. Imagine seeing errors like: “Sorry! You need to subscribe to the ‘social package’ to access this site.”

What’s the president’s stance on all this? He’s pro net neutrality and has urged the FCC to establish strong rules that would protect it. However since the FCC is an independent government agency, Obama has no direct influence. Additionally, in a bitterly divided congress some hardline Republicans are taking an anti-Net Neutrality stance to pander to their base. See The Oatmeal on Ted Cruz.

What’s next? The FCC does have the power to reclassify ISPs as telecommunication service providers and thus subject them to the Open Internet Rules. What it decided to do instead is to create a new net neutrality framework that would hold up in court while at the same time satisfy both sides.

Right now, everyone is in a holding pattern waiting for the FCC to make a final announcement.

Boston: Who's hiring PHP developers?

Last week, I was catching up with some friends when one of them asked an interesting question - Which Boston area companies are currently hiring PHP developers? Surprisingly, I didn’t really have a good answer so I decided to find out. To figure this out, I searched job posts that were specifically looking for PHP developers and started pulling together a spreadsheet about the posts. As I was looking at the data, I decided to put together a graphic which is available below along with the list of companies. As always, questions or comments welcome!

CompanyCityCompanyCity
AcquiaBurlingtonADTRANBurlington
ADTRANBurlingtonAllen & GerritsenBoston
Allen & GerritsenBostonApplauseFramingham
ApplauseFraminghamArbor NetworksBurlington
Arbor NetworksBurlingtonBerklee College Of MusicBoston
Berklee College Of MusicBostonBiogen IdecCambridge
Biogen IdecCambridgeBlack Duck SoftwareBurlington
Black Duck SoftwareBurlingtonBlue State DigitalBoston
Blue State DigitalBostonBrafton Inc.Boston
Brafton Inc.BostonBrigham And Women’s HospitalWellesley
Brigham And Women’s HospitalWellesleyBrightcoveBoston
BrightcoveBostonCatalina MarketingBoston
Catalina MarketingBostonComsolBurlington
ComsolBurlingtonConstant ContactWaltham
Constant ContactWalthamContentLEADBoston
ContentLEADBostonD50 MediaWellesley
D50 MediaWellesleyDemandwareBurlington
DemandwareBurlingtonDesire2Learn (D2L)Boston
Desire2Learn (D2L)BostonDew Softech (contract Position)Boston
Dew Softech (contract Position)BostonDigital BungalowSalem
Digital BungalowSalemDynatraceWaltham
DynatraceWalthamEgenerationmarketingBoston
EgenerationmarketingBostonFASTHockeyBoston
FASTHockeyBostonFlipkey, Inc.Boston
Flipkey, Inc.BostonGenscape, Inc.Boston
Genscape, Inc.BostonHarvard Medical SchoolBoston
Harvard Medical SchoolBostonHarvard School Of Public HealthBoston
Harvard School Of Public HealthBostonHill HollidayBoston
Hill HollidayBostonHubspot, Inc.Cambridge
Hubspot, Inc.CambridgeIntegrated Computer SolutionsBedford
Integrated Computer SolutionsBedfordIntersystemsCambridge
IntersystemsCambridgeMediamathCambridge
MediamathCambridgeMedtouchCambridge
MedtouchCambridgeMITCambridge
MITCambridgeModo LabsCambridge
Modo LabsCambridgeMotus (crs)Boston
Motus (crs)BostonNamemediaWaltham
NamemediaWalthamNanigansBoston
NanigansBostonNortheastern UniversityBoston
Northeastern UniversityBostonNorthpoint DigitalBoston
Northpoint DigitalBostonNutraclickBoston
NutraclickBostonPegasystemsCambridge
PegasystemsCambridgePlacesterBoston
PlacesterBostonPolar DesignWoburn
Polar DesignWoburnSevone, Inc.Boston
Sevone, Inc.BostonSilverskyBoston
SilverskyBostonSmartertravel.comBoston
Smartertravel.comBostonSource Of Future Technology, Inc.Cambridge
Source Of Future Technology, Inc.CambridgeStudypointBoston
StudypointBostonSurfmerchants LLCBoston
Surfmerchants LLCBostonTatto MediaBoston
Tatto MediaBostonTufts UniversityBoston
Tufts UniversityBostonUmass BostonBoston
Umass BostonBostonUnitrendsBurlington
UnitrendsBurlingtonWayfairBoston
WayfairBostonZipcarBoston

PHP: Using Gearman for a MapReduce inspired workflow

Over the last few weeks we’ve been utilizing Gearman to help us do some realtime stream processing. In production, what we’ve basically been doing is reading messages off an Amazon Kinesis stream, creating jobs in Gearman for anything that’s computationally expensive, and then gathering up the processed data for a batched insert into Amazon Redshift on a Gearman job as well. Conceptually, this workflow is reasonably similar to how MapReduce works where a series of input jobs is transformed by “mappers” and then results are collected in a “reduce” step.

From a practical point of view, using Gearman like this offers some interesting benefits:

  • Adding additional “map” capacity is relatively straightforward since you can just add additional machines that connect to the Gearman server.
  • Developing and testing the “map” and “reduce” functionality is easy since nothing is shared and you can run the code directly, independently of Gearman.
  • In our experience so far, the Gearman server can handle a high volume of jobs/minute - we’ve pushed ~300/sec without a problem.
  • Since Gearman clients exist for dozens of languages, you could write different pieces of the system in whatever language fits best.

Overview

OK, so how does all of this actually work. For the purposes of a demonstration, lets assume you’ve been tasked with scraping the META keywords and descriptions from a few hundred thousand sites and counting up word frequencies across all the sites. Assuming you were doing this in straight PHP, you’d end up with code that looks something like this.

The problem is that since you’re making the requests sequentially, scraping a significant number of URLs is going to take an intractable amount of time. What we really want to do is fetch the URLs in parallel, extract the META keywords, and then combine all that data in a single data structure.

To keep the amount of code down, I used the Symfony2 Console component, Guzzle and Monolog to provide infastructure around the project. Walking through the files of interest:

  • GearmanCommand.php: Command to execute either the “node” or the “master” Gearman workers.
  • StartScrapeCommand.php: Command to create the Gearman jobs to start the scrapers
  • Master.php: The code to gather up all the extracted keywords and maintain a running count.
  • Node.php: Worker code to extract the meta keywords from a given URL

Setup

Taking this for a spin is straightforward enough. Fire up an Ubuntu EC2 and then run the following:

ubuntu@ip-10-0-0-152:~$ sudo apt-get install git php5-cli php5-curl php5-gearman gearman supervisor
ubuntu@ip-10-0-0-152:~$ git clone https://github.com/adatta02/gearman_mapreduce.git
ubuntu@ip-10-0-0-152:~$ cd gearman_mapreduce/
ubuntu@ip-10-0-0-152:~/gearman_mapreduce$ curl -sS https://getcomposer.org/installer | php
ubuntu@ip-10-0-0-152:~/gearman_mapreduce$ php composer.phar install --prefer-dist

OK, now that everything is setup lets run the normal PHP implementation.

ubuntu@ip-10-0-0-21:~/gearman_mapreduce$ php bin/application.php setfive:no-gearman-scraper 100sites.txt
[2014-10-14 00:54:56] gearman.INFO: Total time: 55 [] []

Looks like about 10-12 seconds to process 100 URLs. Not terrible but assuming linear growth that means processing 100,000 URLs would take almost 2.5 hours which is a bit painful. You can verify it worked by looking at the “bin/nogearman_keyword_results.json” file.

Now, lets look at the Gearman version. Running the Gearman version is straightforward, just run the following:

ubuntu@ip-10-0-0-152:~/gearman_mapreduce$ php bin/application.php setfive:start-scraper 100sites.txt
ubuntu@ip-10-0-0-152:~/gearman_mapreduce$ php bin/application.php setfive:gearman worker &> /dev/null &
ubuntu@ip-10-0-0-152:~/gearman_mapreduce$ php bin/application.php setfive:gearman master
[2014-10-14 01:44:46] gearman.INFO: Zero jobs left. Total time: 44 [] []

You’ll eventually get an output from the “master” when it finishes with the total elapsed time. It’ll probably come in somewhere around 15ish seconds again because we’re still just using a single process to fetch the URLs.

Party in parallel

But now here’s where things get interesting, we can start adding multiple “worker” processes to do some of the computation in parallel. In my experience, the easiest way to handle this is using Supervisor since it makes starting and stopping groups of processes easy and also handles collecting their output. Run the following to copy the config file, restart supervisor, and verify the workers are running:

ubuntu@ip-10-0-0-152:~/gearman_mapreduce$ sudo cp bin/workers.conf /etc/supervisor/conf.d/
ubuntu@ip-10-0-0-152:~/gearman_mapreduce$ sudo service supervisor restart
ubuntu@ip-10-0-0-152:~/gearman_mapreduce$ ps ax | grep setfive:gearman | wc -l
6

And now, you’ll want to run “application.php setfive:gearman master” in one terminal and in another run “php setfive:start-scraper 100sites.txt” to kick off the jobs.

ubuntu@ip-10-0-0-152:~/gearman_mapreduce$ php bin/application.php setfive:gearman master
[2014-10-14 02:00:36] gearman.INFO: Registering Setfive\Gearman\Master::countKeywords [] []
[2014-10-14 02:00:52] gearman.INFO: Zero jobs left. Total time: 11 [] []

Boom! Much faster. We’re still only doing 100 URLs so the effect of processing in parallel isn’t that dramatic. Again, you can check out the results by looking at “bin/keyword_results.json”.

The effects of using multiple workers will be more apparent when you’ve got a larger number of URLs to scrape. Inside the “bin” directory there’s a file named “quantcast_site_lists.tar.gz” which has site lists of different sizes up to the full 1 million from Quantcast.

I ran a some tests on the lists using different numbers of workers and the results are below.

0 Workers10 Workers25 Workers
100 URLs12 sec.12 sec.5 sec.
1000 URLs170 sec.34 sec.33 sec.
5000 URLs1174 sec.195 sec.183 sec.
10000 URLs2743 sec.445 sec.424 sec.

One thing to note, is if you run:

ubuntu@ip-10-0-0-21:~/gearman_mapreduce$ watch gearadmin --status

And notice that “processUrl” has zero jobs but there’s a lot waiting for “countKeywords”, you’re actually saturating the “reducer” and adding additional worker nodes in Supervisor isn’t going to increase your speed. Testing on a m3.small, I was seeing this happen with 25 workers.

Another powerful feature of Gearman is that it makes running jobs on remote hosts really easy. To add a “remote” to the job server, you’d just need to start a second machine, update the IP address in Base.php, and user the same Supervisor config to start a group of workers. They’d automatically register to your Gearman server and start processing jobs.

Anyway, as always questions and comments appreciated and all the code is on GitHub.