As we continue to expand in 2015 we’re looking to add another developer to our team.  Currently we’re seeking a junior level engineer to join us!  A few attributes of a person that we’re looking for:

1. 1-2 years of experience with MVC based frameworks (we use Symfony2)
2. 1-2 years of real world experience
3. Comfortable talking directly with clients, no account managers here!
4. Works well in a team environment, but also self-managed.

A few of the perks:

1. Flexible hours
2. 100% paid health care (PPO)
3. 401(k) with matching
4. Company outings

For some more detailed information on the job please visit the posting.  If you are, or know, a developer who is looking for a new opportunity lets connect!

As promised, here’s the follow up on my previous post Javascript: Using Canvas to cut an area of an image where we looked at how to use Canvas to cut a mask out of an image. To quickly recap, in the last post we looked at how to crop a patterned mask out of an image using a HTML5 Canvas. Using this technique, you’d be able to provide an image that looks something like:

So how do you go about making a puzzle? You can see the end result at HTML5 Canvas Puzzle and the code is online at https://github.com/Setfive/setfive.github.com/tree/master/canvas_puzzle.

As it turns out generating an arbitrary puzzle programatically is reasonably complicated. The best explanation I could find on how to accomplish this is at https://www.allegro.cc/forums/thread/586750/603411#target. Conceptually, the process looks straightforward enough and you could probably manually do it on a whiteboard. Unfortunately, the issue I ran into with this approach is that drawing bezier curves and splines programmatically on a Canvas is a bit involved. I also don’t have a background in vector graphics so I was getting stuck in the weeds drawing lines.

Discounting generating the puzzle entirely on the fly, an alternative approach would be to use a fixed set of available pieces and then “fill in” a grid depending on how large the image area is. Conceptually, the idea is to construct a closed grid of pieces where some number of the pieces can be repeated and then repeat those pieces as needed to cover the target image. The templated pieces I used are in /puzzle_pieces/.

Technically, I decided to use fabric.js to facilitate Canvas interaction along with lodash.js and of course the ubituqous jQuery.

Walking through the code, the steps to build a puzzle are fairly straightforward:

1. Load images: The first step is to load all the template images and target image so that they’re available to use on a Canvas. Since jQuery is available, one approach is to create a deferred for each image, resolve it as the image loads, and use $.when to wait for all of the images to load. See here for example.
2. Build pieces grid: Next you’ll need to figure out how many repeated pieces you need to fill into the grid. One issue here is that since the puzzles need to fit snuggly the image dimensions of a given piece won’t be what you need to use to calculate the grid. Because of this, I ended up with a bit of goofy code for this.
3. Create image masks: Once you have the number of pieces to create you’ll need to cut masks for each piece out of the source image and create fabric.js objects for them. See copyImageChunk.
4. Place masks: Placing the “pieces” is also complicated because of the dimension issue above. See kludgy code.
5. Shuffle and track movements: Finally, you just need to shuffle the positions of the images and then track their movement to report a “correct” position.

And that’s about it. One other “trick” is that you can use Window.requestAnimationFrame to avoid locking the UI when you’re creating the masked images since it’s a compute intensive task.

Anyway, as always questions and comments welcome.

Recently we’ve been working with one of our clients to build application for use with AppNexus.  We were faced with a challenge which required a bunch of different technologies to all come together and work together.  Below I’ll try to list out how we approached it and what additional challenges we faced.

First came the obvious challenge:  How to handle at least 25,000 requests per second.  Our usual language of choice is PHP and knew it was not a good candidate for the project.  Instead we wanted to do some benchmarks on a number of other other languages and frameworks.  We looked at Rusty/Nginx/Lua, Go, Scala, and Java.  After some testing it appeared that Java was the best bet for us.  We initially loaded up Jetty.  We knew that this had a bit more baked in than we needed, but it was also the quickest way to get up and running and could be migrated away from fairly easily.    The idea overall was to keep the parsing of the request logic separate from the business logic.  In our initial tests we were able to get around 20,000 requests a second using Jetty, which was good, but we wanted better.

Jetty was great at breaking down the incoming HTTP requests to easily work with, it even provided an out of the box general statistics package.  However, we didn’t need much heavy lifting on the HTTP side, what we were building required very little complexity on with regards to HTTP protocol.   Jetty in the end was spending too many CPU cycles for what we needed.  We looked to Netty next.

Netty out of the box is not as friendly as Jetty as it is much lower level.   That said, it wasn’t too much work to get Netty up and running responding to HTTP request.  We ported over most of the business logic from our Jetty code and were off to the races.  We did have to add our own statistics layer as Netty didn’t have an embedded one for what we were looking for.  After some fine tuning with Netty we were able to start to handle over 40,000 requests per second.  This part of the puzzle was solved.

On our DB side we had heard great things about Aerospike in terms of performance and some of its features.  We ended up using this on the backend.  When we query Aerospike we have the timeout set at 3ms.  We’ll get around one or two request timeouts per second, or about 0.0025% of the time we’ll timeout, not too shabby. One of the nice features of Aerospike is the XDR function of the enterprise version.  With this we can have multiple Aerospike clusters which all stay in sync from a master cluster.  This lets us load our data onto one machine, which isn’t handling all the requests, and then it is replicated to the machines which are handling all the requests.

All in all we’ve had a great experience with the Netty and Aerospike integration.  We’re able to consistently handle around 40,000 requests a second with the average response time (including network time) of 4ms.

This simple tutorial will show you how to create a PhantomJS script that will scrape the state/population html table data from http://www.ipl.org/div/stateknow/popchart.html and output it in a PHP application. For those of you who don’t know about PhantomJS, it’s basically a headless WebKit scriptable with a JavaScript API.

Prerequisites:

• PhantomJS (for this tutorial we’ll be using Linux for this particular example) http://phantomjs.org/download.html
• PHP

1. Create the PhantomJS Script

The first step is to create a script that will be executed by PhantomJS. This script will do the following:

• Take in a JSON “configuration” object with the site URL and a CSS selector of the HTML element that contains the target data
• Load up the page based on the Site URL from the JSON configuration object
• Include jQuery on the page (so we can use it even if the target site doesn’t have it!)
• Use jQuery and CSS selector from configuration object to find and alert the html of the target element. You’ll notice on line 37 that we wrap the target element in a paragraph tag then traverse to it in order to pull the entire table html.
• We can save this file as ‘phantomJsBlogExample.js’
• One thing to note is that on line 24 below we set a timeout inside the evaluate function to allow for the page to fully load before we call the pullHtmlString function. To learn more about the ins and outs of PhantomJS functions read here http://phantomjs.org/documentation/

var page = require('webpage').create();

page.onError = function (msg, trace) {

    phantom.exit();

};

page.onAlert = function( msg ) {

    console.log( msg );

    if( msg == "EXIT" ){
        phantom.exit();
    }
};

page.open(config.url, function(status) {

    page.includeJs('https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js', function() {

        page.evaluate(function(config){

            window.setTimeout(function(){
                setInterval(function(){
                    pullHtmlString(config);
                }, 2000);
            }, 1);

        }, config);
    });

});

function pullHtmlString(config){

    alert($(config.selector).wrap('<p/>').parent().html());

    alert( "EXIT" );

}

2. Create PHP function to run PhantomJS script and convert output into a SimpleXmlElement Object

Next, we want to create a PHP function that actually executes the above script and converts the html to a SimpleXmlElement object.

• On line 3 below you’ll construct a “configuration” object that we’ll pass into the PhantomJS script above that will contain the site url and CSS selector
• Next on line 10 we’ll actually read in the base PhantomJs Script we created in step 1. Notice that we actually make a copy of the script so that we leave the base script intact. This becomes important if you are executing this multiple times in production using different site urls each time.
• On line 20 we prepend the configuration object onto the copied version of the phantomJS script, make sure you json_encode this so it’s inserted as a proper json object.
• Next on line 29 we execute the phantomJs script using the PHP exec function and save the output into an $output array. Each time the PhantomJS script alerts a string, it’s added as an element in this array. Alerted html strings will split out as one line per element in the array. After we get the output from the script we can go ahead and delete the copied version of the script.
• Starting on line 38, we clean up the $output array a bit, for example when we initially inject jQuery in PhantomJS a line is alerted into the output array which we do not want as it doesn’t represent the actual html data we are scraping. Similarly, want to remove the last element of the $output array where we alert (‘EXIT’) to end the script.
• Now that it’s cleaned up, we have an array of individual html strings representing our target data. We’ll want to remove the whitespace and also join all the elements into one big html string to use for constructing a SimpleXmlElement on line 49.

 public function pullXmlObjBlogExample($siteUrl,$cssSelector){

        //create configuration object containing jquery selector and target site url to pass to the phantom script

        $config = array(
            "selector"=>$cssSelector,
            "url"=>$siteUrl
        );

        //read in the base phantom script and create a copy of it so we don't mess with the original base script

        $templateScript = "phantomJsBlogExample.js";
        $templateFileCopy = "phantomJsBlogExample-copy-".time().".js";

        if (!copy($templateScript, $templateFileCopy)) {
            echo "failed to copy $templateFileCopy";
            return false;
        }

        //Prepend configuration object onto script

        $configObj = file_get_contents($templateFileCopy);
        $configObj = 'var config = ' . json_encode($config,JSON_UNESCAPED_SLASHES). ';' . "\n" . $configObj;

        file_put_contents($templateFileCopy,$configObj);

        //Run the phantom script with php exec function, redirect output of script to an $output array;

        echo exec("phantomjs $templateFileCopy 2>&1",$output);

        //delete the copied version of the phantom script as we don't need it anymore

        if ( !unlink( $templateFileCopy ) ) {
            echo "failed to delete $templateFileCopy";
            return false;
        }

        // The first element of the output will be message about adding jquery and the last element will be the 'EXIT' message from the script,
        // lets remove those so all we have is the html lines

        array_shift($output);
        array_pop($output);

        //remove any whitespace from the array elements and join all the html lines into one string of all the html

        $output= array_map('trim', $output);
        $output = join("",$output);

        //construct an XML element from the html string

        $xmlObj = new \SimpleXMLElement($output);

        return $xmlObj;
    }

3. Call the function and iterate through the SimpleXmlElement Object to get to the table data

• Call the function from step 2 making sure to pass in the target site url and CSS selector
• Now that we have the SimpleXmlObject on line 7 we’ll want to iterate through the rows of the table body and pull out the state name and population table cells. It may help to var_dump the entire SimpleXmlObject to get a sense for what the structure looks like.
• For purposes of this example we’ll just echo out the state name and population but you could really do anything you wanted with the data at this point (i.e., persist to database etc.)

private function scrapePopulationsByState(){

        $cssSelector = "table.sk_popcharttable";

        $siteUrl = "http://www.ipl.org/div/stateknow/popchart.html";

        $tableXmlObject = pullXmlObjBlogExample($siteUrl,$cssSelector);

        $cnt = 0;

        foreach($tableXmlObject->tbody->tr as $tableRow){

            //the first two rows are the header and "All United States" rows so disregard

            if($cnt++ < 2)
                continue;

            //grab the state and population from the corresponding table cell of the row and output!

            $state = (string) $tableRow->td[1]->a;

            $population = (string) $tableRow->td[2];

            echo $state . " has a population of " . $population . "\n";

        }

}

4. Final Output

Finally, running the function from step 3 should result in something like this.

California has a population of 37,253,956
Texas has a population of 25,145,561
New York has a population of 19,378,102
Florida has a population of 18,801,310
Illinois has a population of 12,830,632
Pennsylvania has a population of 12,702,379
Ohio has a population of 11,536,504
Michigan has a population of 9,883,640
Georgia has a population of 9,687,653
North Carolina has a population of 9,535,483
New Jersey has a population of 8,791,894
Virginia has a population of 8,001,024
Washington has a population of 6,724,540
Massachusetts has a population of 6,547,629
Indiana has a population of 6,483,802
Arizona has a population of 6,392,017
Tennessee has a population of 6,346,105
Missouri has a population of 5,988,927
Maryland has a population of 5,773,552
Wisconsin has a population of 5,686,986
Minnesota has a population of 5,303,925
Colorado has a population of 5,029,196
Alabama has a population of 4,779,736
South Carolina has a population of 4,625,364
Louisiana has a population of 4,533,372
Kentucky has a population of 4,339,367
Oregon has a population of 3,831,074
Oklahoma has a population of 3,751,351
Connecticut has a population of 3,574,097
Iowa has a population of 3,046,355
Mississippi has a population of 2,967,297
Arkansas has a population of 2,915,918
Kansas has a population of 2,853,118
Utah has a population of 2,763,885
Nevada has a population of 2,700,551
New Mexico has a population of 2,059,179
West Virginia has a population of 1,852,994
Nebraska has a population of 1,826,341
Idaho has a population of 1,567,582
Hawaii has a population of 1,360,301
Maine has a population of 1,328,361
New Hampshire has a population of 1,316,470
Rhode Island has a population of 1,052,567
Montana has a population of 989,415
Delaware has a population of 897,934
South Dakota has a population of 814,180
Alaska has a population of 710,231
North Dakota has a population of 672,591
Vermont has a population of 625,741
Washington, D. C. has a population of 601,723
Wyoming has a population of 563,626

Over the few weeks I’ve been working on a Canvas based side project (more on that soon) that involved cutting a mask out of a source image and placing it on a Canvas. In Photoshop parlance, this would be similar to creating a clipping mask and then using it to extract a path from the image into a new layer. So visually, we’re looking to achieve something similar to:

At face value, it looks like doing this with Canvas is pretty straightforward using the getImageData function. Unfortunately, if you look at the parameters that function accepts it’ll only support slicing out rectangular areas which isn’t what we’re looking to do. Luckily, if you look a bit further in the docs it turns out Canvas supports setting globalCompositeOperation which allows you to control how image data is drawn onto the canvas. The idea is to draw the mask on a canvas, turn on the “source-in” setting, and then draw on the image that you want to generate the slice off. The big thing to note here is that putImageData isn’t effected by the globalCompositeOperation setting so you have to use drawImage to draw the mask and image data.

So concretely how do you do this? Well check it out:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title></title>

    <link href='http://fonts.googleapis.com/css?family=Open+Sans:400,700,800' rel='stylesheet' type='text/css'>
    <link href='http://fonts.googleapis.com/css?family=Lato:400,700,900' rel='stylesheet' type='text/css'>
    
    <link href="bootstrap/css/bootstrap.min.css" rel="stylesheet">
        
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
    <script src="bootstrap/js/bootstrap.min.js"></script>
    <script src="underscore-min.js"></script>    
    <script src="fabric.min.js"></script>
    
    <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
    <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
    <!--[if lt IE 9]>
      <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
      <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
    <![endif]-->    
  </head>
  <body>
  
  <canvas id="target" width="800px" height="400px" style="margin: auto"></canvas>
  
  <script>
  
  $(document).ready(function(){
      
      var mask = new Image(), bg = new Image();
      var mDf = $.Deferred(), bgDf = $.Deferred();
            
      mask.src = "grass_overlay.png";
      bg.src = "grass_texture.png";
      
      mask.addEventListener("load", function(){ mDf.resolve(this); });
      bg.addEventListener("load", function(){ bgDf.resolve(this); });
      
      var canvas = document.createElement("canvas"), ctx = canvas.getContext('2d');
      var target = document.getElementById("target"), targetCtx = target.getContext('2d');
      
      
      $.when(mDf, bgDf).done(function(){
          var l = (bg.width / 2) - (mask.width / 2), t = (bg.height / 2) - (mask.height / 2);
          
          canvas.width = bg.width;
          canvas.height = bg.height;
          
          ctx.drawImage(mask, l, t);
          ctx.globalCompositeOperation = "source-in";
          ctx.drawImage(bg, 0, 0);
                 
          var imageData = ctx.getImageData(l, t, mask.width, mask.height);                                       
          targetCtx.putImageData(imageData, 50, 50);                         
      });
      
  });
  
  </script>
  
  </body>
</html>

The code is running over at http://symf.setfive.com/canvas_puzzle/grass.html if you want to see it in action.

Anyway, happy canvasing!