Category: PHP

I’ll be the first to criticize PHP from a language design perspective. But, PHP does possess compelling aspects that have contributed to its popularity today.

• “Hit refresh workflow” - When using PHP, you can make a change to a source file, and hit refresh, and your changes are immediately available. This function prevents compiling and transpiling, fiddling around with “hot reloads,” and bouncing servers. Having a tight feedback loop is hugely productive when you’re building a CRUD app. Interestingly, Play Framework is now touting this as a feature, Build Modern & Scalable Web Apps with Java and Scala
• Isolated HTTP request handling - If you’re deploying PHP via mod_php or php-fpm, every HTTP request is handled effectively through a separate PHP process. In practice, this means your code can’t store any state between requests, or accidentally retain information from a previous request. Typically, this makes PHP apps easier to scale at the app level since the stateless nature enables horizontal scaling without any app changes.
• Composer with Packagist - Composer and Packagist work pretty well together for dependency management. Composer is easier to work with than Maven, arguably more “sane” than npm, and usually does a good job at staying out of the way. Packagist has a critical mass of useful, popular packages and works well with Composer.

When Amazon Web Services rolled out their version 4 signature we started seeing sporadic errors on a few projects when we created pre-authenticated link to S3 resources with a relative timestamp. Trying to track down the errors wasn’t easy. It seemed that it would occur rarely while executing the same exact code. Our code was simply to get a pre-authenticated URL that would expire in 7 days, the max duration V4 signatures are allowed to be valid. The error we’d get was “The expiration date of a signature version 4 presigned URL must be less than one week”. Weird, we kept passing in “7 days” as the expiration time. After the error occurred a couple of times over a few weeks I decided to look into it.

The code throwing the error was located right in the SignatureV4 class. The error is thrown when the end timestamp minus the start timestamp for the signature was greater than a week. Looking through the way the timestamps were generated it went something like this:

1. Generate the start timestamp as current time for the signature assuming one is not passed.
2. Do a few other quick things not related to this problem.
3. Do a check to insure that the end minus start timestamp is less than a week in seconds.

So a rough example with straight PHP could of the above steps for a ‘7 days’ expiration would be as follows:

<?php
// Generate the start timestamp
$start = time();

// Do other stuff....

// If the end timestamp is over 7 weeks in seconds throw an error
if( strtotime('+7 days') - $start > 86400) {
  throw new \Exception('must be less than 7 weeks');
}

Straight forward enough, right? the problem lies when a second “rolls” between generating the $start and the end timestamp check. For example, if you generate the $start at 2017-08-20 12:01:01.999999. Let’s say this gets assigned the timestamp of 2017-08-20 12:01:01. Then the check for the 7 weeks occurs at 2017-08-27 12:01:02.0000 it’ll throw an exception as duration between the start and end it’s actually now for 86,401 seconds total. It turns outs triggering this error is easier than you’d think. Run this script locally:

<?php
while(true){
    $start = time();
    // Sleeps below are not even necessary.
    // time_nanosleep(0,1);
    //or can use usleep(1); 
    $end = strtotime('1 week');
    if($end-$start > 604800)
    {
        echo "Failed!! ".date('Y-m-d H:i:s',$start).' vs '.date('Y-m-d H:i:s',$end).' start: '.$start.' end: '.$end;
        exit;
    }
}

That will throw an exception within a few seconds of running most likely.

After I figured out the error, the next step was to submit a issue to make sure I’m not misunderstanding how the library should be used. The simplest fix for me was to generate the end expiration timestamp before generating the start timestamp. After I made the PR, Kevin S. from AWS pointed out that while this fixed the problem, the duration still wasn’t guaranteed to always be the same for the same relative time period. For example, if you created 1000 presigned URLs all with ‘+7 days’ as the valid period, some may be 86400 in duration others may be 86399. This isn’t a huge problem, but Kevin made a great point that we could solve the problem by locking the relative timestamp for the end based on the start timestamp. After adding that to the PR it was accepted. As of release 3.32.4 the fix is now included in the SDK.

On one of our projects that I am working on I had the following problem: I needed to create an aggregate temporary table in the database from a few different queries while still using Doctrine2. I needed to aggregate the results in the database rather than memory as the result set could be very large causing the PHP process to run out of memory. The reason I wanted to still use Doctrine to get the base queries was the application passes around a QueryBuilder object to add restrictions to the query which may be defined outside of the current function, every query in the application goes through this process for security purposes.

After looking around a bit, it was clear that Doctrine did not support (and shouldn’t support) what I was trying to do. My next step was to figure out how to get an executable query from Doctrine2 without ever running it. Doctrine2 has a built in SQL logger interface which basically lets you to listen for executed queries and to see what the actual SQL and parameters were for the executed query. The problem I had was I didn’t want to actually execute the query I had built in Doctrine, I just wanted the SQL that would be executed via PDO. After digging through the code a bit further I found the routines that Doctrine used to actually build the query and parameters for PDO to execute, however, the methods were all private and internalized. I came up with the following class to take a Doctrine Query and return a SQL statement, parameters, and parameter types that can be used to execute it via PDO.

<?php
namespace Example\Doctrine\Util;

use Doctrine\ORM\Query;

class QueryUtils
{
    /**
     * @param Query $query
     * @return array An array with 3 indexes, sql the SQL statement with parameters as ?, params the ordered parameters, and paramTypes as the types each parameter is.
     */
    public static function getRunnableQueryAndParametersForQuery(Query $query)
    {
        $sql = $query->getSQL();
        $c = new \ReflectionClass('Doctrine\ORM\Query');
        $parser = $c->getProperty('_parserResult');
        $parser->setAccessible(true);
        /** @var \Doctrine\ORM\Query\ParserResult $parser */
        $parser = $parser->getValue($query);
        $resultSet = $parser->getResultSetMapping();
        
        // Change the aliases back to what was originally specified in the QueryBuilder.
        $sql = preg_replace_callback('/AS\s([a-zA-Z0-9_]+)/',function($matches) use($resultSet) {
            $ret = 'AS ';
            if($resultSet->isScalarResult($matches[1]))
                $ret.=$resultSet->getScalarAlias($matches[1]);
            else
                $ret.=$matches[1];

            return $ret;

        },$sql);

        $m = $c->getMethod('processParameterMappings');
        $m->setAccessible(true);

        list($params,$types)= $m->invoke($query,$parser->getParameterMappings());

        return ['sql' => $sql, 'params' => $params,'paramTypes' => $types];
    }

}

In the ExampleUsage.php file above I take a query builder, get the runnable query, and then insert it into my temporary table. In my circumstance I had about 3-4 of these types of statements.

If you look at the QueryUtils::getRunnableQueryAndParametersForQuery function, it does a number of things.

• First, it uses Reflection Classes to be able to access private member of the Query. This breaks a lot of programming principles and Doctrine could change the interworkings of the Query class and break this class. It’s not a good programming practice to be flipping private variables public, as generally they are private for a reason.
• Second, Doctrine aliases any alias you give it in your select. For example if you do “SELECT u.myField as my_field” Doctrine may realias that to “my_field_0”. This make it difficult if you want to read out specific columns from the query without going back through Doctrine. This class flips the aliases back to your original alias, so you can reference ‘my_field’ for example.
• Third, it returns an array of parameters and their types. The Doctrine Connection class uses these arrays to execute the query via PDO. I did not want to reimplement some of the actual parameters and types to PDO, so I opted to pass it through the Doctrine Connection class.

Overall this was the best solution I could find at the time for what I was trying to do. If I was ok with running the query first, capturing the actual SQL via an SQL Logger would have been the proper and best route to go, however I did not want to run the query.

Hope this helps if you find yourself in a similar situation!

On many of our projects we use Gearman to do background processing. One of problems with doing things in the background is that the web debug toolbar isn’t available to help with debugging problems, including queries. Normally when you want to see your queries you can look at the debug toolbar and get a runnable version of the query quickly. However, when its running in the background, you have to look at the application logs to see what the query is. The logs don’t contain a runnable format of the query, for example they may look like this:

SELECT 
  t0.username AS username1, t0.username_canonical AS username_canonical2, t0.email AS email3, 
  t0.email_canonical AS email_canonical4, t0.enabled AS enabled5, t0.salt AS salt6, t0.password AS password7, 
  t0.last_login AS last_login8, t0.locked AS locked9, t0.expired AS expired10, t0.expires_at AS expires_at11, 
  t0.confirmation_token AS confirmation_token12, t0.password_requested_at AS password_requested_at13, t0.roles AS roles14, 
  t0.credentials_expired AS credentials_expired15, t0.credentials_expire_at AS credentials_expire_at16, t0.id AS id17,
  t0.first_name AS first_name18, t0.last_name AS last_name19
FROM app_user t0 
WHERE t0.id = ? LIMIT 1 [1] []

Problem is you can’t quickly take that to your database and run it to see the results. Plugging in the parameters is easy enough, but it takes time. I decided to quickly whip up a script that will take what is in the gist above and convert it to a runnable format. I’ve posted this over at http://code.setfive.com/doctrine-query-log-converter/ . This hopefully will save you some time when you are trying to debug your background processes.

It should work with both Doctrine 1.x/symfony 1.x and Doctrine2.x/Symfony2.x. If you find any issues with it let me know.

Good luck debugging!

This simple tutorial will show you how to create a PhantomJS script that will scrape the state/population html table data from http://www.ipl.org/div/stateknow/popchart.html and output it in a PHP application. For those of you who don’t know about PhantomJS, it’s basically a headless WebKit scriptable with a JavaScript API.

Prerequisites:

• PhantomJS (for this tutorial we’ll be using Linux for this particular example) http://phantomjs.org/download.html
• PHP

1. Create the PhantomJS Script

The first step is to create a script that will be executed by PhantomJS. This script will do the following:

• Take in a JSON “configuration” object with the site URL and a CSS selector of the HTML element that contains the target data
• Load up the page based on the Site URL from the JSON configuration object
• Include jQuery on the page (so we can use it even if the target site doesn’t have it!)
• Use jQuery and CSS selector from configuration object to find and alert the html of the target element. You’ll notice on line 37 that we wrap the target element in a paragraph tag then traverse to it in order to pull the entire table html.
• We can save this file as ‘phantomJsBlogExample.js’
• One thing to note is that on line 24 below we set a timeout inside the evaluate function to allow for the page to fully load before we call the pullHtmlString function. To learn more about the ins and outs of PhantomJS functions read here http://phantomjs.org/documentation/

var page = require('webpage').create();

page.onError = function (msg, trace) {

    phantom.exit();

};

page.onAlert = function( msg ) {

    console.log( msg );

    if( msg == "EXIT" ){
        phantom.exit();
    }
};

page.open(config.url, function(status) {

    page.includeJs('https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js', function() {

        page.evaluate(function(config){

            window.setTimeout(function(){
                setInterval(function(){
                    pullHtmlString(config);
                }, 2000);
            }, 1);

        }, config);
    });

});

function pullHtmlString(config){

    alert($(config.selector).wrap('<p/>').parent().html());

    alert( "EXIT" );

}

2. Create PHP function to run PhantomJS script and convert output into a SimpleXmlElement Object

Next, we want to create a PHP function that actually executes the above script and converts the html to a SimpleXmlElement object.

• On line 3 below you’ll construct a “configuration” object that we’ll pass into the PhantomJS script above that will contain the site url and CSS selector
• Next on line 10 we’ll actually read in the base PhantomJs Script we created in step 1. Notice that we actually make a copy of the script so that we leave the base script intact. This becomes important if you are executing this multiple times in production using different site urls each time.
• On line 20 we prepend the configuration object onto the copied version of the phantomJS script, make sure you json_encode this so it’s inserted as a proper json object.
• Next on line 29 we execute the phantomJs script using the PHP exec function and save the output into an $output array. Each time the PhantomJS script alerts a string, it’s added as an element in this array. Alerted html strings will split out as one line per element in the array. After we get the output from the script we can go ahead and delete the copied version of the script.
• Starting on line 38, we clean up the $output array a bit, for example when we initially inject jQuery in PhantomJS a line is alerted into the output array which we do not want as it doesn’t represent the actual html data we are scraping. Similarly, want to remove the last element of the $output array where we alert (‘EXIT’) to end the script.
• Now that it’s cleaned up, we have an array of individual html strings representing our target data. We’ll want to remove the whitespace and also join all the elements into one big html string to use for constructing a SimpleXmlElement on line 49.

 public function pullXmlObjBlogExample($siteUrl,$cssSelector){

        //create configuration object containing jquery selector and target site url to pass to the phantom script

        $config = array(
            "selector"=>$cssSelector,
            "url"=>$siteUrl
        );

        //read in the base phantom script and create a copy of it so we don't mess with the original base script

        $templateScript = "phantomJsBlogExample.js";
        $templateFileCopy = "phantomJsBlogExample-copy-".time().".js";

        if (!copy($templateScript, $templateFileCopy)) {
            echo "failed to copy $templateFileCopy";
            return false;
        }

        //Prepend configuration object onto script

        $configObj = file_get_contents($templateFileCopy);
        $configObj = 'var config = ' . json_encode($config,JSON_UNESCAPED_SLASHES). ';' . "\n" . $configObj;

        file_put_contents($templateFileCopy,$configObj);

        //Run the phantom script with php exec function, redirect output of script to an $output array;

        echo exec("phantomjs $templateFileCopy 2>&1",$output);

        //delete the copied version of the phantom script as we don't need it anymore

        if ( !unlink( $templateFileCopy ) ) {
            echo "failed to delete $templateFileCopy";
            return false;
        }

        // The first element of the output will be message about adding jquery and the last element will be the 'EXIT' message from the script,
        // lets remove those so all we have is the html lines

        array_shift($output);
        array_pop($output);

        //remove any whitespace from the array elements and join all the html lines into one string of all the html

        $output= array_map('trim', $output);
        $output = join("",$output);

        //construct an XML element from the html string

        $xmlObj = new \SimpleXMLElement($output);

        return $xmlObj;
    }

3. Call the function and iterate through the SimpleXmlElement Object to get to the table data

• Call the function from step 2 making sure to pass in the target site url and CSS selector
• Now that we have the SimpleXmlObject on line 7 we’ll want to iterate through the rows of the table body and pull out the state name and population table cells. It may help to var_dump the entire SimpleXmlObject to get a sense for what the structure looks like.
• For purposes of this example we’ll just echo out the state name and population but you could really do anything you wanted with the data at this point (i.e., persist to database etc.)

private function scrapePopulationsByState(){

        $cssSelector = "table.sk_popcharttable";

        $siteUrl = "http://www.ipl.org/div/stateknow/popchart.html";

        $tableXmlObject = pullXmlObjBlogExample($siteUrl,$cssSelector);

        $cnt = 0;

        foreach($tableXmlObject->tbody->tr as $tableRow){

            //the first two rows are the header and "All United States" rows so disregard

            if($cnt++ < 2)
                continue;

            //grab the state and population from the corresponding table cell of the row and output!

            $state = (string) $tableRow->td[1]->a;

            $population = (string) $tableRow->td[2];

            echo $state . " has a population of " . $population . "\n";

        }

}

4. Final Output

Finally, running the function from step 3 should result in something like this.

California has a population of 37,253,956
Texas has a population of 25,145,561
New York has a population of 19,378,102
Florida has a population of 18,801,310
Illinois has a population of 12,830,632
Pennsylvania has a population of 12,702,379
Ohio has a population of 11,536,504
Michigan has a population of 9,883,640
Georgia has a population of 9,687,653
North Carolina has a population of 9,535,483
New Jersey has a population of 8,791,894
Virginia has a population of 8,001,024
Washington has a population of 6,724,540
Massachusetts has a population of 6,547,629
Indiana has a population of 6,483,802
Arizona has a population of 6,392,017
Tennessee has a population of 6,346,105
Missouri has a population of 5,988,927
Maryland has a population of 5,773,552
Wisconsin has a population of 5,686,986
Minnesota has a population of 5,303,925
Colorado has a population of 5,029,196
Alabama has a population of 4,779,736
South Carolina has a population of 4,625,364
Louisiana has a population of 4,533,372
Kentucky has a population of 4,339,367
Oregon has a population of 3,831,074
Oklahoma has a population of 3,751,351
Connecticut has a population of 3,574,097
Iowa has a population of 3,046,355
Mississippi has a population of 2,967,297
Arkansas has a population of 2,915,918
Kansas has a population of 2,853,118
Utah has a population of 2,763,885
Nevada has a population of 2,700,551
New Mexico has a population of 2,059,179
West Virginia has a population of 1,852,994
Nebraska has a population of 1,826,341
Idaho has a population of 1,567,582
Hawaii has a population of 1,360,301
Maine has a population of 1,328,361
New Hampshire has a population of 1,316,470
Rhode Island has a population of 1,052,567
Montana has a population of 989,415
Delaware has a population of 897,934
South Dakota has a population of 814,180
Alaska has a population of 710,231
North Dakota has a population of 672,591
Vermont has a population of 625,741
Washington, D. C. has a population of 601,723
Wyoming has a population of 563,626