Recently I was working on a project where we had a page which loads tons of data from numerous sources. I decided after a while that we wanted to AJAX each section of data so that the page would load a bit quicker. After splitting up the requests and sending them asyncronously, there was little improvement. I thought at first it may be due to the fact we were pinging a single API for most of the data multiple times, that wasn’t it. Maybe it was a browser limit? Nope was still far below the 6 requests most allow. I setup xdebug and kcachegrind and to my surprise it was the session_start() that was taking the most time on the requests.

I looked around the web for a while trying to figure out what in the world was going on. It turns out that PHP’s default session_start will block future session_starts for the same session until the session is closed. This is because the default method uses a file on the filesystem which it locks until you close it. If you want more information on this and how to close it you can read a bit more here.

We switched over to database based sessions and it fixed it. In symfony 1.4 the default session storage uses the file system, however switching over to sfPDOSessionStorage is very easy and quick.

Earlier this week I was completely dumbfounded by a PHP script that launched a Java app that seemed to work fine when it was run from the command line but kept failing when it was run from a cron.

The Java app in question was “ec2-describe-group” out of the Amazon EC2 API Tools package.  Basically, the ec2-describe-group tool hits the EC2 API and returns information about your account’s currently configured security groups.

The issue I was having was that when the PHP script was launched from a cron ec2-describe-group would keep returning an empty string, but when the script was launched from the CLI ec2-describe-group behaved normally.

After some poking around, I found this StackOverflow post which points out that most the environment variables your shell has aren’t available in a cronjob.

With that in mind, I tried adding JAVA_HOME as well as EC2_HOME to my crontab. Doing this is pretty straight forward, just add these two lines above any of your scheduled jobs:

EC2_HOME=/opt/ec2-api-tools-1.3.36506
JAVA_HOME=/etc/java-config-2/current-system-vm

Unfortunately, this still didn’t resolve the issue. On a whim, I decided to check what type of file ec2-describe-group actually is and discovered that its a Bash script not a Java JAR. Looking at the Bash, the file is actually just executing “EC2_HOME/bin/ec2-cmd DescribeGroups” but it utilizes other environment variables that my cron didn’t have.

For simplicity’s sake, I decided to just switch the PHP script to run ec2-cmd directly and finally everything started working as expected.

Recently one of our clients decided to white label their product.  With that we had to setup the server to use multiple IPs as the application requires you communicate over SSL and we needed a SSL per domain.  We did not want to buy a UCC(a multiple domain)  SSL certificate as right now it wasn’t required for the small number of white labels. After we added the additional IP we had the issue that the application which connects to off site MySQL servers, was sometimes going over the new IPs and then getting denied accessed.

We knew the solution was with iptables so after some digging and testing, we came up with the following command.  This command we use will redirect all traffic that is not over port 443 (in this example) to go out over the ‘YYY.YYY.YYY.YYY’ address that is about to go out over the XXX.XXX.XXX.XXX ip.

iptables -t nat -A POSTROUTING -p tcp ! --dport 443 -s XXX.XXX.XXX.XXX -j SNAT --to-source YYY.YYY.YYY.YYY

We didn’t see any examples of this clearly defined (after a quick google that is), on the web, so hopefully this will save you time from having to read through the iptables documentation.

I was working on an application earlier today that required allowing a user to upload a large file (several hundred MB) which would eventually be stored on Amazon S3. After reviewing the requirements, I realized it made sense to just upload the file directly to S3 instead of having to first stage the file on a server and then use PHP to push the file to S3.

Amazon has a nice walk through of using a plain HTML form to upload a file directly to S3 here.

I had all ready been using SWFUpload to upload files to the server so I decided to look into using it to uploading directly to S3. After some head banging, I finally got it to work – here’s the quick n dirty.

1. Download SWFUpload 2.5
2. Get SWFUpload ready to use in your project. Copy the SWF file somewhere accessible and include their swfupload.js Javascript file. More info here
3. Setup an S3 bucket. You’ll need to set the policy to allow uploads from your own user (its the default).
4. Place a crossdomain.xml file in the root of your S3 bucket. This file “authorizes” flash player to upload files into this host. The content of the file is below.
5. Initialize the SWFUpload object (example below).
6. Before beginning the upload, you need to set the appropriate postParams in the SWFUpload object. This is really the “magic” of this process. Example is below.
7. Start the upload with startUpload()

Thats it! It’s pretty straight forward once you have things going. As an FYI, you can put SWFUpload into “debug” mode by adding debug: true as a property to the initialization object. You can also debug the responses from Amazon by using a packet sniffer like Wireshark.

crossdomain.xml

You probably want to make this file a little less permissive. More details here. Also note, there are differences in the implementation of the file between various versions of Flash player.

Initialize SWFUpload

        var swfu = new SWFUpload(
                       {  flash_url: "/assets/swfupload.swf",
                          flash9_url: "/assets/swfupload_fp9.swf",
                          file_size_limit: "1000 MB",
                          file_types: "*.*",
                          debug: false,
                          upload_url: "http://your-bucket.s3.amazonaws.com",
                          button_placeholder_id : "SWFUploadButton",
                          button_image_url : "/assets/select_filesbtn.png",
                          button_width: '112',
                          button_height: '33',
                          button_cursor : SWFUpload.CURSOR.HAND,
                          http_success : [201, 303, 200], /* Amazon returns a 303 on success */
                          file_post_name: "file", /* Amazon expects the file data to be in a input named "file"

                          file_queued_handler: function(f){

                            // track the filenames so you can upload them later
                            cachedUploadFiles[ f.index ] = f.name;
                          },
                          upload_complete_handler: function(e){ uploadSWFFile( ); },
                          upload_start_handler: function(e){
                                  // reset the progress bar
                        	  $("#progressBar").progressbar( 'value', 0 );
                          },
                          upload_error_handler: function(e){

                          },
                          upload_progress_handler: function(f, c, t){
                             // update the progress bar as the process continues
                             $("#progressBar").progressbar( 'value', Math.ceil( ( c/t ) * 100 ) );
                          }
        });

Set SWFUpload postParams

The HMAC signature MUST be calculated on the server because it uses your S3 secret. You MUST keep that value secret in order to maintain the security of your S3 buckets. I’m using Don Schonknecht’s S3 PHP library to calculate the HMAC signatures but you could just as easily do it in straight PHP.

/* In PHP */
$encodedPolicy = json_encode( array(
              "expiration" => "2011-4-22T13:54:23.000Z",
              "conditions" => array(
                  0 => array( "acl" => "public-read" ),
                  1 => array( "bucket" => "your-bucket" ),
                  2 => array( "x-amz-meta-sig" => 'some meta signature to ensure authentic requests'),
                  3 => array( "redirect" => $'URL to redirect a success request (its doesnt matter)' ),
                  4 => array( "key" => "the S3 key for the file (the S3 filename)" ),
                  5 => array( "Filename" => "The original filename of the file. THIS IS IMPORTANT." )
              ),
            )
);

$encodedPolicy = base64_encode( $encodedPolicy );
$s3 = new S3( sfConfig::get("app_amazon_s3_id"), sfConfig::get("app_amazon_s3_secret") );
list($dist, $hmacSignature) = explode(":", $s3->__getSignature( $encodedPolicy ));

/* END PHP */

            var swfConfig = {
                    'AWSAccessKeyId': 'your amazon ID',
                    'acl': 'public-read',
                    'key': 'the S3 key for the file (the S3 filename)',
                    'policy': '<?php echo $encodedPolicy?>',
                    'signature': '<?php echo $signature'?>,
                    'redirect': 'URL to redirect a success request (its doesnt matter)',
                    'x-amz-meta-sig': 'some meta signature to ensure authentic requests',
            };

            // this line sets the post params so that SWFUpload will send the additional fields when it uploads the file.
            $.swfu.setPostParams( swfConfig );

So we decided to use the Google Calendar API in one of our applications to allow users to easily view and export events from outside the app. In general, the API was working well – I was using the Zend library to interact with Google and things seemed fine.

That was until I tried to embed the calendar using Google’s iframe embed code. For some reason, events weren’t showing up in the embeded iframe calendar even though they were showing up in the actual calendar on calendar.google.com. Even stranger, the events were present in a JSON object on the embeded page and they were showing up in the RSS feed for the calendar.

After literally days of debugging and experimenting I finally found out the culprit.

For some reason, events created via the API that start and end at exactly the same time – say a start date of 08-05-2009 10:00:00 and an end date of 08-05-2009 10:00:00 don’t render on the embeded iframe calendar.

What is even more bizarre is that if you create an event via the web interface that starts and ends at the same time, it will render correctly on an embeded calendar.

Anyway, that was weird. All the events without explicit start and end times now last a grand total of one minute.

PS. Kudos to Daum for finding a constant for PHP’s date() function to generate RFC3339 timestamps.

Use like so:

  $date = date(DATE_RFC3339, $timestamp);

To get back a valid RFC3339 for the Google Calendar API.

Recently we were working on a project in which we needed to switch from forms auto-formatting themselves ( <?php echo $form;?>) to allow for much more customization in the output.   While there is a Symfony Forms for Designers chapter in the forms documentation, it doesn’t help much for iterating over a form object and customizing the output.  There is a simple foreach($form as $field) that will iterate over every field in the form.  The problem with this is when you have embedded forms and you want to do something different with the formatting on them.  This will iterate over all the fields and you will not know when the field is the embedded form or not.  So we came up with the following which works:

foreach($form as $field)
{
  // If this is true, then that means its an embedded form.
  if(get_class($field)=='sfFormFieldSchema')
  {
    foreach($field as $f)
    {
      // Don't want to see hidden field labels
      if(!$f->isHidden())
        echo $f->renderLabel()." ";
      echo $f->renderError();
      echo $f->render();
    }
  }
  else
  {
    if(!$field->isHidden())
        echo $field->renderLabel()." ";
    echo $field->renderError();
    echo $field->render();
  }
}

This will work fine for a form with as many as single embedded forms.  It allows you to easily use the same view for multiple forms and be able to customize their embedded forms easily.  If you have only one form that you will be doing this with, and are worried about performance we recommend then not using a foreach loop and doing it by hand, this will save you on performance as you will not have as many if statements in each iteration.

All other options you pass are added to the form as attributes.

Today I want to take the chance to stress the importance of having someone onboard that knows some technology. Through our experience we’ve often had clients who were mislead or taken advantage of by other technology firms. Whether it is being given poor advice or being overcharged for simple tasks, we’ve seen it quite a bit. In many of these situations a person with some technological knowledge would have saved them money and time.

What can a person with technical knowledge do?

• Know what questions to ask – “Do you provide documentation for your code? Do you use a framework? Is your code MVC compliant? etc.”
• Know when you are being taken advantage of – We’ve seen in the past many clients purchased servers and functionality they just don’t need. A technical person can make sure that you aren’t purchasing five servers when one will do, that you aren’t overpaying for simple hosting, etc.
• To help choose your development firm. It is very difficult to tell whether a firms code is good. If you do not have knowledge in the area, good code and bad code appear to be the same. Choosing the development firm for your project can make or break the project. If the firm chosen produces messy, inefficient code, it dooms your product before it is even launched.
• To monitor code quality. Once you’ve picked your development firm, it is important to check in now and then to make sure the quality of the code is of the caliber you are paying for. It is crucial to catch poor code in the beginning, to make sure the firm doesn’t continue to use the same coding style.

What if you don’t want to have a technical person on staff full time? Hire a consulting firm. Many firms provide assistance with choosing your development team and to monitor most of the technical aspects of the product. We offer these services.

Moral: Having technical knowledge onboard prevents your company from being taken advantage of and will often pay for itself in saved time and a quality delivered product.