The Java app in question was “ec2-describe-group” out of the Amazon EC2 API Tools package.  Basically, the ec2-describe-group tool hits the EC2 API and returns information about your account’s currently configured security groups.

The issue I was having was that when the PHP script was launched from a cron ec2-describe-group would keep returning an empty string, but when the script was launched from the CLI ec2-describe-group behaved normally.

After some poking around, I found this StackOverflow post which points out that most the environment variables your shell has aren’t available in a cronjob.

With that in mind, I tried adding JAVA_HOME as well as EC2_HOME to my crontab. Doing this is pretty straight forward, just add these two lines above any of your scheduled jobs:

EC2_HOME=/opt/ec2-api-tools-1.3.36506
JAVA_HOME=/etc/java-config-2/current-system-vm

Unfortunately, this still didn’t resolve the issue. On a whim, I decided to check what type of file ec2-describe-group actually is and discovered that its a Bash script not a Java JAR. Looking at the Bash, the file is actually just executing “EC2_HOME/bin/ec2-cmd DescribeGroups” but it utilizes other environment variables that my cron didn’t have.

For simplicity’s sake, I decided to just switch the PHP script to run ec2-cmd directly and finally everything started working as expected.

Amazon has a nice walk through of using a plain HTML form to upload a file directly to S3 here.

I had all ready been using SWFUpload to upload files to the server so I decided to look into using it to uploading directly to S3. After some head banging, I finally got it to work – here’s the quick n dirty.

1. Download SWFUpload 2.5
2. Get SWFUpload ready to use in your project. Copy the SWF file somewhere accessible and include their swfupload.js Javascript file. More info here
3. Setup an S3 bucket. You’ll need to set the policy to allow uploads from your own user (its the default).
4. Place a crossdomain.xml file in the root of your S3 bucket. This file “authorizes” flash player to upload files into this host. The content of the file is below.
5. Initialize the SWFUpload object (example below).
6. Before beginning the upload, you need to set the appropriate postParams in the SWFUpload object. This is really the “magic” of this process. Example is below.
7. Start the upload with startUpload()

Thats it! It’s pretty straight forward once you have things going. As an FYI, you can put SWFUpload into “debug” mode by adding debug: true as a property to the initialization object. You can also debug the responses from Amazon by using a packet sniffer like Wireshark.

crossdomain.xml

You probably want to make this file a little less permissive. More details here. Also note, there are differences in the implementation of the file between various versions of Flash player.

Initialize SWFUpload

        var swfu = new SWFUpload(
                       {  flash_url: "/assets/swfupload.swf",
                          flash9_url: "/assets/swfupload_fp9.swf",
                          file_size_limit: "1000 MB",
                          file_types: "*.*",
                          debug: false,
                          upload_url: "http://your-bucket.s3.amazonaws.com",
                          button_placeholder_id : "SWFUploadButton",
                          button_image_url : "/assets/select_filesbtn.png",
                          button_width: '112',
                          button_height: '33',
                          button_cursor : SWFUpload.CURSOR.HAND,
                          http_success : [201, 303, 200], /* Amazon returns a 303 on success */
                          file_post_name: "file", /* Amazon expects the file data to be in a input named "file"

                          file_queued_handler: function(f){

                            // track the filenames so you can upload them later
                            cachedUploadFiles[ f.index ] = f.name;
                          },
                          upload_complete_handler: function(e){ uploadSWFFile( ); },
                          upload_start_handler: function(e){
                                  // reset the progress bar
                        	  $("#progressBar").progressbar( 'value', 0 );
                          },
                          upload_error_handler: function(e){

                          },
                          upload_progress_handler: function(f, c, t){
                             // update the progress bar as the process continues
                             $("#progressBar").progressbar( 'value', Math.ceil( ( c/t ) * 100 ) );
                          }
        });

Set SWFUpload postParams

The HMAC signature MUST be calculated on the server because it uses your S3 secret. You MUST keep that value secret in order to maintain the security of your S3 buckets. I’m using Don Schonknecht’s S3 PHP library to calculate the HMAC signatures but you could just as easily do it in straight PHP.

/* In PHP */
$encodedPolicy = json_encode( array(
              "expiration" => "2011-4-22T13:54:23.000Z",
              "conditions" => array(
                  0 => array( "acl" => "public-read" ),
                  1 => array( "bucket" => "your-bucket" ),
                  2 => array( "x-amz-meta-sig" => 'some meta signature to ensure authentic requests'),
                  3 => array( "redirect" => $'URL to redirect a success request (its doesnt matter)' ),
                  4 => array( "key" => "the S3 key for the file (the S3 filename)" ),
                  5 => array( "Filename" => "The original filename of the file. THIS IS IMPORTANT." )
              ),
            )
);

$encodedPolicy = base64_encode( $encodedPolicy );
$s3 = new S3( sfConfig::get("app_amazon_s3_id"), sfConfig::get("app_amazon_s3_secret") );
list($dist, $hmacSignature) = explode(":", $s3->__getSignature( $encodedPolicy ));

/* END PHP */

            var swfConfig = {
                    'AWSAccessKeyId': 'your amazon ID',
                    'acl': 'public-read',
                    'key': 'the S3 key for the file (the S3 filename)',
                    'policy': '<?php echo $encodedPolicy?>',
                    'signature': '<?php echo $signature'?>,
                    'redirect': 'URL to redirect a success request (its doesnt matter)',
                    'x-amz-meta-sig': 'some meta signature to ensure authentic requests',
            };

            // this line sets the post params so that SWFUpload will send the additional fields when it uploads the file.
            $.swfu.setPostParams( swfConfig );

Over the last few months, I’ve been seeing a good amount of chatter surrounding “spec” design sites especially 99designs.com. After taking a look around the site I figured now was a good time to give it a shot. We were on a tight budget, tight time line, and my Dad didn’t have much direction for the logo.

I posted up a contest last Sunday here and we were looking at entries by Monday afternoon. Now things got more difficult. We were having a hard time coming up with “star ratings” and constructive feedback in general. My Dad’s staff was having a hard time not getting pigeon holed by the submitted designs and Setfive wasn’t doing a great job helping them along.

We did our best and we felt like the entries were moving in the right direction. Then the contest closed. In the last 8 hours of the contest the number of entries nearly doubled. With 70 entries we now had the problem with objectively picking a winning logo.

At this point, I wanted some more input on what people thought about the logos. I decided to create a set of Amazon Mechanical Turk tasks to get some feedback.

After about a day, I had 200 responses asking for user’s top three logos and any additional comments they had.

Some of the comments I got back were insightful and moving:

• Don’t pick any of the logos on the second page. They all look terrible.
• Due to nature of your business I would prefer a sober and serious looking logo.
• I chose these three because they are visually appealing, and convey a sense of being able to ease pain.
• I suffer from cronic pain. I wish you the best of luck in finding your logo. People that do your type of work are a life line for people like me. Hope I hope have helped.

I tallied up the results by weighting +3, +2, +1 for first, second, and third choices respectively. The results were interesting.

• Every logo received at least one vote.
• The top ten logos accounted for just about 41% of all the votes.
• Only counting the top choice caused 3 logos to fall out of the top ten.

The top ten logos as voted by the Amazon Mechanical Turks were:

Personally, I like the top ten logos and my Dad’s staff seems to like many of the same logos that were voted up. It’s been an interesting experiment almost exclusively using the “crowd” to design and then select a logo. I’m not sure if we’ll use 99designs in the future but it has been a pleasant experience.

We still haven’t picked a winning logo but I’ll update once we do!

Update:

We finally picked a winner! We decided to go with the crowd and selected http://99designs.com/contests/24619/entries/88 as the winning logo.

• We had a Google Maps application which loaded an image layer with about 3000 discrete overlays.
• We had lat/lng coordinates for all of these overlays.
• Each of these images had a text label that uniquely described it.
• We wanted the Turks to transcribe these labels.

Next, we began to investigate how the Mechanical Turk service actually works. The process is delightfully simple. The idea is to break the tasks out into discrete and reproducible actions called human intelligence tasks (HITs). This pattern aligned well with our problem because all of the overlays were independent and we had coordinates for each of them. Amazon displays each of these HITs inside a template that the “requester” constructs.

The templates are HTML pages (Amazon allows javascript) which plug-in variables for each HIT when the tasks go live. They also capture the data that you want to save from the task – in our case the labels. Designing templates is pretty straightforward and javascript is a nice touch.

For our task, we embeded Google Maps with our image overlaid and used HIT variables to display 3 markers per task. This is where we hit our first and only snag. Because of directory restrictions on Google Maps API keys our HIT template kept generating javascript errors because it was being executed on un-predictable domain names. We didn’t think this was a huge deal so we plowed ahead as planned.
We ran the first set of HITs with a 2 cent/HIT payment and a minimum approval rating of 95%.

The results were less than stellar – within an hour or so we received this email:

A strange error message pops up for your/these HITS.
“The Google Maps API key used on this web site was registered for a different web site. You can generate a new key for this web site at http://code.google.com/apis/maps/.”
Also, the mouseover that displays at screen bottom is “javascript: void (0)”…and no label appears below the marker either.  I do NOT have any javascript blockers operational, so no problem there.
Also, I would appreciate it if you NOT deny me credit for this HIT, as the technical error message was not my fault and not explained or warned in the instructions.
I would like to do several, possibly many of these HITS, if you can tell me what to do to overcome this problem.
Thanks,
[REDACTED]

People were obviously not getting the instructions and were scared away by the JS error and fear of HIT rejection. One of the problems with the service is that you can’t modify your HIT template once a set has been published. You have to cancel the batch and re-start after your edits. At this point, we decided to cancel the run and modify the instructions. With the new instructions people seemed to “get it” and were generally more willing to forgive the JS error.

At the end of the run, the accuracy of the Turks was around 90% or so on transcribing the labels. We got about 25 man hours of work accomplished in about 42 hours of actual time at a cost of about $25. All told we were really impressed with the service as well as the Turks themselves. We definitely recommend the service for any discrete and repeatable tasks.

Things we learned:

• Be EXTREMELY clear in your instructions – try to be as un-ambiguity as possible.
• The Turks live and die by their approval rating so be nice (we just accepted every task).
• Unexpected popups and JavaScript errors probably scare people away so try to avoid them.
• Obviously higher reward rates are going to attract more Turks – something to take into account.

We’ve also recently become fascinated by other things that could be played out or experimented with the Turks. Notably it seems like an ideal venue to experiment with some game theory topics.