Tag: amazon

SwiftMailer: Expected response code 250 but got code 421

By Ashish Datta | 7 Comments on SwiftMailer: Expected response code 250 but got code 421

Last week we deployed a background script for a client which was used intermitently to batch send a couple of hundred emails. We were using SwiftMailer but weren’t able to use the “Spool” strategy to send because the messages contained Unicode characters which was breaking the serialization. Anyway, we ended up with code that looked something like the following:

Nothing to crazy going on. We were also sending the emails through Amazon SES which is why we introduced the sleep(..) to keep ourselves below the sending limits.

Things seemed like they were fine but then we’d seemingly randomly get the following exception:

PHP Fatal error:  Uncaught exception 'Swift_TransportException' with message 'Expected response code 250 but got code "421", with message "421 Timeout waiting for data from client.
"' in /usr/share/php/symfony/vendor/swiftmailer/classes/Swift/Transport/AbstractSmtpTransport.php:406
Stack trace:
#0 /usr/share/php/symfony/vendor/swiftmailer/classes/Swift/Transport/AbstractSmtpTransport.php(290): Swift_Transport_AbstractSmtpTransport->_assertResponseCode('421 Timeout wai...',$
#1 /usr/share/php/symfony/vendor/swiftmailer/classes/Swift/Transport/EsmtpTransport.php(197): Swift_Transport_AbstractSmtpTransport->executeCommand('MAIL FROM: <adm...', Array, Arra$
#2 /usr/share/php/symfony/vendor/swiftmailer/classes/Swift/Transport/EsmtpTransport.php(267): Swift_Transport_EsmtpTransport->executeCommand('MAIL FROM: <adm...', Array)
#3 /usr/share/php/symfony/vendor/swiftmailer/classes/Swift/Transport/AbstractSmtpTransport.php(441): Swift_Transport_EsmtpTransport->_doMailFromCommand('admin@chatthrea...')
#4 /usr/share/php/symfony/ve in /usr/share/php/symfony/vendor/swiftmailer/classes/Swift/Transport/AbstractSmtpTransport.php on line 406

After doing some digging around, it turns out Amazon’s SES service has a connection timeout which SwiftMailer was tripping up on. I couldn’t actually find an official published timeout limit but looking at the SwiftMailer code it seemed like it was possible to set a timeout inside Swift. We added a “timeout: 5” setting to our Symfony factories.yml file inside the SwiftMailer settings and it seemed to fix our issues.

Running Java apps from the crontab

By Ashish Datta | 1 Comment on Running Java apps from the crontab

Earlier this week I was completely dumbfounded by a PHP script that launched a Java app that seemed to work fine when it was run from the command line but kept failing when it was run from a cron.

The Java app in question was “ec2-describe-group” out of the Amazon EC2 API Tools package.  Basically, the ec2-describe-group tool hits the EC2 API and returns information about your account’s currently configured security groups.

The issue I was having was that when the PHP script was launched from a cron ec2-describe-group would keep returning an empty string, but when the script was launched from the CLI ec2-describe-group behaved normally.

After some poking around, I found this StackOverflow post which points out that most the environment variables your shell has aren’t available in a cronjob.

With that in mind, I tried adding JAVA_HOME as well as EC2_HOME to my crontab. Doing this is pretty straight forward, just add these two lines above any of your scheduled jobs:

EC2_HOME=/opt/ec2-api-tools-1.3.36506
JAVA_HOME=/etc/java-config-2/current-system-vm

Unfortunately, this still didn’t resolve the issue. On a whim, I decided to check what type of file ec2-describe-group actually is and discovered that its a Bash script not a Java JAR. Looking at the Bash, the file is actually just executing “EC2_HOME/bin/ec2-cmd DescribeGroups” but it utilizes other environment variables that my cron didn’t have.

For simplicity’s sake, I decided to just switch the PHP script to run ec2-cmd directly and finally everything started working as expected.

Upload directly to S3 with SWFUpload

By Ashish Datta | 14 Comments on Upload directly to S3 with SWFUpload

I was working on an application earlier today that required allowing a user to upload a large file (several hundred MB) which would eventually be stored on Amazon S3. After reviewing the requirements, I realized it made sense to just upload the file directly to S3 instead of having to first stage the file on a server and then use PHP to push the file to S3.

Amazon has a nice walk through of using a plain HTML form to upload a file directly to S3 here.

I had all ready been using SWFUpload to upload files to the server so I decided to look into using it to uploading directly to S3. After some head banging, I finally got it to work – here’s the quick n dirty.

  1. Download SWFUpload 2.5
  2. Get SWFUpload ready to use in your project. Copy the SWF file somewhere accessible and include their swfupload.js Javascript file. More info here
  3. Setup an S3 bucket. You’ll need to set the policy to allow uploads from your own user (its the default).
  4. Place a crossdomain.xml file in the root of your S3 bucket. This file “authorizes” flash player to upload files into this host. The content of the file is below.
  5. Initialize the SWFUpload object (example below).
  6. Before beginning the upload, you need to set the appropriate postParams in the SWFUpload object. This is really the “magic” of this process. Example is below.
  7. Start the upload with startUpload()

Thats it! It’s pretty straight forward once you have things going. As an FYI, you can put SWFUpload into “debug” mode by adding debug: true as a property to the initialization object. You can also debug the responses from Amazon by using a packet sniffer like Wireshark.

crossdomain.xml

You probably want to make this file a little less permissive. More details here. Also note, there are differences in the implementation of the file between various versions of Flash player.

Initialize SWFUpload

Set SWFUpload postParams

The HMAC signature MUST be calculated on the server because it uses your S3 secret. You MUST keep that value secret in order to maintain the security of your S3 buckets. I’m using Don Schonknecht’s S3 PHP library to calculate the HMAC signatures but you could just as easily do it in straight PHP.

99designs and Amazon. Design. Crowd sourced.

By Ashish Datta | Leave a Comment on 99designs and Amazon. Design. Crowd sourced.

A week or so ago my Dad asked if we could have our designer put together a logo for him. Unfortunately, our guy was buried under a mound of work and generally couldn’t help us out. We haven’t always had the best luck with Craigslist so I was ready to try something new.

Over the last few months, I’ve been seeing a good amount of chatter surrounding “spec” design sites especially 99designs.com. After taking a look around the site I figured now was a good time to give it a shot. We were on a tight budget, tight time line, and my Dad didn’t have much direction for the logo.

I posted up a contest last Sunday here and we were looking at entries by Monday afternoon. Now things got more difficult. We were having a hard time coming up with “star ratings” and constructive feedback in general. My Dad’s staff was having a hard time not getting pigeon holed by the submitted designs and Setfive wasn’t doing a great job helping them along.

We did our best and we felt like the entries were moving in the right direction. Then the contest closed. In the last 8 hours of the contest the number of entries nearly doubled. With 70 entries we now had the problem with objectively picking a winning logo.

At this point, I wanted some more input on what people thought about the logos. I decided to create a set of Amazon Mechanical Turk tasks to get some feedback.

After about a day, I had 200 responses asking for user’s top three logos and any additional comments they had.

Some of the comments I got back were insightful and moving:

  • Don’t pick any of the logos on the second page. They all look terrible.
  • Due to nature of your business I would prefer a sober and serious looking logo.
  • I chose these three because they are visually appealing, and convey a sense of being able to ease pain.
  • I suffer from cronic pain. I wish you the best of luck in finding your logo. People that do your type of work are a life line for people like me. Hope I hope have helped.

I tallied up the results by weighting +3, +2, +1 for first, second, and third choices respectively. The results were interesting.

  • Every logo received at least one vote.
  • The top ten logos accounted for just about 41% of all the votes.
  • Only counting the top choice caused 3 logos to fall out of the top ten.

The top ten logos as voted by the Amazon Mechanical Turks were:

Entry ID Votes URL

88

76

 http://99designs.com/contests/24619/entries/88

78

65

 http://99designs.com/contests/24619/entries/78

75

60

 http://99designs.com/contests/24619/entries/75

87

55

 http://99designs.com/contests/24619/entries/87

91

51

 http://99designs.com/contests/24619/entries/91

58

48

 http://99designs.com/contests/24619/entries/58

47

42

 http://99designs.com/contests/24619/entries/47

90

41

 http://99designs.com/contests/24619/entries/90

43

36

 http://99designs.com/contests/24619/entries/43

Personally, I like the top ten logos and my Dad’s staff seems to like many of the same logos that were voted up. It’s been an interesting experiment almost exclusively using the “crowd” to design and then select a logo. I’m not sure if we’ll use 99designs in the future but it has been a pleasant experience.

We still haven’t picked a winning logo but I’ll update once we do!

Update:

We finally picked a winner! We decided to go with the crowd and selected http://99designs.com/contests/24619/entries/88 as the winning logo.

Artificial artificial intelligence – our experience with Mechanical Turk

By Ashish Datta | 2 Comments on Artificial artificial intelligence – our experience with Mechanical Turk

So Amazon Web Services (AWS) has a pretty neat service called the Mechanical Turk. The name of the service is inspired by this story where a human was hidden inside a chess playing “robot” which impressed crowds during the late 18th century. Amazon’s service doesn’t hide the fact that its powered by humans but the end result is the same – humans performing tasks in lieu of a computer.
The task we needed “turked” was the transcription of various text labels that were on top of overlays on a particular large image. The goal of all of this was a Google Maps product so we all ready had the overlays loaded into a Google Maps UI.  To recap:

  • We had a Google Maps application which loaded an image layer with about 3000 discrete overlays.
  • We had lat/lng coordinates for all of these overlays.
  • Each of these images had a text label that uniquely described it.
  • We wanted the Turks to transcribe these labels.

Next, we began to investigate how the Mechanical Turk service actually works. The process is delightfully simple. The idea is to break the tasks out into discrete and reproducible actions called human intelligence tasks (HITs). This pattern aligned well with our problem because all of the overlays were independent and we had coordinates for each of them. Amazon displays each of these HITs inside a template that the “requester” constructs.

The templates are HTML pages (Amazon allows javascript) which plug-in variables for each HIT when the tasks go live. They also capture the data that you want to save from the task – in our case the labels. Designing templates is pretty straightforward and javascript is a nice touch.

For our task, we embeded Google Maps with our image overlaid and used HIT variables to display 3 markers per task. This is where we hit our first and only snag. Because of directory restrictions on Google Maps API keys our HIT template kept generating javascript errors because it was being executed on un-predictable domain names. We didn’t think this was a huge deal so we plowed ahead as planned.
We ran the first set of HITs with a 2 cent/HIT payment and a minimum approval rating of 95%.

The results were less than stellar – within an hour or so we received this email:

A strange error message pops up for your/these HITS.
“The Google Maps API key used on this web site was registered for a different web site. You can generate a new key for this web site at http://code.google.com/apis/maps/.”
Also, the mouseover that displays at screen bottom is “javascript: void (0)”…and no label appears below the marker either.  I do NOT have any javascript blockers operational, so no problem there.
Also, I would appreciate it if you NOT deny me credit for this HIT, as the technical error message was not my fault and not explained or warned in the instructions.
I would like to do several, possibly many of these HITS, if you can tell me what to do to overcome this problem.
Thanks,
[REDACTED]

People were obviously not getting the instructions and were scared away by the JS error and fear of HIT rejection. One of the problems with the service is that you can’t modify your HIT template once a set has been published. You have to cancel the batch and re-start after your edits. At this point, we decided to cancel the run and modify the instructions. With the new instructions people seemed to “get it” and were generally more willing to forgive the JS error.

At the end of the run, the accuracy of the Turks was around 90% or so on transcribing the labels. We got about 25 man hours of work accomplished in about 42 hours of actual time at a cost of about $25. All told we were really impressed with the service as well as the Turks themselves. We definitely recommend the service for any discrete and repeatable tasks.

Things we learned:

  • Be EXTREMELY clear in your instructions – try to be as un-ambiguity as possible.
  • The Turks live and die by their approval rating so be nice (we just accepted every task).
  • Unexpected popups and JavaScript errors probably scare people away so try to avoid them.
  • Obviously higher reward rates are going to attract more Turks – something to take into account.

We’ve also recently become fascinated by other things that could be played out or experimented with the Turks. Notably it seems like an ideal venue to experiment with some game theory topics.