sfPropelPager and GROUP BY criteria

So for one reason or another (actually a few bad ones) I ended up having to use a Criteria object looking like this:

<?php
$c = new Criteria ( );
$having = $c->getNewCriterion('the_count', count($tags), Criteria::GREATER_EQUAL);
$c->add(TaggingPeer::TAG_ID, $tags, Criteria::IN);
$c->addSelectColumn("COUNT(*) AS the_count");
$c->addSelectColumn(TaggingPeer::ID);
$c->addSelectColumn(TaggingPeer::TAGGABLE_ID);
$c->addSelectColumn(TaggingPeer::TAGGABLE_MODEL);
$c->addGroupByColumn(TaggingPeer::TAGGABLE_MODEL);
$c->addGroupByColumn(TaggingPeer::TAGGABLE_ID);
$c->addHaving($having);

It makes SQL that looks something like this:

SELECT tag_id, tag_model, id, COUNT(*) AS the_count FROM sf_tagging WHERE tag_id IN (1,2,3) GROUP BY taggable_model, taggable_id HAVING the_count > int

The query sucks but whatever it works.

My issue came when I tried to use it with a sfPropelPager. I set up the pager per usual but for some reason the results that were coming back weren’t correct. For some reason, the COUNT being returned by the sfPropelPager was completely wrong. It turns out the offending lines are here in sfPropelPager.class.php :

<?php
public function init()
{
  $hasMaxRecordLimit = ($this->getMaxRecordLimit() !== false);
  $maxRecordLimit = $this->getMaxRecordLimit();

  $cForCount = clone $this->getCriteria();
  $cForCount->setOffset(0);
  $cForCount->setLimit(0);
  $cForCount->clearGroupByColumns();

}

For whatever reason, sfPropelPager clears the GROUP BY clauses before it calculates the COUNT for a criteria object. I’m not sure why it does this – but it certainly is unexpected and breaks my query in particular.

There are a handful of posts about this on the Symfony forums and it looks like the Propel people know about the issue to.

The solution to this is to use the setPeerCountMethod() from sfPropelPager. The setPeerCountMethod() function allows you to specify a custom COUNT() method inside the peer for your Criteria. I went ahead and added a new function to put the GROUP BY columns back in:

<?php
public static function advancedSearchCount($c){
  $c->addGroupByColumn(TaggingPeer::TAGGABLE_MODEL);
  $c->addGroupByColumn(TaggingPeer::TAGGABLE_ID);
  return self::doCount($c);
}

This solution works but it is extremely rigid. Since the custom count function has to be static you’d really be out of luck if you had variable columns or other dynamic requirements.

I’d love to know if someone has a cleaner/better/more elegant solution for this.

Playing with appjet

The other day one of my friends jokingly mentioned that he thought stripperswere being “marginalized” as profession. I thought this was funny, so I leveraged the powers of the internet to prove to him that in fact strippers are less marginalized then say podiatrists.

Enter amimarginalized.com

Anyway, with all of the hub-dub about Javascript being the next “big thing” I decided to give appjet a whirl. Appjet allows developers to develop and deploy server side Javascript applications. Additionally, appjet recently released “appjet in a jar” which lets users deploy the appjet platform on a different server.

Getting the appjet platform up and running was really simple. I already had Java 6 installed on one of our servers so all I had to do was download the appjet jar and fire it up with “java -jar appjet.jar”

In general, getting things moving was pretty easy with appjet. It also just “feels” natural to write Javascript for the client and the server. Switching between client and server is done with a comment directive:

/* appjet:client */
/* appjet:server */

appjet also provides a special directive for CSS styles:

/* appjet:css */

The other feature I really liked about appjet is its support for printing HTML elements. Appjet comes with a tag library that makes creating HTML tags particularly simple. The syntax looks something like this:

print(

FORM({action:"/", method:"get", onsubmit: "javascript: return submitJob()"},
SPAN({id: "job-label"}, "Job Title: "),
INPUT({text:"text", name:"jobField", id: "jobField"}),
INPUT({type:"hidden", name:"isPost", id: "isPost", value: "1"}),
INPUT({type: "submit", value: "Submit"})
)
);

And boom you have a XHTML compliant form.

The other neat feature, (from the appjet docs) is that “You can also treat a tag like a JavaScript array (it has all the same methods) and add to it programmatically.” so something like this works:

var list = UL();

list.push(LI("One item"));
list.push(LI("Two item"));

["Red","Blue"].forEach(
function(color) {
list.push(LI(color+" item"));
}
);

printp("The following "+list.length+" items may be of interest.");
print(list);

I was also really impressed with the appjet persistent storage system. The platform lets you persist arbitrary Javascript objects in “collections” that can then be iterated, filtered, and sorted. On amimarginalized.com I have about 800 elements loaded up and it seems to perform reasonably well.

The one issue I have with the storage library is that there isn’t any way to just load a bunch of data into the system. You can only load data from inside an appjet script file. The problem I ran into is that I hit the Java 64kb file size limit pretty quickly. It would be really awesome if the JAR had some functionality to load up say a file full of JSON objects.

All and all, using appjet has been a positive experience. It was really easy and fun to build an “easy” app using it. I’d really like to know some more about how appjet is put together but the documentation is sparse. The only information I could find was the logos on the download page. They suggest that appjet is composed off AppJet, Rhino, Jetty and of course Java. I’d be interested to know if there are any plans to expose JVM libraries to appjet code. It seems like this would allow the platform to quickly gain extensive library support - including the JDBC.

Anyway, the amimarginalized.com site uses “advanced” algorithms to determine how marginalized your career is. It just counts the number of results on Yahoo BOSS for the job you enter and compares it to a set of about 800 saved jobs.

You can download the source code here.

JSON explorer for Firebug!

As Firefox plugins go, Firebug is definitely one of my favorites. It has saved me and the rest of the Setfive team dozens of development hours. One of the nicest features of Firebug is the NET panel. The NET panel allows a developer to monitor HTTP requests that Firefox is making during the course of a page’s execution. This is particularly useful to AJAX heavy applications because it allows a developer to easily debug XHR requests and responses.

Although this is great, the NET panel is severely lacking when it comes to dealing with JSON responses. Since JSON must be a legal JavaScript string, it can’t contain any line breaks. As a consequence, debugging long JSON strings is extremely painful.

This issue has been on the Firebug TODO list for some time. I figured this would be a good place to jump into the Firebug codebase. I dropped an email to the Firebug working group and received some great feedback. Honza gave me a template extension to extend the NET panel and provided some great insight and advice on implementing a JSON viewer.

After a couple of weeks of intermittent work, the JSON-Viewer extension is ready to play with. Honza has committed the code into the Firebug 1.4+ branch so if you grab a new build you’ll be able to use the JSON viewer.

Screenie:

Read more at Honza’s blog post

Artificial artificial intelligence - our experience with Mechanical Turk

So Amazon Web Services (AWS) has a pretty neat service called the Mechanical Turk. The name of the service is inspired by this story where a human was hidden inside a chess playing “robot” which impressed crowds during the late 18th century. Amazon’s service doesn’t hide the fact that its powered by humans but the end result is the same - humans performing tasks in lieu of a computer. The task we needed “turked” was the transcription of various text labels that were on top of overlays on a particular large image. The goal of all of this was a Google Maps product so we all ready had the overlays loaded into a Google Maps UI. To recap:

  • We had a Google Maps application which loaded an image layer with about 3000 discrete overlays.
  • We had lat/lng coordinates for all of these overlays.
  • Each of these images had a text label that uniquely described it.
  • We wanted the Turks to transcribe these labels.

Next, we began to investigate how the Mechanical Turk service actually works. The process is delightfully simple. The idea is to break the tasks out into discrete and reproducible actions called human intelligence tasks (HITs). This pattern aligned well with our problem because all of the overlays were independent and we had coordinates for each of them. Amazon displays each of these HITs inside a template that the “requester” constructs.

The templates are HTML pages (Amazon allows javascript) which plug-in variables for each HIT when the tasks go live. They also capture the data that you want to save from the task - in our case the labels. Designing templates is pretty straightforward and javascript is a nice touch.

For our task, we embeded Google Maps with our image overlaid and used HIT variables to display 3 markers per task. This is where we hit our first and only snag. Because of directory restrictions on Google Maps API keys our HIT template kept generating javascript errors because it was being executed on un-predictable domain names. We didn’t think this was a huge deal so we plowed ahead as planned. We ran the first set of HITs with a 2 cent/HIT payment and a minimum approval rating of 95%.

The results were less than stellar - within an hour or so we received this email:

A strange error message pops up for your/these HITS. “The Google Maps API key used on this web site was registered for a different web site. You can generate a new key for this web site at http://code.google.com/apis/maps/.” Also, the mouseover that displays at screen bottom is “javascript: void (0)”…and no label appears below the marker either. I do NOT have any javascript blockers operational, so no problem there. Also, I would appreciate it if you NOT deny me credit for this HIT, as the technical error message was not my fault and not explained or warned in the instructions. I would like to do several, possibly many of these HITS, if you can tell me what to do to overcome this problem. Thanks, [REDACTED]

People were obviously not getting the instructions and were scared away by the JS error and fear of HIT rejection. One of the problems with the service is that you can’t modify your HIT template once a set has been published. You have to cancel the batch and re-start after your edits. At this point, we decided to cancel the run and modify the instructions. With the new instructions people seemed to “get it” and were generally more willing to forgive the JS error.

At the end of the run, the accuracy of the Turks was around 90% or so on transcribing the labels. We got about 25 man hours of work accomplished in about 42 hours of actual time at a cost of about $25. All told we were really impressed with the service as well as the Turks themselves. We definitely recommend the service for any discrete and repeatable tasks.

Things we learned:

  • Be EXTREMELY clear in your instructions - try to be as un-ambiguity as possible.
  • The Turks live and die by their approval rating so be nice (we just accepted every task).
  • Unexpected popups and JavaScript errors probably scare people away so try to avoid them.
  • Obviously higher reward rates are going to attract more Turks - something to take into account.

We’ve also recently become fascinated by other things that could be played out or experimented with the Turks. Notably it seems like an ideal venue to experiment with some game theory topics.

Timelapse Twitter+Election map

This is an update to our Twitter+Election ’08 mashup that was over at Setfive Election HQ

Well as everyone saw last Tuesday night, Obama won the election by a pretty significant margin and has all ready taken steps to announce his transition agenda.

Anyway, at the end of our run, we captured 11021 tweets with the breakdown being 2501 for McCain and 8520 for Obama. Since we had been generating maps all day we decided to take snapshots at 5 minute intervals so that we could watch the progression of the map. The timelapse map is embedded below:

We hope everyone had a good election experiance - we had a lot of fun building this mashup. Now to find the next big thing…