Client side RSS aggregator

One of our clients came to us a few days ago asking us if we could build something to aggregate several RSS feeds and then display it on their site.

Easy enough. Except the caveats were 1) This had to be done on a shoe string budget and 2) We had no permission to script on the server side (only client side scripting allowed kids).

We put our heads together and realized that Yahoo Pipes would provide the functionality to easily combine several RSS feeds, sort them, filter them, ect. And best part? It’s all built and all free. Pipes also provides the functionality to export a pipe as a JSON feed (as opposed to RSS).

So that takes care of problem 1 and almost fixes problem 2. Of course the remaining problem is getting around the cross domain XHR restrictions on the JSON. I did some poking around the Yahoo Pipes documentation and it turns out you can add a _callback parameter to the URL and bang it wraps the JSON in a JS callback!

Issues solved. The final part was to mix in a little jQuery to load the JSON+callback feed into a <script> tag and then display it.

Total lines of code: 27 lines of javascript.

EDIT. Now with code!

Hello Android!

In the last few weeks the battle and buzz over the smart phone market seems to have seriously intensified.

First there was the usual iPhone buzz, news about the Android powered HTC Magic, the Windows Mobile marketplace, and of course the obligatory ridiculousness at Microsoft.

I’d been considering experimenting with a mobile platform for sometime and finally decided to take the plunge. I decided to give Android a whirl primarily because I don’t have easy access to OSX or Visual Studio and my Java is less rusty than my .NET.

Anyway, getting going with Android was deliciously simple – download the SDK+Emulator and Eclipse plugin and you’re off.

After the necessary “Hello World” application I tried to write something a bit more substantive. Personally, one of the coolest facets of mobile development is the ability for applications to be location aware (GPS). Mix this together with some openly available geo tagged data and the result is probably going to be interesting.

With this in mind, the plan became to mash together Android’s GPS coordinates with flickr’s geotagged photos.

Getting access to Android’s location service is fairly straightforward. You basically register to receive updates either when the device moves a certain distance or on some time interval:

The biggest “gotcha” with this is that you NEED to remember to modify the default Application security settings to allow you to access the device’s location. In Eclipse, edit AndroidManifest.xml and add “UsesPermission” for the following: android.permission.ACCESS_MOCK_LOCATION, android.permission.ACCESS_COARSE_LOCATION, android.permission.ACCESS_FINE_LOCATION

So on to part II – using the device’s location to pull down Flickr photos. I’d used the Flickr API before so I knew how to do it but I’d never used it from Java. I tried loading the JAR for the flickrj client library but the Android JVM was having some strange issues with it. I was under the impression you can link to external JARs from Eclipse but I may be wrong (anyone?).

Anyway, the Flickr requests were un-authenticated and pretty straightforward so I decided to use Java’s URL class. Accessing sockets was another “gotcha” – Android requires your application to have the “UsesPermission” android.permission.INTERNET to use sockets. The exception when the permission isn’t set is notably cryptic – “unknown error”.

I decided to download all the Flickr photos to the device so that the UX would be generally smoother. This introduced threading to the project so that the UI wouldn’t freeze up while the photos were downloading. Android threads work just like traditional Java threads and the process was generally painless:

With the photos pulled down the final task was displaying them. After poking around the Android documentation I discovered the Gallery widget. It basically allows you to display a set of items in a list and specify a “renderer” for the gallery. I’m not sure if there is a default way to make it “fisheye” (like on an iPhone) but I rolled a quick n dirty solution for that. I also couldn’t get it to look really sexy but that’s also probably possible.

So that’s about it. Here are some screen shots of the application running in the emulator:

And without further a due here is the code as an Eclipse project.
geoflickr

Anyway, before the bashing starts – I know I’m a terrible Java programmer and that this project isn’t really engineered beautifully. It was just supposed to be a way to get my (and anyone else’s) feet wet with Android. Any comments/thoughts/improvements are of course welcome!

sfPropelPager and GROUP BY criteria

So for one reason or another (actually a few bad ones) I ended up having to use a Criteria object looking like this:

It makes SQL that looks something like this:

“SELECT tag_id, tag_model, id, COUNT(*) AS the_count FROM sf_tagging WHERE tag_id IN (1,2,3) GROUP BY taggable_model, taggable_id HAVING the_count > int”

The query sucks but whatever it works.

My issue came when I tried to use it with a sfPropelPager. I set up the pager per usual but for some reason the results that were coming back weren’t correct. For some reason, the COUNT being returned by the sfPropelPager was completely wrong. It turns out the offending lines are here in sfPropelPager.class.php :

For whatever reason, sfPropelPager clears the GROUP BY clauses before it calculates the COUNT for a criteria object. I’m not sure why it does this – but it certainly is unexpected and breaks my query in particular.

There are a handful of posts about this on the Symfony forums and it looks like the Propel people know about the issue to.

The solution to this is to use the setPeerCountMethod() from sfPropelPager. The setPeerCountMethod() function allows you to specify a custom COUNT() method inside the peer for your Criteria. I went ahead and added a new function to put the GROUP BY columns back in:

This solution works but it is extremely rigid. Since the custom count function has to be static you’d really be out of luck if you had variable columns or other dynamic requirements.

I’d love to know if someone has a cleaner/better/more elegant solution for this.

Playing with appjet

The other day one of my friends jokingly mentioned that he thought strippers were being “marginalized” as profession. I thought this was funny, so I leveraged the powers of the internet to prove to him that in fact strippers are less marginalized then say podiatrists.

Enter amimarginalized.com

Anyway, with all of the hub-dub about Javascript being the next “big thing” I decided to give appjet a whirl. Appjet allows developers to develop and deploy server side Javascript applications. Additionally, appjet recently released “appjet in a jar” which lets users deploy the appjet platform on a different server.

Getting the appjet platform up and running was really simple. I already had Java 6 installed on one of our servers so all I had to do was download the appjet jar and fire it up with “java -jar appjet.jar”

In general, getting things moving was pretty easy with appjet. It also just “feels” natural to write Javascript for the client and the server. Switching between client and server is done with a comment directive:

/* appjet:client */
/* appjet:server */

appjet also provides a special directive for CSS styles:

/* appjet:css */

The other feature I really liked about appjet is its support for printing HTML elements. Appjet comes with a tag library that makes creating HTML tags particularly simple. The syntax looks something like this:

print(

FORM({action:"/", method:"get", onsubmit: "javascript: return submitJob()"},
SPAN({id: "job-label"}, "Job Title: "),
INPUT({text:"text", name:"jobField", id: "jobField"}),
INPUT({type:"hidden", name:"isPost", id: "isPost", value: "1"}),
INPUT({type: "submit", value: "Submit"})
)
);

And boom you have a XHTML compliant form.

The other neat feature, (from the appjet docs) is that “You can also treat a tag like a JavaScript array (it has all the same methods) and add to it programmatically.” so something like this works:

var list = UL();

list.push(LI("One item"));
list.push(LI("Two item"));

["Red","Blue"].forEach(
function(color) {
list.push(LI(color+" item"));
}
);

printp("The following "+list.length+" items may be of interest.");
print(list);

I was also really impressed with the appjet persistent storage system. The platform lets you persist arbitrary Javascript objects in “collections” that can then be iterated, filtered, and sorted. On amimarginalized.com I have about 800 elements loaded up and it seems to perform reasonably well.

The one issue I have with the storage library is that there isn’t any way to just load a bunch of data into the system. You can only load data from inside an appjet script file. The problem I ran into is that I hit the Java 64kb file size limit pretty quickly. It would be really awesome if the JAR had some functionality to load up say a file full of JSON objects.

All and all, using appjet has been a positive experience. It was really easy and fun to build an “easy” app using it. I’d really like to know some more about how appjet is put together but the documentation is sparse. The only information I could find was the logos on the download page. They suggest that appjet is composed off AppJet, Rhino, Jetty and of course Java. I’d be interested to know if there are any plans to expose JVM libraries to appjet code. It seems like this would allow the platform to quickly gain extensive library support – including the JDBC.

Anyway, the amimarginalized.com site uses “advanced” algorithms to determine how marginalized your career is. It just counts the number of results on Yahoo BOSS for the job you enter and compares it to a set of about 800 saved jobs.

You can download the source code here.

JSON explorer for Firebug!

As Firefox plugins go, Firebug is definitely one of my favorites. It has saved me and the rest of the Setfive team dozens of development hours. One of the nicest features of Firebug is the NET panel. The NET panel allows a developer to monitor HTTP requests that Firefox is making during the course of a page’s execution. This is particularly useful to AJAX heavy applications because it allows a developer to easily debug XHR requests and responses.

Although this is great, the NET panel is severely lacking when it comes to dealing with JSON responses. Since JSON must be a legal JavaScript string, it can’t contain any line breaks. As a consequence, debugging long JSON strings is extremely painful.

This issue has been on the Firebug TODO list for some time. I figured this would be a good place to jump into the Firebug codebase. I dropped an email to the Firebug working group and received some great feedback. Honza gave me a template extension to extend the NET panel and provided some great insight and advice on implementing a JSON viewer.

After a couple of weeks of intermittent work, the JSON-Viewer extension is ready to play with. Honza has committed the code into the Firebug 1.4+ branch so if you grab a new build you’ll be able to use the JSON viewer.

Screenie:

Read more at Honza’s blog post