Category: Demo

Random acts of madness: JS+Flex+Rhino – WebWorkers for IE

By Ashish Datta | 2 Comments on Random acts of madness: JS+Flex+Rhino – WebWorkers for IE

Preface: This is a bad idea with untested code. If you deploy it on a production server bad things will happen.

A few weeks ago I was trolling the Internet and ran across an interesting piece over at John Resig’s blog about Javascript WebWorkers. Basically, WebWorkers are a draft recommendation that allow you to run Javascript functions on a background (non-UI thread) thread. Essentially, they would allow you to do long running computations without hanging the browser’s UI. Pretty neat. Problem is that they are currently only available in Firefox 3.5+, Safari 4, and Chrome 3.0

In my never ending quest to use every buzzword at least once I decided to try and implement a compatibility layer to bring support for WebWorkers to other browsers. The plan was to use Java6’s new embeded Javascript interpreter (it’s just Rhino) to execute the WebWorker code server side and then pipe the output back to the client. Again, this is really just a proof of concept.

There are three parts to the rig: the client Javascript library, a Flex/AS3 application for streaming client to server communication, and a Java application that uses Rhino to execute the Javascript.

Client Javascript

The client Javascript detects the user’s browser and then will define a “Worker” object if the user’s browser doesn’t support WebWorkers. The new “Worker” object uses the Flex application to pass messages back and forth to the server and calls the user’s onmessage function when data arrives from the server.

I sniped the browser detection code from Quirksmode and it seems to work fairly well. The rest of the code is below:

Flex/AS3 Application

The Flex application is basically a dumb conduit between the server and the client. All it really does is pass messages between the Java on the server and the Javascript on the client.

The trickiest part of getting this to work was Adobe’s insane rules for allowing their Socket classes to connect to servers. In order for the client to successfully connect to the server you need to serve a XML policy file from port 843. Additionally, this file can’t be served by a HTTP server but must be a custom server that only spits back the file along with a null carriage return. A detailed description of this abortion is here http://www.adobe.com/devnet/flashplayer/articles/socket_policy_files.html

This really posses two problems. One, you need to be running some random “policy file server” for Flex sockets to be of any use. And two, since 843 is a privileged port, this server can’t be started by a regular user.

The most interesting parts of the ActionScript are probably the snippets that call out to Javascript functions:

ExternalInterface.call("sfWebWorkersSWFReady", true);

Java Server

The most complicated part of this whole thing is probably the Java application that actually executes the WebWorker Javascript. All the communication between the Flex and Java is done entirely with JSON. The server basically does the following:

  1. Listen for connections from the Flex and accept them when they come in.
  2. When a message comes in – it can either be a request to create a new web worker or a postMessage() event containing some data for an existing worker.
  3. If it’s a request for a new worker, the server will download the Javascript file and then execute it inside a Rhino container.
  4. Otherwise if Flex passed a postMessage() message the server will forward that data to the running web worker.
  5. The other event that happens is that a web worker can send messages back to the Flex.

Anyway, I tested this on IE7+ and it seemed to work decently well. Per the warning on top I don’t want to leave this running on a live server anywhere.

If you want to get it to actually run, do the following:

  1. Download the zip of all the sources here.
  2. Start the JAR in WebWorkerServer/WebWorkerServer.jar with java -jar WebWorkerServer.jar 9999
  3. On the top of web/sfwwcompat.js change the IP address or the server to where your Java server is located (localhost if you want)
  4. Open web/wwsha1.html in IE or Chrome 2.0 and you should see stuff happen.

What’s in the box:

  • web/ contains the Javascript and a demo.
  • WebWorkerConduit/ contains the Flex applicaiton.
  • WebWorkerServer/ contains the Java server.

Credits: I borrowed the WebWorker SHA1 implementation from John Resig who adapted it from Ray C Morgan.

So here is another crazy idea. Instead of executing the WebWorker code on the server, would it be possible to dynamically make the WebWorker code re-entrant using setTimeout() on the client where loop structures exist?

The Redline Challenge

By Ashish Datta | Leave a Comment on The Redline Challenge

For one reason or another we decided to sponsor a pub crawl this weekend. The plan was hatched over some beers at Underbones on Thursday night for a Saturday morning go time. We knew we basically needed three things: a list of bars, some swag (tshirt?), and obviously a website. We decided that the route of the crawl should follow the MBTA Redline so that we could start downtown and then finish in Somerville. This made picking bars pretty simple, gave us some branding, and of course we registered
REDLINECHALLENGE.COM.

We wanted the website to have some informative information, live location updates, and of course pictures of the debauchery. The biggest problem was that neither Daum nor I have location aware phones. To get around this, we decided to update Twitter with our current location along with a “#loc” hashtag and then have the site update based on that. Since we were all ready using Twitter, we decided to use Twitpic to allow us to post pictures to twitter on the fly. Additionally, we took advantage of Verizon Wireless’s email to SMS service and allowed people to contact us via the website. All told, we built the site in about 3 hours and it proved to be pretty useful. People used it to find us on the crawl and to contact us while we were out. Everyone also got a kick of seeing a live photo stream.

What’s next? Clearly, The Greenline Challenge.

Words of Congress: Fun with Hadoop

By Ashish Datta | Leave a Comment on Words of Congress: Fun with Hadoop

For the last few weeks we’ve been working on a project that involved dealing with bills in the US House and Senate. Naturally, I decided it was time to make a word cloud from the frequencies of the words in the bills!

Checkout the final product here.

I decided to use only the bills from the 111th congress (the current one), all the bills (6703 of them) were downloaded from the THOMAS library at http://thomas.loc.gov/home/gpoxmlc111/ The files are XML documents that have the full text of the bills along with some meta data.

Not really to many files but I decided to use Hadoop and try and Map/Reduce the bills to count up the word frequencies. Getting Hadoop to run locally was pretty straightforward – just tell it where JAVA_HOME is and I was off to the races. Fortunately enough, one of the pre-canned examples was a word frequency counter so I decided to modify that for what I wanted.

The example map/reduce was written to process plain text files so I had to modify it to work with the XML documents. What this involved was writing a custom InputFormat class to open each bill, extract the appropriate plain text from the XML, and then pass this back as the “data”. I also modified the word counter to ignore words shorter than 6 characters.

I tested locally with a small subset of bills and everything seemed to be working fine. The trouble started when I tried to bring up Daum’s machine as a slave to my machine. After some finagling and hair pulling I finally got it working. The takeaways were:

  • You can’t run your DataNode on localhost, it needs to be your computer’s hostname to accept connections.
  • Hostnames are important. If you don’t have a DNS server make sure your hostnames are aliased in /etc/hosts
  • If your HDFS set up is showing 100% utilization but you know it isn’t true, try rm’ing the data file and then re-formatting your namenode.
  • If a copy or reduce step fails in distributed mode the error messages are usually really cryptic – check the actual logs.
  • When something throws an exception during a map or reduce operation, the error won’t be reported to STDOUT

Anyway, it was a slightly frustrating but rewarding experience – I even got to code some Java! The visualization of the word frequencies is here.

Might be about time to process one of the Amazon datasets with EC2…

FOSS Fridays – Tracking Your Users

By Matt Daum | Leave a Comment on FOSS Fridays – Tracking Your Users

The other night I thought it’d be helpful to see how people browse your site.  I think you can probably learn a lot about how a user moves over your site.  You can tell a lot about your user’s experience by watching their mouse.  You can see where they look on the page for specific information. I’ve created a demo of the tracking.

I know that there are some products out there that already do user session tracking and replays.  Also there are click heat maps which are interesting when you are looking on your site to see what links the user clicks the most.  I decided to just rebuild the session replay just out of curiousity on how difficult it’d be to do.  It was fairly simple and took me only 15-20 minutes. There are a number of improvements you could make to the script such as the window.unload handling is not 100% depending on your browser.  You could also do much more parsing on the client side of the information by using JSON.  If you wanted to store more than one page of tracking data you could quickly modify the script to pass the name page which it was tracking and to store the data separately for each page.

To use the script all you need to do is add a little Javascript on the bottom of the page you want to track a user.   The script tracks a users mouse movement as soon as they open the site.  It keeps track of time so that during the replay you can get the proper mouse movement at the right times to replay the users session.  While the user moves their mouse it continues to store all the data client side.  Once the window is closed it sends all the information to the server.  The server simply parses the data string.  For session replay it is done via setTimeout and it moved an image(of a cursor) at different intervals to simulate the users session.

While the script is not very pretty it was written very quickly and just as a proof of concept.  It goes to show you can easily track your user’s session without having to purchase expensive products, and that it can be done fairly simply.

The code is below with descriptions of what each snippet does. The script uses jQuery. To deploy this on numerous pages all you really would need to add would be a script tag that pulls in the tracking javascript.

Tracking the users movements and sending the server the information javascript:

var points='';
var timeSeconds=0;
$(document).ready(function(){
  $('body').append('');
  timer();
   $().mousemove(function (e){
    points=points+e.pageX+","+e.pageY+","+timeSeconds+"|";
  });
  $(window).unload(function(){
    sendData();
  });
});
function timer()
{
  timeSeconds=parseInt(timeSeconds)+1;
  setTimeout("timer()",10);
}
function sendData(){
    $.post('index.php','data='+points);
}

To parse the information on the server:

if($_SERVER['REQUEST_METHOD']=='POST')
{
  $fp=fopen($_SERVER['REMOTE_ADDR'].'tracking.dat','w+');
   fwrite($fp,$_POST['data']);
  fclose($fp);
  exit(1);
}

To replay the session first get the data:

if(file_exists($_SERVER['REMOTE_ADDR'].'tracking.dat'))
{
 $data=explode("|",file_get_contents($_SERVER['REMOTE_ADDR'].'tracking.dat'));
}

The moving of the cursor image javascript:

function moveMouse(x,y){
 $('#cursor').attr('style','position:absolute;left:'+x+"px;top:"+y+"px;");
}

Create different calls for each time the mouse was moved and have them execute at the times the user moved the mouse:

foreach($data as $d)
{
  $parts=explode(',',$d);
  if(count($parts)==3)
  echo 'setTimeout("moveMouse('.$parts[0].','.$parts[1].')",'.($parts[2]*10).");\n";
}

And you are all set.  As I said the script is only for proof of concept and not too pretty.  Let me know if you have any questions.

FOSS Fridays: MacGyvered Key/Value in Symfony

By Ashish Datta | 3 Comments on FOSS Fridays: MacGyvered Key/Value in Symfony

On a project we’re currently working on, we arrived at a situation where our client had a loose and very fluid idea of the information he wanted to store about certain objects in his application. We didn’t specifically know the number of fields or the format of the data. Continually modifying the schema would of been painful so I wanted to try something different.

Since the data is more or less non-relational (it only relates to the object that owns it), what I really wanted was an ad-hoc key/value store. But I didn’t want to break Propel’s ORM abstractions. I still wanted to be able to do:

$company->getMission();

With the new system.

Turns out you basically can. Here’s how it works:

  1. Add a “dynamic_field” table to your schema. (definition is below)
  2. Override the __call(), hydrate(), and save() functions in Propel model file that you want to MacGyver.
  3. Pray.

Definition of the dynamic_field table:

So the idea is we want to basically build a Propel Behaviour to capture any undefined get/set calls and “get” the data out of the dynamic_field table or “set” the data by storing the value into the table. Since the table stores the model class and model id, the “keys” only have to be unique by model (just like Propel normally works).

Here is the code you need to add to the model file:

The code captures any undefined get/set calls and then deals with them appropriately. It won’t serialize the fields until the save() call (just like regular Propel objects). I also overloaded the hydrate() function so that the object will fetch all of its dynamic fields in one shot, as opposed one query per get.

Using the modified objects is exactly like regular Propel objects, the changes are entirely transparent except that you can get/set anything you want.

For example:

Will work even though there is no “vision” column on the company table. Magic.

There is one big problem with this trick though. Because of the Propel class hierarchy, there isn’t any way to introduce this code in one file and have other objects inherit the changes. You have to manually copy it to any model file that you want to enable it for.