{5} Setfive - Talking to the World - Page 59 of 66 - Ramblings on code, startups, and everything in between

For the last few weeks we’ve been working on a project that involved dealing with bills in the US House and Senate. Naturally, I decided it was time to make a word cloud from the frequencies of the words in the bills!

Checkout the final product here.

I decided to use only the bills from the 111th congress (the current one), all the bills (6703 of them) were downloaded from the THOMAS library at http://thomas.loc.gov/home/gpoxmlc111/ The files are XML documents that have the full text of the bills along with some meta data.

Not really to many files but I decided to use Hadoop and try and Map/Reduce the bills to count up the word frequencies. Getting Hadoop to run locally was pretty straightforward – just tell it where JAVA_HOME is and I was off to the races. Fortunately enough, one of the pre-canned examples was a word frequency counter so I decided to modify that for what I wanted.

The example map/reduce was written to process plain text files so I had to modify it to work with the XML documents. What this involved was writing a custom InputFormat class to open each bill, extract the appropriate plain text from the XML, and then pass this back as the “data”. I also modified the word counter to ignore words shorter than 6 characters.

I tested locally with a small subset of bills and everything seemed to be working fine. The trouble started when I tried to bring up Daum’s machine as a slave to my machine. After some finagling and hair pulling I finally got it working. The takeaways were:

You can’t run your DataNode on localhost, it needs to be your computer’s hostname to accept connections.
Hostnames are important. If you don’t have a DNS server make sure your hostnames are aliased in /etc/hosts
If your HDFS set up is showing 100% utilization but you know it isn’t true, try rm’ing the data file and then re-formatting your namenode.
If a copy or reduce step fails in distributed mode the error messages are usually really cryptic – check the actual logs.
When something throws an exception during a map or reduce operation, the error won’t be reported to STDOUT

Anyway, it was a slightly frustrating but rewarding experience – I even got to code some Java! The visualization of the word frequencies is here.

Might be about time to process one of the Amazon datasets with EC2…

The other night I thought it’d be helpful to see how people browse your site. I think you can probably learn a lot about how a user moves over your site. You can tell a lot about your user’s experience by watching their mouse. You can see where they look on the page for specific information. I’ve created a demo of the tracking.

I know that there are some products out there that already do user session tracking and replays. Also there are click heat maps which are interesting when you are looking on your site to see what links the user clicks the most. I decided to just rebuild the session replay just out of curiousity on how difficult it’d be to do. It was fairly simple and took me only 15-20 minutes. There are a number of improvements you could make to the script such as the window.unload handling is not 100% depending on your browser. You could also do much more parsing on the client side of the information by using JSON. If you wanted to store more than one page of tracking data you could quickly modify the script to pass the name page which it was tracking and to store the data separately for each page.

To use the script all you need to do is add a little Javascript on the bottom of the page you want to track a user. The script tracks a users mouse movement as soon as they open the site. It keeps track of time so that during the replay you can get the proper mouse movement at the right times to replay the users session. While the user moves their mouse it continues to store all the data client side. Once the window is closed it sends all the information to the server. The server simply parses the data string. For session replay it is done via setTimeout and it moved an image(of a cursor) at different intervals to simulate the users session.

While the script is not very pretty it was written very quickly and just as a proof of concept. It goes to show you can easily track your user’s session without having to purchase expensive products, and that it can be done fairly simply.

The code is below with descriptions of what each snippet does. The script uses jQuery. To deploy this on numerous pages all you really would need to add would be a script tag that pulls in the tracking javascript.

Tracking the users movements and sending the server the information javascript:

var points='';
var timeSeconds=0;
$(document).ready(function(){
  $('body').append('');
  timer();
   $().mousemove(function (e){
    points=points+e.pageX+","+e.pageY+","+timeSeconds+"|";
  });
  $(window).unload(function(){
    sendData();
  });
});
function timer()
{
  timeSeconds=parseInt(timeSeconds)+1;
  setTimeout("timer()",10);
}
function sendData(){
    $.post('index.php','data='+points);
}

To parse the information on the server:

if($_SERVER['REQUEST_METHOD']=='POST')
{
  $fp=fopen($_SERVER['REMOTE_ADDR'].'tracking.dat','w+');
   fwrite($fp,$_POST['data']);
  fclose($fp);
  exit(1);
}

To replay the session first get the data:

if(file_exists($_SERVER['REMOTE_ADDR'].'tracking.dat'))
{
 $data=explode("|",file_get_contents($_SERVER['REMOTE_ADDR'].'tracking.dat'));
}

The moving of the cursor image javascript:

function moveMouse(x,y){
 $('#cursor').attr('style','position:absolute;left:'+x+"px;top:"+y+"px;");
}

Create different calls for each time the mouse was moved and have them execute at the times the user moved the mouse:

foreach($data as $d)
{
  $parts=explode(',',$d);
  if(count($parts)==3)
  echo 'setTimeout("moveMouse('.$parts[0].','.$parts[1].')",'.($parts[2]*10).");\n";
}

And you are all set. As I said the script is only for proof of concept and not too pretty. Let me know if you have any questions.

We use jQuery on almost every project we do. As many know updating your theme for your website widgets can take a long time. Recently we found the jQuery UI – ThemeRoller. This allows you to quickly skin all of your jQuery UI widgets within a matter of couple of mouse clicks. For those of us who can’t pick matching colors for their life, ThemeRoller has many template themes. ThemeRoller allows you to start with a templated theme, and to easily modify it via the GUI.

This will save you time and money as hand editing the CSS files to update your jQuery UI widgets is slow and tedicious.

In the past we have always used Propel as our main ORM for our Symfony projects. Recently with Doctrine becoming the default ORM for Symfony 1.3 we decided we should make sure our plugin supports Doctrine. For those who don’t know about the plugin, it automatically creates ignores for your Source Code Management(SCM) if you use Git or CVS. When you have a large project it gets tiring to create all the different ignores for all the bases, logs, configuration files and such.

Let us know if you find any problems with Doctrine support or have any additional suggestions for the plugin.

More information on the plugin can be found on the Symfony Plugins site: http://www.symfony-project.org/plugins/sfSCMIgnoresTaskPlugin

You can download the most recent pear package manually at: http://plugins.symfony-project.org/get/sfSCMIgnoresTaskPlugin/sfSCMIgnoresTaskPlugin-1.0.3.tgz

or you can install it via:

./symfony plugin:install sfSCMIgnoresTaskPlugin

On a project we’re currently working on, we arrived at a situation where our client had a loose and very fluid idea of the information he wanted to store about certain objects in his application. We didn’t specifically know the number of fields or the format of the data. Continually modifying the schema would of been painful so I wanted to try something different.

Since the data is more or less non-relational (it only relates to the object that owns it), what I really wanted was an ad-hoc key/value store. But I didn’t want to break Propel’s ORM abstractions. I still wanted to be able to do:

$company->getMission();

With the new system.

Turns out you basically can. Here’s how it works:

Add a “dynamic_field” table to your schema. (definition is below)
Override the __call(), hydrate(), and save() functions in Propel model file that you want to MacGyver.
Pray.

Definition of the dynamic_field table:

So the idea is we want to basically build a Propel Behaviour to capture any undefined get/set calls and “get” the data out of the dynamic_field table or “set” the data by storing the value into the table. Since the table stores the model class and model id, the “keys” only have to be unique by model (just like Propel normally works).

Here is the code you need to add to the model file:

The code captures any undefined get/set calls and then deals with them appropriately. It won’t serialize the fields until the save() call (just like regular Propel objects). I also overloaded the hydrate() function so that the object will fetch all of its dynamic fields in one shot, as opposed one query per get.

Using the modified objects is exactly like regular Propel objects, the changes are entirely transparent except that you can get/set anything you want.

For example:

Will work even though there is no “vision” column on the company table. Magic.

There is one big problem with this trick though. Because of the Propel class hierarchy, there isn’t any way to introduce this code in one file and have other objects inherit the changes. You have to manually copy it to any model file that you want to enable it for.

Words of Congress: Fun with Hadoop

FOSS Fridays – Tracking Your Users

Skinning your jQuery UI Components quick and easily – ThemeRoller

FOSS Fridays: sfSCMIgnoresTaskPlugin version 1.0.3 released – Doctrine Supported

FOSS Fridays: MacGyvered Key/Value in Symfony