internOwl Launched!

Today we are proud to unveil  internOwl.  internOwl is a site for students to research internships and find them.  As the site grows students will be able to gain invaluable insight into the quality of different internships around the country.   Currently the site is being launched with a focus on targeting Massachusetts’ students.  We are excited to see how it performs.

If you are a student in the Amherst or Northampton area you can get a FREE burrito via the following url: http://www.internowl.com/bueno

We hope you all enjoy and there will be more updates about the site to follow as well as the technology used behind the site!

FOSS Saturday: sfFbConnectGuardPlugin – sfGuard meets FB Connect

I was slaving over a hot keyboard all Friday!

But at last it is done – FBConnect for sfGuard.

Get it here http://www.symfony-project.org/plugins/sfFbConnectGuardPlugin

A detailed explanation of how to install it and use it is on the Symfony site.

Anyway, the plugin basically just introduces a new table to keep track of Facebook IDs <---> sfGuardUserIds

Here’s a fun nugget. One of the problems with using FB Connect is that you can’t mug a user’s email address from Facebook. Obviously this is a smart move on Facebook’s part but it makes life hard for my Nigerian spammer friends. If you want to snag a user’s email address (or anything else for that matter) while still using Facebook Connect here’s a sketch of how to do it.

Everything is the same except you can’t use Facebook’s FBML to render the FB Connect button. What you want to do instead is trigger the “connect” event by hand. Here is basically how we do it:

  1. The user requests to sign up.
  2. We pop up a Lightbox using Thickbox
  3. We ask the user for their email address and verify that is valid and unique via AJAX in the background.
  4. The validation routing sets an attribute on the user using setAttribute() that contains the entered email address.
  5. We close the Lightbox and initiate a Facebook Connect request with FB.Connect.requireSession
  6. In our createFbUser() method we get the attribute back and save it with the new user

Bam. Got the user’s email address and logged them in via FB Connect.

Words of Congress: Fun with Hadoop

For the last few weeks we’ve been working on a project that involved dealing with bills in the US House and Senate. Naturally, I decided it was time to make a word cloud from the frequencies of the words in the bills!

Checkout the final product here.

I decided to use only the bills from the 111th congress (the current one), all the bills (6703 of them) were downloaded from the THOMAS library at http://thomas.loc.gov/home/gpoxmlc111/ The files are XML documents that have the full text of the bills along with some meta data.

Not really to many files but I decided to use Hadoop and try and Map/Reduce the bills to count up the word frequencies. Getting Hadoop to run locally was pretty straightforward – just tell it where JAVA_HOME is and I was off to the races. Fortunately enough, one of the pre-canned examples was a word frequency counter so I decided to modify that for what I wanted.

The example map/reduce was written to process plain text files so I had to modify it to work with the XML documents. What this involved was writing a custom InputFormat class to open each bill, extract the appropriate plain text from the XML, and then pass this back as the “data”. I also modified the word counter to ignore words shorter than 6 characters.

I tested locally with a small subset of bills and everything seemed to be working fine. The trouble started when I tried to bring up Daum’s machine as a slave to my machine. After some finagling and hair pulling I finally got it working. The takeaways were:

  • You can’t run your DataNode on localhost, it needs to be your computer’s hostname to accept connections.
  • Hostnames are important. If you don’t have a DNS server make sure your hostnames are aliased in /etc/hosts
  • If your HDFS set up is showing 100% utilization but you know it isn’t true, try rm’ing the data file and then re-formatting your namenode.
  • If a copy or reduce step fails in distributed mode the error messages are usually really cryptic – check the actual logs.
  • When something throws an exception during a map or reduce operation, the error won’t be reported to STDOUT

Anyway, it was a slightly frustrating but rewarding experience – I even got to code some Java! The visualization of the word frequencies is here.

Might be about time to process one of the Amazon datasets with EC2

FOSS Fridays – Tracking Your Users

The other night I thought it’d be helpful to see how people browse your site.  I think you can probably learn a lot about how a user moves over your site.  You can tell a lot about your user’s experience by watching their mouse.  You can see where they look on the page for specific information. I’ve created a demo of the tracking.

I know that there are some products out there that already do user session tracking and replays.  Also there are click heat maps which are interesting when you are looking on your site to see what links the user clicks the most.  I decided to just rebuild the session replay just out of curiousity on how difficult it’d be to do.  It was fairly simple and took me only 15-20 minutes. There are a number of improvements you could make to the script such as the window.unload handling is not 100% depending on your browser.  You could also do much more parsing on the client side of the information by using JSON.  If you wanted to store more than one page of tracking data you could quickly modify the script to pass the name page which it was tracking and to store the data separately for each page.

To use the script all you need to do is add a little Javascript on the bottom of the page you want to track a user.   The script tracks a users mouse movement as soon as they open the site.  It keeps track of time so that during the replay you can get the proper mouse movement at the right times to replay the users session.  While the user moves their mouse it continues to store all the data client side.  Once the window is closed it sends all the information to the server.  The server simply parses the data string.  For session replay it is done via setTimeout and it moved an image(of a cursor) at different intervals to simulate the users session.

While the script is not very pretty it was written very quickly and just as a proof of concept.  It goes to show you can easily track your user’s session without having to purchase expensive products, and that it can be done fairly simply.

The code is below with descriptions of what each snippet does. The script uses jQuery. To deploy this on numerous pages all you really would need to add would be a script tag that pulls in the tracking javascript.

Tracking the users movements and sending the server the information javascript:

var points='';
var timeSeconds=0;
$(document).ready(function(){
  $('body').append('');
  timer();
   $().mousemove(function (e){
    points=points+e.pageX+","+e.pageY+","+timeSeconds+"|";
  });
  $(window).unload(function(){
    sendData();
  });
});
function timer()
{
  timeSeconds=parseInt(timeSeconds)+1;
  setTimeout("timer()",10);
}
function sendData(){
    $.post('index.php','data='+points);
}

To parse the information on the server:

if($_SERVER['REQUEST_METHOD']=='POST')
{
  $fp=fopen($_SERVER['REMOTE_ADDR'].'tracking.dat','w+');
   fwrite($fp,$_POST['data']);
  fclose($fp);
  exit(1);
}

To replay the session first get the data:

if(file_exists($_SERVER['REMOTE_ADDR'].'tracking.dat'))
{
 $data=explode("|",file_get_contents($_SERVER['REMOTE_ADDR'].'tracking.dat'));
}

The moving of the cursor image javascript:

function moveMouse(x,y){
 $('#cursor').attr('style','position:absolute;left:'+x+"px;top:"+y+"px;");
}

Create different calls for each time the mouse was moved and have them execute at the times the user moved the mouse:

foreach($data as $d)
{
  $parts=explode(',',$d);
  if(count($parts)==3)
  echo 'setTimeout("moveMouse('.$parts[0].','.$parts[1].')",'.($parts[2]*10).");\n";
}

And you are all set.  As I said the script is only for proof of concept and not too pretty.  Let me know if you have any questions.