Author: Ashish Datta

Last week, President Obama made headlines by suggesting that every American in school should learn how to code. Predictably, the comment sparked some heated discussion across the web from Fred Wilson’s blog to several threads on Hacker News. Surprisingly, some of the viewpoints were extremely polarized ranging from “its useless, some people will never get it” to “of course!”. Personally, I think everyone should definitely be exposed to some form of programming while they’re in school.

An inescapable reality is that in 2013 computers are a part of everyone’s personal and professional day to day. From non-technical roles in technical fields like account managers or project managers to traditionally non-technical jobs, like teachers, everyone is ultimately interacting with computers on a daily basis. With that in mind, having a basic understanding of how computing abstractions and programming work will benefit everyone. From being able to modify a VBA macro to construct a complex Gmail search query, having a basic understanding of how the pieces fit together certainly can’t hurt.

Looking back at high school, drawing an analogy between studying programming and studying a foreign language isn’t really accurate. A better analogy is really the general experience people have studying math in middle and high school. For people that don’t take a math class in college, that’ll normally be the last time they study math in an academic setting. Although most people forget most of the details they learned, they still retain the overarching fundamentals of how things like algebra and geometry work. Because of this, when people are faced with a basic math problem they generally know what they need to look up in order to solve it. Extending this, if people were introduced to basic programming early on they’d have a sense that there might be an easier way to approach certain tasks. Need to format a list of names in Excel? There might be a function for that.

So how can we make this happen? The good news is there’s already a push to make high quality, programming focused education material available to everyone. There are already dozens of masively online open course projects including Khan Academy, Coursera, and Code Academy providing free, interactive, computer science resource for everyone. The next step is pushing states and school systems to actively adopt CS education for their middle school and high school students. Hopefully it’ll prove and easy and effective step to keeping everyone competitive in an increasingly technology powered workplace.

As far as “big data” solutions go, Hive is probably one of the more recognizable names. Hive basically offers the end user an abstraction layer to run “SQL like” queries as MapReduce jobs across data that they have in HDFS. Concretely, say you had several hundred million rows of data and you wanted to count the number of unique IDs Hive would let you do that. One of the issues with Hadoop and by proxy Hive is that it’s notably difficult to setup a cluster to try things out. Tools like Whirr exist to make things easier they’re, a bit rough around the edges and in my experience hit up against “version hell”. One alternative that I’m surprised isn’t more popular is using Amazon’s Elastic Map Reduce to bootstrap a Hadoop cluster to experiment with.

Fire up the cluster

The first thing you’ll need to do is fire up an EMR cluster from the AWS backend. It’s mostly just point and click but the settings I used were:

• Termination protection? No
• Logging? Disabled
• Debugging? Off since no logging
• Tags - None
• AMI Version: 2.4.2 (latest)
• Applications to be installed:
• Hive 0.11.0.1
• Pig 0.11.1.1
• Hardware Configuration:
• One m1.small for the master
• Two m1.small for the cores

The “security and access” section is important, you need to select an existing key pair that you have access to so that you can SSH into your master node to use the Hive CLI client.

Then finally, under Steps since you’re not specifying any pre-determined steps make sure you mark “Auto-terminate” as “No” so that the cluster doesn’t terminate immediately after it boots.

Click “Create Cluster” and you’re off to the races.

Pull some data, and load HDFS

Once the cluster launches, you’ll see a dashboard screen with a bunch of information about the cluster including the public DNS address for the “Master”. SSH into this machine using the user “hadoop” and whatever key you launched the cluster with:

ashish@ashish:~$ ssh hadoop@ec2-107-20-21-245.compute-1.amazonaws.com
Linux (none) 3.2.30-49.59.amzn1.x86_64 #1 SMP Wed Oct 3 19:54:33 UTC 2012 x86_64
--------------------------------------------------------------------------------

Welcome to Amazon Elastic MapReduce running Hadoop and Debian/Squeeze.
 
Hadoop is installed in /home/hadoop. Log files are in /mnt/var/log/hadoop. Check
/mnt/var/log/hadoop/steps for diagnosing step failures.

The Hadoop UI can be accessed via the following commands: 

  JobTracker    lynx http://localhost:9100/
  NameNode      lynx http://localhost:9101/
 
--------------------------------------------------------------------------------
Last login: Thu Dec 12 02:53:54 2013 from 50.136.18.114
hadoop@ip-10-29-191-137:~$ 

Once you’re in, you’ll want to grab some data to play with. I pulled down Wikipedia Page View data since it’s just a bunch of gzipped text files which are perfect for Hive. You can pull down a chunk of files using wget, be aware though that the small EC2s don’t have much storage so you’ll need to keep an eye on your disk space.

hadoop@ip-10-29-191-137:~$ wget -r --no-parent --reject "index.html*" http://dumps.wikimedia.org/other/pagecounts-raw/2013/

Once you have some data (grab a few GB), the next step is to push it over to HDFS, Hadoop’s distributed filesystem. As an aside, Amazon EMR is tightly integrated with Amazon S3 so if you already have a dataset in S3 you can copy directly from S3 to HDFS. Anyway, to push your files to HDFS just run:

hadoop@ip-10-29-191-137:~/dumps.wikimedia.org/other/pagecounts-raw/2013$ cd ~/dumps.wikimedia.org/other/pagecounts-raw/2013/
hadoop@ip-10-29-191-137:~/dumps.wikimedia.org/other/pagecounts-raw/2013$ hadoop dfs -mkdir /mnt/pageviews-2013-12
hadoop@ip-10-29-191-137:~/dumps.wikimedia.org/other/pagecounts-raw/2013$ hadoop dfs -put * /mnt/pageviews-2013-12/

Build some tables, query some data!

And finally, it’s time to query some of the pageview data using Hive. The first step is to let Hive know about your data and what format it’s stored in. To do this, you need to create an external table that points to the location of the files that you just pushed to HDFS. Start the Hive client by running “hive” and then do the following:

hadoop@ip-10-29-191-137:~/dumps.wikimedia.org/other/pagecounts-raw/2013$ hive

Logging initialized using configuration in file:/home/hadoop/.versions/hive-0.11.0/conf/hive-log4j.properties
Hive history file=/mnt/var/lib/hive_0110/tmp/history/hive_job_log_hadoop_21902@ip-10-29-191-137.ec2.internal_201312120412_1458262789.txt
hive> CREATE EXTERNAL TABLE page_views (
    >     project STRING, title STRING,
    >     req_count STRING, pg_size STRING
    > ) ROW FORMAT DELIMITED 
    >    FIELDS TERMINATED BY ' '
    >    LINES TERMINATED BY '\n'
    >    STORED AS TEXTFILE
    > LOCATION '/mnt/pageviews-2013-12';
OK
Time taken: 0.313 seconds

Now select some data from your newly created table!

hive> select * from page_views limit 10; 
OK
*.b	Ingl\xC3\x01\x00\x00\x00\x00Z\xBB\xB9\x01\x00\x00\x00\x00;kGS\xF6\xC3aH\x0D\x00\x00	1	325
*	100013-11-30T23:59:57.3	1	325
*	100\x00\x00\x00\x00\x00\x00\x00\xA1\x0D\xA1\x01\x00\x00\x00\x00\x10\x00\x00\x00\x00	1	325
*	\x5Cx-\x22\x5Cx/(\x5Cx/*\x5Cx/(\x5Cx/\x5Cx/0\x82\x01\xB3\x06\x09`\x86H\x01\x86\xFDl\x01\x010\x82\x01\xA40:\x06\x08+\x06\x01\x05\x05\x07\x02\x01\x16.http://www.digicert.com/ss	1	325
*	\x5Cx-\x22\x5Cx/(\x5Cx/*\x5Cx/(\x5Cx/\x5Cx/\x00\x00\x00\x00\xDE~\xC4\xD1\x91\xA5\x09\x00\x0A\x00\x00\x00\x00\x00\x00\x00\x19%~\x01\x00\x00\x00\x00\x0A\x00\x00\x00\x00\x00\x00\x00%%~\x01\x00\x00\x00\x004Qm\x01\x00\x00\x00\x00\x08\xB5'\xAB\x00\x00\x00	1	325
AR	%D9%82%D8%A7%D8%A6%D9%85%D8%A9_%D8%A3%D9%84%D8%B9%D8%A7%D8%A8_%D8%A5%D9%8A_%D8%A2%D9%8A%D9%87_%D9%84%D9%88%D8%B3_%D8%A3%D9%86%D8%AC%D9%84%D9%88%D8%B3	1	44480
De.mw	De	1	10302
De	Customer-Relationship-Management	3	96858
De	Include	3	24679
EN.mw	EN	1	4693
Time taken: 14.766 seconds, Fetched: 10 row(s)

Pretty sweet huh? Now feel free to run any arbitrary query against the data. Note: since we used m1.small EC2s the performance of Hive/Hadoop is going to be pretty abysmal. But hey, give it a shot:

hive> select count(*) AS c, title from page_viewsn group by title order by c desc limit 1000;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 6
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>

Anyway, don’t forget to tear down the cluster once you’re done. As always, let me know if you run into any issues!

We’ve been using Hive a bit lately to help clients tackle some of their data needs and without a doubt one of the most powerful features is Hive’s SerDe functionality. Taking a step back, Hive is an open source Apache project that lets you run “SQL Like” queries using Hadoop on data that you have in HDFS. It’s a lot of moving pieces but what it fundamentally comes down to is that Hive will let you run what look like SQL queries across the text files that you have in HDFS. A typical use case would be using Hive to run ad-hoc queries across web server (like nginx) logs. Want to a breakdown of response times by frontend web server? Hive would let you do that.

SerDe?

SerDe is actually short for Serialize/Deserialize and its the mechanism that Hive uses to make sense of your text files in HDFS. Lets take a typical nginx log line:

- - 66.249.66.164 - - [09/Dec/2013:22:11:49 -0500] "GET /feed/ HTTP/1.1" 200 5281 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"

Now the magic comes in how Hive uses a SerDe to translate a line like that into something that’s queryable. This is contrived but lets assume that for some reason we’re interested in querying on the client IP address and the request size of each log line. So we’d be interested in creating a table that looks like:

ip, request_size
123.125.71.51, 16060
54.224.19.57, 30075
180.76.5.18, 6971

Turns out, Hive makes this particularly easy. You’d end up using the RegexSerDe to match a regular expression and then extract the two fields you’re interested in.

A custom one

The next step after extraction is to do some transformation during the extraction stage and this is where the custom SerDe comes in. For example, lets say that you wanted to geocode the client’s IP address and also convert your dates into Unix timestamps. So your table would be something like:

ip, city, state, country, request_size, timestamp
123.125.71.51, Cambridge, MA, US, 16060, 1386645919
54.224.19.57, Cambridge, MA, US, 30075, 1386645919
180.76.5.18, Cambridge, MA, US, 6971, 1386645919

Your custom SerDe would let you do exactly this. You’d be able to use something like the MaxMind database to geocode your IP addresses and then use some extra Java to convert your timestamps.

Unfortunately, there doesn’t seem to be too much documentation on how to actually write a custom class so here’s a couple of tidbits I’ve picked up:

• It looks like at some point the SerDe class was refactored so depending on what Hive version you’re using you’ll need to extend a different class. On Hive 0.11 the class you’ll want to extend is “org.apache.hadoop.hive.serde2.SerDe”
• You’ll need to include a couple of JARs in order to get the class to build. I had to include commons-logging-1.0.4.jar, hadoop-0.20.1-core.jar, hive-contrib-0.10.0-cdh4.4.0.jar, hive-exec-0.10.0-cdh4.4.0.jar, junit-4.5.jar
• As noted above, you need to pull the specific versions of the JARs that you’re going to end up running this SerDe against
• Make sure you target the right Java JRE version. If your servers are running Java 1.6 and you target 1.7 you end up getting really cryptic error messages.
• If you create a table using your SerDe, you’ll need to have that JAR available to drop that table

The best way I’ve found to bootstrap this is to create an Eclipse project, include the necessary JARs, and then get the RegExSerDe to build inside the project. Once that works, test the JAR by creating a table using it and then you’ll be able to modify the class from there.

Even with my awful Java, the RegexSerDe class was easy enough to grok and then modify as needed.

Stuck? Need Help?

Drop me a comment or shoot me an email and I’ll do my best to help you out.

Note: I haven’t actually tried this in production, it’s probably a terrible idea.

We’ve been using MySQL temporary tables to run some analytics lately and it got me wondering how difficult would it be to hydrate Doctrine2 objects from these tables? We’ve primarily been using MySQL temporary tables to allow us to break apart complicated SQL queries, cache intermediate steps, and generally make debugging analytics a bit easier. Anyway, given that use case this is a bit of a contrived example but it’s still an interesting look inside Doctrine.

For arguments sake, lets say we’re using the FOSUserBundle and we have a table called “be_user” that looks something like:

"id","username","enabled","first_name"
"161965","8857049215","1","Tom"
"161964","7783806403","1","Larry"
"161963","1702340214","1","Mark"
"161962","6140583379","1","Nick"
"161961","5474626482","1","Wes"
"161960","2335083853","1","Mark"
"161959","6807578353","1","Mark"
"161958","5928840701","1","Mark"
"161957","1695583227","1","Mark"
"161956","1276759998","1","Mark"
"161955","1714401966","1","Mark"
"161954","9967684636","1","Mark"
"161953","4778383968","1","Mark"
"161952","9616931158","1","Mark"
"161951","8597344302","1","Mark"
"161950","4661522024","1","Mark"
"161949","8085406700","1","Mark"
"161948","1766644457","1","Mark"
"161947","3052034756","1","Mark"
"161946","2151235620","1","Mark"
"161945","6028556862","1","Mark"
"161944","4428949926","1","Mark"
"161943","8650207427","1","Mark"
"161942","6847827313","1","Mark"
"161941","9572768113","1","Mark"
"161940","3230071069","1","Mark"
"161939","7336484313","1","Mark"
"161938","9209189107","1","Mark"
"161937","8708069230","1","Mark"
"161936","7537175130","1","Mark"

Now, for some reason we’re going to end up creating a separate MySQL table (temporary or otherwise) with a subset of this data but identical columns:

<?php

$sql = "CREATE TEMPORARY TABLE active_users AS (SELECT * FROM be_user WHERE enabled = 1 AND LOCATE('968', username) > 0)";
$doctrine->getEntityManager()->getConnection()->query( $sql );

So now how do we load data from this secondary table into Doctrine2 entities? Turns out it’s relatively straightforward. By using Doctrine’s createNativeQuery along with ResultSetMapping you’ll be able to pull data out of the alternative table and return regular User entitites. One key point, is that by using DisconnectedClassMetadataFactory it’s actually possible to introspect your Doctrine entities at runtime so that you can add the ResultSetMapping fields dynamically.

Anyway, my code inside a Command to test this out ended up looking like:

<?php

use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
use Doctrine\ORM\Tools\Console\MetadataFilter;
use Doctrine\ORM\Tools\EntityGenerator;
use Doctrine\ORM\Tools\DisconnectedClassMetadataFactory;
use Doctrine\ORM\Query\ResultSetMapping;

protected function execute(InputInterface $input, OutputInterface $output) {

  $doctrine = $this->getContainer()->get("doctrine");
  $em = $doctrine->getEntityManager();

  // This lets us introspect Doctrine entities
  $cmf = new DisconnectedClassMetadataFactory();
  $cmf->setEntityManager($em);
  
  // Get the entity data
  $classMetadata = $cmf->getMetadataFor("CT\\BEBundle\\Entity\\User");

  $rsm = new ResultSetMapping();
  $rsm->addEntityResult('CT\\BEBundle\\Entity\\User', 'u');
  
  // Add the field info for each column/field       
  foreach( $classMetadata->fieldMappings as $id => $obj ){
    $rsm->addFieldResult('u', $obj["columnName"], $obj["fieldName"]);
  }	    

  $res = $doctrine->getEntityManager()
                ->getRepository("CTBEBundle:User")
                ->createQueryBuilder("u")
                ->select("COUNT(u.id) AS u_cnt")
                ->getQuery()
                ->getResult();

  $output->writeln("Total users: " . $res[0]["u_cnt"]);

  // Select some data into the other table
  $sql = "CREATE TEMPORARY TABLE active_users AS (SELECT * FROM be_user WHERE enabled = 1 AND LOCATE('968', username) > 0)";
  $doctrine->getEntityManager()->getConnection()->query( $sql );

  $query = $doctrine->getManager()->createNativeQuery('SELECT * FROM active_users', $rsm);
  $users = $query->getResult();

  $output->writeln("Active users: " . count($users));
}

A few days ago, I was browsing my Feedly dashboard and ran across this AdWeek post describing how big retailers are gearing up to poach their competitors customers this holiday season. The article goes into some specifics, but the idea is basically that brands are planning to monitor Twitter for relevant conversations and then “at” message potential customers with special offers, product details, or even local store inventory information.

So imagine @MikeBruins65 from Boston tweeting “Wtf! @BestBuy offering 25% off all 4K TVs in-store…except nothing in stock.” and then @target replying “Cheer up @MikeBruins65! We have 4K TVs in-stock in Everett, MA! Grab coupons at http://bit.ly/target-4k-ma”. Since these brands are certainly leveraging powerful tools like Radian6 or even the full Twitter Firehose, it seems like it would be straightforward for them to execute strategies like this around high value markets. But what about as an individual, could you employ a similar strategy to make a few bucks?

Amazon Associates Links

The most obvious, least risky, and least lucrative approach would be to monitor Twitter for tweets that sounded like they were from frustrated buyers and then message them Amazon associates links for the product they're looking for. Looking at Amazon's fee structure, you'd want to target high margin categories with moderately expensive products and then hopefully end up doing a decent amount of volume. So imagine searching for Tweets from users frustrated that they can't checkout on a small eCommerce site, finding the product they're searching for on Amazon, and then Tweeting them the link to buy with your Associates link.

Dropshipping

More risky and potentially more upside. I'm not entirely sure how feasible this would be, but I think the idea would be to use a SaaS eCommerce platform like Shopify to setup an eCommerce shop and then dynamically list items which you'll later dropship. The challenge would be two fold, using Twitter to identify which previously obscure items are starting to trend and then figuring out how to introduce enough margin so that you end up profiting on the sale. It might be feasible though, with the explosion of small, boutique eCommerce sites it might be possible to negotiate a "I'll buy 400 for 50% off!" type deal quickly enough to introduce a profitable sale. The bigger challenge would probably be identifying these items as they start trending, but that could be solved by….

Pinterest

Recent member of the billion dollar boys club and frequent target of "haters", it's current traction and latent purchase intent potentially make it the perfect place for affiliate marketing. Beyond that, the wealth of potential gift pins and the follower/repin graph might hold the key to identifying relatively obscure products right before they begin to go viral. Anyway, I don't have any concrete ideas on how you could leverage Pinterest but it definitely seems like the ingredients for success are there.

Totally coincidentally, this article just came across TechCrunch - A Pin On Pinterest Is Worth 25% More In Sales Than Last Year, Can Drive Visits & Orders For Months

Anyway, are any of these actually feasible? Who knows, but I’d love to hear any other ideas.