Author: Ashish Datta

I was working on a Phonegap app recently and one of the goals was to allow the user to record some feedback as an audio file which would later be shared. If you hit Google, there’s plenty of StackOverflow threads which claim to have working sample code. Almost all of the SO code is using the Phonegap Media API with the “startRecording”/“stopRecording” functions. Despite it looking simple, I could never get the Media based code working on iOS. For whatever reason, I kept hitting iOS permission errors trying to create the recording file. On Android, the code works but I kept ending up with an “AMR” file instead of the expected 3gp recording.

So what’s the alternative? Turns out there’s another Phonegap API called Capture which has a captureAudio method which lets you record audio. As opposed to Media, captureAudio uses the device’s native audio recording UI to let you to record audio. Because of this, captureAudio won’t work for all use cases but the implementation is straightforward and consistent across Android and iOS.

Anyway, here’s the code to record some audio and upload it to a server:

navigator.device.capture.captureAudio(function(mediaFiles){

    // mediaFiles will be an array of MediaFile objects

    var options = new FileUploadOptions();
    var ft = new FileTransfer();
    var fileURI = mediaFiles[0].fullPath;

    options.params = _.extend({}, {id: 5);
    options.fileKey = "file";
    options.fileName = fileURI.substr(fileURI.lastIndexOf('/') + 1);
    options.headers = {Connection: "close"};

    // upload the file to the server
    ft.upload(fileURI, encodeURI("http://www.setfive.com/upload.php"), function(resp){
        var data = JSON.parse(resp.response);
        alert( data.successMsg );        
    }, function(error){
        alert("Sorry! Something went wrong. Please try again.");
    }, options);

}, {limit: 1});

Then, on the server you can just grab the file from say the $_FILES[“file”] variable.

I was catching up with a friend of mine yesterday who’s looking to build a company in the wearables space and we started chatting about fast the wearables has been growing. The conversation stuck with me and as I left lunch I started wondering what the next big “tools” markets were going to be. The “tools” metaphor is referring to the observation that during a gold rush it’s usually more profitable to sell the pickaxes, wheelbarrows, and other supplies that miners need versus prospecting for gold yourself. Chris Dixon has an interesting post describing this phenomenom that’s worth a read, Selling pickaxes during a gold rush. Some recent examples would include the growth of collaborative open source development fostering GitHub and the shift to “cloud infrastructure” spawning PaaS companies like Heroku. Anyway, so what areas might end up creating $1bn+ tools markets?

IoT and Wearables

2014 might well go down as the year of The "Internet of things" (IoT), everyone is buzzing about it, everyone wants to leverage it, and everyone is a bit confused by it. The market is still immature but there's already a flurry of competing standards and technologies. Looking just at connectivity, developers could potentially dealing with NFC, RFID, and Bluetooth LE. Given this early fragmentation and the wide range of potential applications, I think it's a good bet that the IoT tools market will grow quickly over the course of the year. Locally, ThingWorx has already had a succesful exit and the Boston Business Journal is already throwing around nicknames.

On the consumer side, wearables is already a large market and its only projected to grow larger. Currently, the “activity tracker” space is fairly consolidated but that’ll certainly change as devices emerge to track different metrics through different technologies. The net result of this is that anyone looking to aggregate data from a heterogeneous set of devices will face an uphill battle. To combat this, we’ll definitely see tools emerging to help manage this complexity and create uniform interfaces. RunKeeper’s Health Graph is an early player here and they’ll certainly continue to innovate.

Cryptocurrencies

Bitcoin (and *coin) baby! Even though an $8bn market cap isn't enough to buy WhatsApp, it's certainly nothing to sneeze at. At this point, it's still to early to declare that Bitcoin is "here to stay" but it's definitely going to hang out for a bit. Given its immense disruptive potential and the archaic nature of existing financial infrastructure software, it's almost a certainty that hugely successful "tool" companies will be built in the cryptocurrency space. From the "Bitcoin" version of payment processors like Braintree to electronic brokerage software like Interactive Brokers I think we're going to see dozens of interesting cryptocurrency companies.

SaaS cloud wrangling

In the last few years, the number of SaaS products a typical company uses has grown exponentially. Nearly every function in a typical company has been influenced by SaaS products, from marketing and sales to accounting and strategy planning everyone's data is now "in the cloud". Despite the the availability of APIs, it's become increasingly difficult to extract, manipulate, and analyze all the data stored within cloud services. Ad-hoc reports that used to involved combining a few Excel sheets now might require costly custom development. Tom Rikert of a16z has a great post describing the early stages of companies addressing this market and more will certainly follow suit. After the groundwork is laid, we'll certainly see Google Now style "smart insights" to help companies discover new opportunities and insights.

Making predictions is always risky business and hopefully I don’t look back with these and facepalm. Anyway, would love to hear what anyone thinks in the comments.

Earlier this week, a buddy of mine reached out asking for a good solution to programmatically taking screenshots of a few thousand URLs. For whatever reason, this question seems to come up ever so often so I pointed him towards PhantomJS and figured he’d be on his way. Wrong. Not one to pass up free beer and the opportunity to learn something I agreed to write up the script to generate screenshots from a list of URLs.

Looking at PhantomJS, it seems relatively straightforward but it’s clear you’d really need something “else” to orchestrate this entire process. After some poking around, NodeJS, everyone’s favorite hipster runtime, seemed to be the obvious choice. There’s a handful of node modules that basically “bridge” node with phantom and allow a node script to asynchronously manipulate a PhantomJS instance. Based solely on the funny description I decided to run with phantomjs-node and was off to the races.

Getting everything setup was straightforward enough but then as I started looking at the phantomjs-node examples I started realizing this was a one way trip to callback soup. I’ve been doing some PhoneGap work recently and using jQuery’s Deferreds has significantly help keep the project from becoming a mess of callbacks. On the NodeJS side, it looks like there’s two functionally equivalent implementations but I decided to run with Q since the “wrapper” function names are shorter.

The Code

Anyway, the main problem we’re trying to address is that with multiple nested callbacks code becomes particularly difficult to follow. It’s hard to read, hard to trace control flow, and obviously hard to debug. Take for example, the phantomjs-node example:

phantom.create(function(ph) {
  ph.createPage(function(page) {
    page.open('http://www.mdscollections.com/cat_mds_accessories.cfm',
      function(status) {
        console.log('Opened site? %s', status);
        another_funny(page, ph);
      });
  });
});

It’s already THREE callbacks deep and all it’s done is initialize PhantomJS and load a page. Imagine layering on a few more asynchronous operations, doing all of this in a loop, and then some post-processing. Enter Deferreds. How To Node has an eloquent explanation of what deferreds are and how Node impliments them but in a nutshell they’re useful for making asynchronous code easier to follow.

The main issue I ran into using Q was that “Q.ninvoke” and “Q.npost” wrapper functions kept causing exceptions while using them with phantomjs-node. What I ended up doing instead was creating my own Deferreds in separate functions and then resolving them inside a normal callback.

My PhantomJS-node code ended up looking like:

var Q = require("q");
var phantom = require("phantom");

var DomainScraper = function(){	
		
	this.createPhantom = function(){	
		var df = Q.defer();
		phantom.create(function(ph){
			df.resolve(ph);
		});
		return df.promise;		
	};
	
	this.createPage = function(ph){
		var df = Q.defer();
		ph.createPage(function(page, err){
			df.resolve(page);
		});
		return df.promise;
	};
	
	this.openPage = function(page, url){
		var df = Q.defer();
		page.open(url, function(status) {			
			df.resolve( {page: page, status: status} );
		});
		return df.promise;
	};
	
	this.screenshotPage = function( url ){
		
		var domainScraper = this;
		var url = "http://" + this.domainList.pop();				
		
		console.log( "Processing: " + url );
		
		var val = domainScraper
		.createPage(domainScraper.ph)
		.then(function(page){
			page.set("viewportSize", { width: 1600, height: 790 });
			return domainScraper.openPage(page, url);
		}).then(function(obj){
			
			if( obj.status == "success" ){											
				obj.page.render( url + ".png" );				
				console.log("Rendered " + url);
			}
			
			return result;
		})
		.done(function(){
			domainScraper.ph.exit();
		});
		
	};
			
	
};

var ds = new DomainScraper();
ds.screenshotPage("http://www.setfive.com");

It’s without a doubt easier to follow and would also make it much easier to do more complicated PhantomJS related tasks.

So how did screenshotting a few thousand domains go? Well there’s a story about that as well…

Over the last two weeks, there’s been two high profile negative Bitcoin incidents. First up, was Mt. Gox announcing that they were temporarily halting withdrawls and then soon after Silk Road 2.0 announcing that they been hacked and ~$2 million of BTC had been stolen. In both situations, the sites are blaming “transaction malleability”, what is supposedly a well known Bitcoin exploit, as the root cause of the issues. Predictably, most of the commentary surrounding both of these incidents has been that they’re both in fact cover ups for the site admins stealing the “lost” bitcoin. Regardless of what turns out to be true, both incidents are raising some interesting questions about bitcoin.

As I understand it, the “transaction malleability” vulnerability is an implementation specific issue that’s already been fixed in the “official” bitcoin client. This is directly contradictory to what Mt. Gox announced and one of the lead Bitcoin developers actually went as far as calling out Mt. Gox in Why Mt. Gox is full of shit. It isn’t clear if Mt. Gox is being intentionally dishonest, but this spat does raise an interesting issue of trusting the software that you’re using. Looking at the software we use on a daily basis, there’s a remarkable lack of transparency into how systems are built, if they’ve been audited, and if they’re composed of independently verifiable open source components. From the software that switches trains on tracks to the code that powers your cell phones, we generally don’t really know how the sausage was ultimately made. In general, things seem to work “OK” without consumers knowing these details but for people to be confident in Bitcoin payment systems they’ll ultimately demand transparency into the underlying implementations.

Another interesting point surfaced by this issue is the irreversibility of Bitcoin transactions. The Silk Road 2.0 announcement really highlights this, since they’re basically pleading with whoever stole the coins to “give them back”. It’s pretty clear that the inability to rollback transactions is going to make combating Bitcoin fraud a herculean task as the volume of transactions grows. Without a mechanism to “undo” a transaction, the majority of fraud prevention will have to rely on preventively blocking transactions as opposed to mediating them after the fact. There are certainly benefits to not being able to reverse transactions but Bitcoin will definitely need a strategy to combat issues like this.

Anyway, I’m still bullish on Bitcoin, the community has shown that it’s resilient and overall it’s definitely better to work out the kinks with $2 million instead of $200 million at stake. It looks like Mt. Gox is close to resuming normal activity and Silk Road 2.0 has recently announced that it’ll reimburse coins to everyone that was affected by the hack. Now if only the price would get back to $1000/coin…

In the last few months there’s been a handful blog posts basically themed “Redshift vs. Hive”. Companies from Airbnb to FlyData have been broadcasting their success in migrating from Hive to Redshift in both performance and cost. Unfortunately, a lot of casual observers have interpreted these posts to mean that Redshift is a “silver bullet” in the big data space. For some background, Hive is an abstraction layer that executes MapReduce jobs using Hadoop across data stored in HDFS. Amazon’s Redshift is a managed “petabyte scale” data warehouse solution that provides managed access to a ParAccel cluster and exposes a SQL interface that’s roughly similar to PostgreSQL. So where does that leave us?

From the outside, Hive and Redshift look oddly similar. They both promise “petabyte” scale, linear scalability, and expose an SQL’ish query syntax. On top of that, if you squint, they’re both available as Amazon AWS managed services through Elastic Mapreduce and of course Redshift. Unfortunately, that’s really where the similarities end which makes the “Hive vs. Redshift” comparisons along the lines of “apples to oranges”. Looking at Hive, its defining characteristic is that it runs across Hadoop and works on data stored in HDFS. Removing the acronym soup, that basically means that Hive runs MapReduce jobs across a bunch of text files that are stored in a distribued file system (HDFS). In comparison, Redshift uses a data model similar to PostgreSQL so data is structured in terms of rows and tables and includes the concept of indexes.

OK so who cares?

Well therein lays the rub that everyone seem to be missing. Hadoop, and by extension Hive (and Pig) are really good at processing text files. So imagine you have 10 million x 1mb XML documents or 100GB worth of nginx logs, this would be a perfect use case for Hive. All you would have to do is push them into HDFS or S3, write a RegEx to extract your data and then query away. Need to add another 2 million documents or 20GB of logs? No problem, just get them into HDFS and you're good to go.

Could you do this with Redshift? Sure, but you’d need to pre-process 10 million XML documents and 100GB of logs to extract the appropriate fields, and then create CSV files or SQL INSERT statements to load into Redshift. Given the available options, you’re probably going to end up using Hadoop to do this anyway.

Where Redshift is really going to excel is in situations where your data is basically already relational and you have a clear path to actually get it into your cluster. For example, if you were running three x 15GB MySQL databases with unique, but related data, you’d be able to regularly pull that data into Redshift and then ad-hoc query it with regular SQL. In addition, since the data is already structured you’d be able to use the existing format to create keys in Redshift to improve performance.

Hammers, screws, etc

When it comes down it, it'll come down to the old "right tool for the right job" aphorism. As an organization, you'll have to evaluate how your data is structured, the types of queries you're interested in running, and what level of abstraction you're comfortable with. What's definitely true is that "enterprise" data warehousing is being commoditized and the "old guard" better innovate or die.