{5} Setfive - Talking to the World - Page 24 of 66 - Ramblings on code, startups, and everything in between

In the last two weeks, there’s been two frontpage stories coming out of the “sharing economy” space. First up, news broke that Seattle passed new regulation to limit the number of drivers that Uber or Lyft can have on the road at any time, effectively hamstringing both services. Then, over the weekend a story started circulating about an Airbnb stay gone awry involving an orgy, Twitter, and of course the cops. While the stories are wildly different, they both sit at the intersection of the new “sharing economy” and the role of government regulation. Because of this, both stories sparked an intense debate everywhere from The Wall Street Journal to Hacker News. The attitudes and viewpoints of the discourse were interesting and revealing about people’s attitudes towards regulation in the taxi and hospitality space.

The opinion towards the new regulations in Seattle were overwhelmingly negative, ranging from claims of stifling innovation to accusations of outright corruption. Empirically, it seems that most people, even outside of early adopters, have generally positive feelings about Uber and Lyft. It might be because of the horrible experiences people have had in normal cabs or because of the perception of hackney companies as entrenched monopolies but I think it’s primarily due to people’s perception of risk surrounding Uber or Lyft. As a non-user, the perception of how “risky” it is to have Uber drivers operating in your city is probably near zero. Given how small a percentage of total drivers they’ll make up and that “licensed cab drivers” typically already operate in the area, I don’t think many people feel threatened by having additional drivers potentially ferrying people around their city. Because of this, it’s been easier for people to take hold of the “sharing economy” narrative where Lyft or Uber who were empowered to make a living are suddenly shut out by corrupt, entrenched interests.

Contrast this with the reactions from the same people to the Airbnb story, which ranged from disgust that someone would rent out their condo to strangers through feelings that Airbnb should be held liable for the actions of an independent third party. There’s no argument that the Airbnb story is significantly more disturbing, but it isn’t the first time something like this has happened and it certainly won’t be the last. So why such a different, visceral reaction? It’s perceived risk. As a non-user, the perceived risk to having Airbnb operate in your area is undeniably significant. Take a typical condo apartment building where every occupant is either a member of a condo association or a vetted rental tenant. Introducing the possibility of short term, “random occupants” certainly sounds risky and unnerving to anyone living in the building. Anyone considering the situation immediately evaluates worst case scenarios, “what if they’re criminals?” or “drug dealers?” and so on. Compared to driving, where interactions with strangers is a given, introducing “random interactions” into a scenario where it’s unexpected seems to push people towards favoring legislation.

As a whole, the emergence of the new “sharing economy” is probably a net positive. Despite that, companies are certainly thumbing their nose at government regulation and entrenched players which is going to cause ruffled feathers along the way. To win the hearts and minds, companies will definitely need to manage the perception of how risky their services are both to users and non-users alike.

I was working on a Phonegap app recently and one of the goals was to allow the user to record some feedback as an audio file which would later be shared. If you hit Google, there’s plenty of StackOverflow threads which claim to have working sample code. Almost all of the SO code is using the Phonegap Media API with the “startRecording”/”stopRecording” functions. Despite it looking simple, I could never get the Media based code working on iOS. For whatever reason, I kept hitting iOS permission errors trying to create the recording file. On Android, the code works but I kept ending up with an “AMR” file instead of the expected 3gp recording.

So what’s the alternative? Turns out there’s another Phonegap API called Capture which has a captureAudio method which lets you record audio. As opposed to Media, captureAudio uses the device’s native audio recording UI to let you to record audio. Because of this, captureAudio won’t work for all use cases but the implementation is straightforward and consistent across Android and iOS.

Anyway, here’s the code to record some audio and upload it to a server:

Then, on the server you can just grab the file from say the $_FILES[“file”] variable.

I was catching up with a friend of mine yesterday who’s looking to build a company in the wearables space and we started chatting about fast the wearables has been growing. The conversation stuck with me and as I left lunch I started wondering what the next big “tools” markets were going to be. The “tools” metaphor is referring to the observation that during a gold rush it’s usually more profitable to sell the pickaxes, wheelbarrows, and other supplies that miners need versus prospecting for gold yourself. Chris Dixon has an interesting post describing this phenomenom that’s worth a read, Selling pickaxes during a gold rush. Some recent examples would include the growth of collaborative open source development fostering GitHub and the shift to “cloud infrastructure” spawning PaaS companies like Heroku. Anyway, so what areas might end up creating $1bn+ tools markets?

IoT and Wearables

2014 might well go down as the year of The “Internet of things” (IoT), everyone is buzzing about it, everyone wants to leverage it, and everyone is a bit confused by it. The market is still immature but there’s already a flurry of competing standards and technologies. Looking just at connectivity, developers could potentially dealing with NFC, RFID, and Bluetooth LE. Given this early fragmentation and the wide range of potential applications, I think it’s a good bet that the IoT tools market will grow quickly over the course of the year. Locally, ThingWorx has already had a succesful exit and the Boston Business Journal is already throwing around nicknames.

On the consumer side, wearables is already a large market and its only projected to grow larger. Currently, the “activity tracker” space is fairly consolidated but that’ll certainly change as devices emerge to track different metrics through different technologies. The net result of this is that anyone looking to aggregate data from a heterogeneous set of devices will face an uphill battle. To combat this, we’ll definitely see tools emerging to help manage this complexity and create uniform interfaces. RunKeeper’s Health Graph is an early player here and they’ll certainly continue to innovate.

Cryptocurrencies

Bitcoin (and *coin) baby! Even though an $8bn market cap isn’t enough to buy WhatsApp, it’s certainly nothing to sneeze at. At this point, it’s still to early to declare that Bitcoin is “here to stay” but it’s definitely going to hang out for a bit. Given its immense disruptive potential and the archaic nature of existing financial infrastructure software, it’s almost a certainty that hugely successful “tool” companies will be built in the cryptocurrency space. From the “Bitcoin” version of payment processors like Braintree to electronic brokerage software like Interactive Brokers I think we’re going to see dozens of interesting cryptocurrency companies.

SaaS cloud wrangling

In the last few years, the number of SaaS products a typical company uses has grown exponentially. Nearly every function in a typical company has been influenced by SaaS products, from marketing and sales to accounting and strategy planning everyone’s data is now “in the cloud”. Despite the the availability of APIs, it’s become increasingly difficult to extract, manipulate, and analyze all the data stored within cloud services. Ad-hoc reports that used to involved combining a few Excel sheets now might require costly custom development. Tom Rikert of a16z has a great post describing the early stages of companies addressing this market and more will certainly follow suit. After the groundwork is laid, we’ll certainly see Google Now style “smart insights” to help companies discover new opportunities and insights.

Making predictions is always risky business and hopefully I don’t look back with these and facepalm. Anyway, would love to hear what anyone thinks in the comments.

On a few of our projects we have a few different needs to either queue items to be processed in the background or we need a single request to be able to process something in parallel. Generally we use Gearman and the GearmanBundle. Let me explain a few different situations where we’ve found it handy to have Gearman around.

Background Processing

Often we’ll need to do something which takes a bit more time to process such as sending out a couple thousand push notifications to resizing several images. For this example lets use sending push notifications. You could have a person sit around as each notification is sent out and hope the page doesn’t timeout, however after a certain number of notifications, not to mention a terrible user experience, this approach will fail. Enter Gearman. With Gearman you are able to basically queue the event that a user has triggered a bunch of notifications that need to be processed and sent.

What we’ve done above is sent to the Gearman server a job to be processed in the background which means we don’t have to wait for it to finish. At this point all we’ve done is queued a job on the Gearman server, Gearman itself doesn’t know how to run the actual job. For that we create a ‘worker’ which reads jobs and processes them:

The worker will consume the job and then process it as it sees fit. In this case we just loop over each user ID and send them a notification.

Parallel Processing

One one of our applications users can associate their account with multiple databases. From there we go through each database and create different reports. On some of the application screens we let users poll each of their databases and we aggregate the data and create a real time report. The problem with doing this synchronously is that you have to go to each database one by one, meaning if you have 10 databases and each one takes 1 seconds to get the data from, you have at least ten seconds the user is waiting around; this doesn’t go well when you have 20 databases and so on. Instead, we use Gearman to farm out the task of going to each database and pull the data. From there, we have the request process total up all the aggregated data and display it. Now instead of waiting 10 seconds for each database, we farm out the work to 10 workers, wait 1 second and then can do any final processing and show it to the user. In the example below for brevity we’ve just done the totaling in a controller.

What we’ve done here is created a job for each connection. This time we add them as tasks, which means we’ll wait until they’ve completed. On the worker side it is similar to except you return some data, ie `return json_encode(array(‘total’=>50000));` at the end of the the function.

What this allows us to do is to farm out the work in parallel to all the databases. Each worker runs queries on the database, computes some local data and passes it back. From there you can add it all together (if you want) and then display it to the user. With the job running in parallel the number of databases you can process is no longer limited on your request, but more on how many workers you have running in the background. The beauty with Gearman is that the workers don’t need to live on the same machine, so you could have a cluster of machines acting as ‘workers’ and be able to process more database connections in this scenario.

Anyways, Gearman has really made parallel processing and farming out work much easier. As the workers are also written in PHP, it is very easy to reuse code between the frontend and the workers. Often, we’ll start a new report without Gearman; getting logic/fixing bugs in a single request without the worker is easier. After we’re happy with how the code works, we’ll move the code we wrote into the worker and have it just return the final result.

Good luck! Feel free to drop us a line if you need any help.

Earlier this week, a buddy of mine reached out asking for a good solution to programmatically taking screenshots of a few thousand URLs. For whatever reason, this question seems to come up ever so often so I pointed him towards PhantomJS and figured he’d be on his way. Wrong. Not one to pass up free beer and the opportunity to learn something I agreed to write up the script to generate screenshots from a list of URLs.

Looking at PhantomJS, it seems relatively straightforward but it’s clear you’d really need something “else” to orchestrate this entire process. After some poking around, NodeJS, everyone’s favorite hipster runtime, seemed to be the obvious choice. There’s a handful of node modules that basically “bridge” node with phantom and allow a node script to asynchronously manipulate a PhantomJS instance. Based solely on the funny description I decided to run with phantomjs-node and was off to the races.

Getting everything setup was straightforward enough but then as I started looking at the phantomjs-node examples I started realizing this was a one way trip to callback soup. I’ve been doing some PhoneGap work recently and using jQuery’s Deferreds has significantly help keep the project from becoming a mess of callbacks. On the NodeJS side, it looks like there’s two functionally equivalent implementations but I decided to run with Q since the “wrapper” function names are shorter.

The Code

Anyway, the main problem we’re trying to address is that with multiple nested callbacks code becomes particularly difficult to follow. It’s hard to read, hard to trace control flow, and obviously hard to debug. Take for example, the phantomjs-node example:

It’s already THREE callbacks deep and all it’s done is initialize PhantomJS and load a page. Imagine layering on a few more asynchronous operations, doing all of this in a loop, and then some post-processing. Enter Deferreds. How To Node has an eloquent explanation of what deferreds are and how Node impliments them but in a nutshell they’re useful for making asynchronous code easier to follow.

The main issue I ran into using Q was that “Q.ninvoke” and “Q.npost” wrapper functions kept causing exceptions while using them with phantomjs-node. What I ended up doing instead was creating my own Deferreds in separate functions and then resolving them inside a normal callback.

My PhantomJS-node code ended up looking like:

It’s without a doubt easier to follow and would also make it much easier to do more complicated PhantomJS related tasks.

So how did screenshotting a few thousand domains go? Well there’s a story about that as well…

Musing: Thoughts on Seattle vs. Uber and an AirBnB orgy

Phonegap: Recording audio with Phonegap

Musing: What’s the next $1bn+ “tools” market?