BackboneJS: JSON over REST

We’ve recently started using BackboneJS on a couple of projects to help create more responsive UIs and avoid having to deal with a tangled mess of jQuery callbacks, ad-hoc Javascript templating, and difficult code re-use. On the whole, I’ve been impressed with Backbone and I think it’ll make a valuable asset in our toolbox.

Anyway, one of the core concepts of Backbone is that client side models will be automatically synced with their server side counterparts via RESTful AJAX calls. Stripping away the buzzword bingo, what that means is that when a Backbone model is created in the UI a AJAX POST request is initiated to create that model, updates trigger a PUT, and destroying a model will cause a DELETE.

Conceptually, this system makes a lot of sense and it fits nicely with how data is structured in most apps. Unfortunately, it falls apart when there isn’t a one to one correspondence between frontend Backbone models and your backend persistence layer.

For example, say on the frontend you wanted to allow a user to enter and edit the various languages she spoke. Naturally, you could define a Backbone “Language” model, managed by a “LanguageList” Collection, and finally drawn with a “LanguageView” View. This would work great but unfortunately on the backend you’re storing the list of languages the user speaks in a single row – say as a CSV list for arguments sake. At this point, you’d be stuck because Backbone would be pushing AJAX requests per model and your backend would have no way to combine the languages into a CSV list.

In this instance it seems like you really want “JSON over REST”, in the sense that you want to push update events for several models on a single request. The Backbone FAQ mentions this technique but unfortunately doesn’t elaborate on specifically how to achieve this behavior.

Batch operations on Models are common, but often best handled differently depending on your server-side setup. Some folks don’t mind making individual Ajax requests. Others create explicit resources for RESTful batch operations: /notes/batch/destroy?ids=1,2,3,4. Others tunnel REST over JSON, with the creation of “changeset” requests:

Google’ing around I stumbled across this StackOverflow answer – http://stackoverflow.com/questions/11298152/how-to-do-batch-operations-in-backbone-js-via-tunneling-rest-over-json which seems to have been removed. The answer basically suggested embedding a Collection within a Model so that Backbone effectively passed a JSON array to your REST endpoint.

Unfortunately, the answer didn’t provide a concrete example either so here is a stripped down version of what I ended up using.

Basically, “Language” is the “real” model which is managed by the “LanguageList” collection. Then, “LanguageCollection” is the container model which holds a single “LanguageList” collection and is subsequently managed by the “LanguageCollectionList”.

If you look through the code, basically the Views are bound to events on ‘LanguagesCollection.at(0).get(“languages”)’ which is a LanguageList object.

The other thing to note is that Language.destroy() function is updated to not make an AJAX DELETE call but instead trigger a save on the collection.

Also, LanguageCollection.parse() is defined as an empty function because ‘LanguagesCollection.at(0).get(“languages”)’ needs to remain a Backbone object and not be converted into the regular JSON which would come back from the server. The caveat here is that if you need to set an ID after a POST, you’ll need to update parse() to process the JSON and then update the corresponding models within the collection.

Anyway, I’m still a Backbone novice so any input or insight is more than welcome. I’d also love to know if theres a more straightforward way to address this. A couple of posts mentioned overriding Backbone.sync but that wasn’t feasible for me because I had other Backbone models on the page which needed to update using the regular REST pattern.

Update 7/20/2012

As pointed out by Aria below, it’s actually possible to override the “sync” method per model to achieve this JSON over REST behavior. Overall, its probably a cleaner solution since it avoids introducing the complexity of collections contained in models simply to get the JSON over REST behavior.

Here’s an example implementation of a sync function that produces the desired results:

NOTE: Almost all of that code is copied directly out of Backbone.js, the exception is the following block which collects up all the models in the current model’s collection and then adds it into the “data” field:

Happy Backbone’ing!

Use Exim and pipes to replicate MailGun routes

Over the last month or so I’ve been working with a friend of mine on a little side project (more on that later). During the course of the project, we ended up using Mailgun to receive email which was then passed to a PHP script using Mailgun routes.

Mailgun was pretty easy to setup and things were great but unfortunately in the last day or so our free app started creeping over the 200 message/day limit on the Mailgun free tier. Since the app is currently free, we couldn’t really justify paying the $19/mon minimum for the first Mailgun tier so I started exploring self-hosted options.

After a big of Googling, the first thing that caught my eye was the LamsonProject. From my understanding, Lamson basically provides a SMTP server which binds to a MVC application framework. After checking out the docs and the code it seemed like Lamson would allow you to build a RESTful application that responded to SMTP messages instead of HTTP requests. It certainly seems like a cool project but seemed like a bit overkill for what I needed to do.

Going further down the rabbit hole, I found out that the Exim allows messages to be delivered to a regular *nix process via a pipe and that the configuration was supposedly relatively straightforward under Ubuntu. Given that, I decided to give it a shot, it turned out to be relatively easy to setup but here is a quick rundown.

Step 1: Setup Exim

By default, Ubuntu systems are configured with sendmail and only setup to receive local mail. You’ll need to remove sendmail, install Exim, and then configure it.

To do this you’ll roughly need to run the following:

sudo apt-get remove sendmail
# for some reason the dependencies aren't being installed correctly
# see http://ubuntuforums.org/showthread.php?p=10411454
sudo apt-get install exim4-base exim4-config
sudo apt-get install exim4
sudo dpkg-reconfigure exim4-config

Next, follow the guide on https://help.ubuntu.com/8.04/installation-guide/hppa/mail-setup.html to configure Exim.

Step 2: Configure Pipes and Aliases

Per https://answers.launchpad.net/ubuntu/+source/mailman/+question/120447 it turns out something isn’t enabled in the default Ubuntu configuration of Exim. To solve this, I added the following

.ifndef SYSTEM_ALIASES_PIPE_TRANSPORT
SYSTEM_ALIASES_PIPE_TRANSPORT = address_pipe
SYSTEM_ALIASES_USER = Debian-exim
SYSTEM_ALIASES_GROUP = daemon
.endif

Into the bottom of /etc/exim4/conf.d/transport/30_exim4-config_address_pipe

After you edit an Exim setting, you’ll need to always run the following for the settings to take effect:

sudo update-exim4.conf.template -r
sudo update-exim4.conf
udo service exim4 restart

Next, you’ll need to enable the catch-all router configuration. This was surprisingly difficult to track down how to do:

Per, http://pontus.ullgren.com/view/Setting_SMTP_server_development_test_server_running_Ubuntu you need to create a file at /etc/exim4/conf.d/router/950_exim4-config_catchall containing

catch_all:
   debug_print = "R: catch_all for $local_part@$domain"
   driver = redirect
   data = ${lookup{*}lsearch{/etc/aliases}}
   # NOTE: I added this line for the catch all pipes to work
   pipe_transport = address_pipe

Finally, add a catch-all alias that pipes to a script to your /etc/aliases file. Mine looks like:

*: "|/home/ubuntu/exim4.php"

Step 2: The PHP script

I did this with PHP since my original Mailgun script was in PHP but anything should work. Key notes, the script needs to be accessible AND executable by the Exim user. Also, you’ll need the appropriate shebang to make the script work without a named interpreter.

Your script is going to receive raw SMTP email so you’ll need to parse that out to do anything meaningful with it. Thankfully there are plenty of libraries to do this. I ended up using MailParse along with PHP MimeMail Parse to abstract the nitty gritty into some cleaner object oriented code.

The part of my script that accepts and parses the email looks like:

From there, you’d be free to do anything you wanted with $msgInfo, just like on the receiving end of a Mailgun route.

Anyway, as always let me know if you have any feedback, questions, or comments.

Gmail Reporting Tons of Used Space? This may help!

Recently I had to upgrade my Gmail account for additional storage. I was nearing the 8 free gigs of data they give you and didn’t want to keep seeing the big red “Buy more storage!”. I bought the $5/year 20 gig plan. A few weeks later I noticed that my Gmail was now reporting i was already using ~18 gigs of my total 30 gigs of data. I couldn’t believe it, how did I manage to more than double my used Gmail space within 2 weeks? I had used Gmail for 8 years to get to 7 gigs of data.

After looking around there didn’t seem to be anyone who could report the problem, nevertheless have a fix. I then tried to empty my Trash, which at the time said ~200 messages. As soon as I emptied it my used space dropped to 7.3 gigs, which is what I expected.

Long story short, it would appear that Gmail has a bug in reporting the number of actual messages in the trash, or doesn’t truly ’empty’ it unless click it. If you think you are using much less space than it is reporting, try emptying your trash manually. It worked for me and a number of other guys in the company.

BigCommerce Bulk Coupon Code Importer

A buddy of mine has an eCommerce storefront through BigCommerce, a hosted eCommerce solution. He was looking to run a Gilt City promotion, so he would have to import ~150 coupon codes into BigCommerce.

My friend spent some time looking around for an import codes feature and I would have assumed that a solution like BigCommerce would support a CSV import for coupon codes as well. However, we Googl’ed around a bit and it turns out BigCommerce does not support CSV imports for coupon codes. We stumbled upon this discussion on the BigCommerce forums discussing this shortcoming at length.

On the forum thread, someone had previously developed an iMacros script to automate importing the codes. Unfortunately, I tried using the iMacros script without any success. It looks like BigCommerce has updated the HTML on the page which is breaking the script. Anyway, looking at the iMacro and the opaque syntax I decided it would be easier to whip together a Chrome extension to automate creating coupon codes so decided to move forward with that.

Chrome extensions naturally lend themselves to automating things like this since it’s pretty straightforward to use the background page to maintain any state and hold variables and then use a content script to manipulate the DOM of the page and submit any forms that you need to.

You can grab the extension here http://setfive.com/misc/BigComm Importer.crx you’ll obviously need to be running Google Chrome for it to work.

Also, fork it at https://github.com/adatta02/bigcommerce-coupon-importer

What it does is add a “Bulk Import” button to the “Coupon Codes” page on your BigCommerce site – http://[yourstore].mybigcommerce.com/admin/index.php?ToDo=viewCoupons

If you click that button, a copy of the “add code” form will be displayed along with a textbox. You’ll just need to fill out the form, enter a list of coupon codes you want to create – one per line, and then click “Start Import”.

Your browser will automatically go through and create all your coupon codes. Thats it!

As always, let me know if you run into any issues or have any comments.

Internet Explorer Extension Quickstart and Skeleton

Recently, one of our clients was looking to build a prototype/proof of concept browser extension for Firefox, Chrome, and Internet Explorer. We were basically looking to inject a script, run it in user space, and modify some of the page’s DOM – like a trimmed down Greasemonkey script.

Doing this in Chrome and Firefox is pretty straightforward since the extensions are built Javascript, this is actually how I built the prototype tru.ly extensions. Unsurprisingly, the “odd man out” is Internet Explorer, this is my first time looking into writing an IE extension and the experience was pretty jarring so hopefully this synopsis can save you some time and frustration.

The first sign of trouble was that there doesn’t seem to be an official Microsoft guide on writing IE extensions. There’s just a bunch of ad-hoc tutorials, some MSDN articles, and then code samples built against every possible combination of language and library.

As it turns out, Internet Explorer has actually supported extensions since IE5 using a technology called Browser Help Objects and continues to support them via BHOs through IE9. In contrast to Chrome and Firefox’s Javascript based extensions, a BHO is a Windows DLL and consequently must be written and compiled using your choice of a Win32 compatible programming language. Given this and from the discussions I saw, the most popular choice seems to be to use C++/ATL to create a COM DLL. Being that I’m deathly afraid of C++ and that this approach was described as “COM DLL hell” I decided to see what else was possible.

After a bit more poking around I found out that it’s possible to use C# and .NET’s Interop libraries to scaffold enough to get the DLL loading into IE. This Code Project article walks through the process but it has several typos and the download containing the files seems to have gone missing. I fixed the typos and built it successfully – you can grab the files from GitHub here.

From my extremely rough understanding of C#, what the code does is create an interface from C# managed code to the unmanaged COM code that IE uses to communicate with extensions. Then, the code registers an event handler to be called once the DOM has finished rendering.

In order to actually build the project, you’ll need to do the following:

  • Install a copy of Visual Studio – I used VS Express 2010 which is free.
  • You’ll then need to import my VS project and build your DLL. If it complains about references to SHDocVw or IHTMLDocument2 you’ll just need to make sure that references to the two DLLs in Greyhound/ exist in your VS project.
  • Once the DLL is built, you’ll need to register it with RegAsm – this is a bit tricky since you need to use the correct version of RegAsm available on your system. This article explains where it should be located. Once you locate it, run the following:
C:\[path to your .NET library]\RegAsm.exe /codebase bin\Release\Greyhound.dll

Thats it. Now start Internet Explorer and once the DOM on a page finishes loading you should see an alert box being generated via Javascript that your DLL is injecting.

The magic is all happening in the following function:

public void OnDocumentComplete(object pDisp, ref object URL)
{
   IHTMLDocument2 doc = (IHTMLDocument2)webBrowser.Document;
   doc.parentWindow.execScript("var d=window.document,s=d.createElement('script'),h=d.getElementsByTagName('body')[0];s.src='http://twitlabs.net/sayhello.js.php';h.appendChild(s);");
}

You’ll just need to edit that script tag to load your own Javascript.

Anyway, pretty gnarly stuff. It’s also extremely frightening that IE extensions are basically full fledged programs that have full reign over your entire system. No sandboxed, no permission limitations, just a fully integrated program that people “casually” download off the Internet.