Symfony 1.4 partial model rebuilds

A couple of months ago we started building out a Symfony 1.4 project for a client that involved allowing a “super admin” to add Doctrine models and columns at runtime. I know, I know, crazy/terrible/stupid idea but it mapped so well in to the problem space that we decided that a little “grossness” to benefit the readability of the rest of the project was worth it. Since users were adding models and columns at runtime we had to subsequently perform model rebuilds as things were added. Things worked fine for awhile, but eventually there were so many tables and columns that a single model rebuild was taking ~1.5 minutes on an EC2 large.

Initially, we decided to move the rebuild to a background cron process but even that began to take a prohibitively long time and made load balancing the app impossible. Then I started wondering is it possible to partially rebuild a Doctrine model for only the pieces that have changed?

Turns out it is possible. Looking at sfDoctrineBuildModelTask it looked like you could reasonably just copy the execute() method out and update a few lines.

Then, the next piece was just building the forms for the corresponding models. Again, looking at sfDoctrineBuildFormsTask it looked like it would be possible to extract and update the execute() method.

Anyway, without further ado here is the class I whipped up:

https://gist.github.com/3791760.js?file=sfFastModelRebuild.class.php

Using it is pretty straightforward, just call FastModelRebuild::doRebuild( array(“sfGuardUser”, “sfGuardUserProfile”) ); and thats it!

Anyway, fair warning I’d only do something like this if you “Know what you are doing” ™

As always, questions and comments are welcome.

Fun stuff: GitHub random repository browser

Per Roger's comment as of 11/14/2013 this no longer works :(

Over the weekend, I was looking for a good way to randomly browse GitHub. I hopped over to the GitHub search page but unfortunately there isn’t anyway to randomly search the repositories.

Just for fun, I decided to throw together a random repository browser. Check it out over at http://v3.setfive.com/adatta02_gitstumble/

Custom Templates with jQuery File Upload

Recently I was working on a project which was using the jQuery File Upload Plugin to do multiple file uploads. I needed to show the progress of each upload (in this case just images). Looking through their documentation it shows how to a few ways to do custom templates. By default it uses the Javascript Templates Engine for all of its templating. I wanted to use just a div on the page for my template. Here’s how I ended up doing it.

First My basic markup for the html:

<input type="file" id="fileUploads" name="file" multiple />
<div id="uploadContainer">
 <div class="uploadedImage">
  <img src="myimg.png">
 </div>
 <span id="newImages"></span>
</div>

<div id="template">
 <div class="uploadedImage">
  <img src="" />
  <div class="progressbar"><span class="bar"></span></div>
 </div>
</div>

Basically my “#template” div was a hidden div on the page which I used as the photo being uploaded. Now for the javascript:

var progressElements = { };

$(document).ready( function(){
	
    $("#fileUploads").fileupload({
      dataType: 'json',
      done: function(e, data){
    	    
          var fileName = data.files[0].name;          
          var el = progressElements[ data.files[0].name ];        
          el.find("img:first").attr("src", data.result.thumbnail);
          el.find(".progressbar").hide();
      },
      progress: function(e, data){
    	    var el = progressElements[ data.files[0].name ];
    	    var progress = parseInt(data.loaded / data.total * 100, 10);
    	    el.find(".bar").css('width',progress+"%" );
      },
      add: function(e, data){

   	  var template = $("#reviewTemplate").html();    	       	 
    	  $("#newImages").before(template);

    	  progressElements[ data.files[0].name ] = $("#uploadContainer .uploadedImage:last");
    	  
    	  $("#fileUploads").fileupload("send", {files: data.files});
      },
      autoUpload: true,
      url: 'myUploadUrl',
    });

});

It ended up being pretty simple and self explanatory. The ‘progress’ function is called each time there is a progress update. You can do more advanced templates using their templating engine, however as I was adapting the code to an existing layout and was on a time constraint this was the route I took.

Hope this saves you sometime if you are looking to just quickly add a progress template for the uploaded images.

Recursive templates with UnderscoreJS

I was working on a BackboneJS project recently that was using UnderscoreJS for templating and started wondering if its possible to invoke UnderscoreJS recursively.

At face value, the _.template function simply takes a template string and converts it to a first class Javascript function so I figured it would be possible. Out of curiosity, I decided to take a look at how the _. function is actually implemented:

  _.template = function(text, data, settings) {
    settings = _.defaults(settings || {}, _.templateSettings);

    // Compile the template source, taking care to escape characters that
    // cannot be included in a string literal and then unescape them in code
    // blocks.
    var source = "__p+='" + text
      .replace(escaper, function(match) {
        return '\\' + escapes[match];
      })
      .replace(settings.escape || noMatch, function(match, code) {
        return "'+\n_.escape(" + unescape(code) + ")+\n'";
      })
      .replace(settings.interpolate || noMatch, function(match, code) {
        return "'+\n(" + unescape(code) + ")+\n'";
      })
      .replace(settings.evaluate || noMatch, function(match, code) {
        return "';\n" + unescape(code) + "\n;__p+='";
      }) + "';\n";

    // If a variable is not specified, place data values in local scope.
    if (!settings.variable) source = 'with(obj||{}){\n' + source + '}\n';

    source = "var __p='';" +
      "var print=function(){__p+=Array.prototype.join.call(arguments, '')};\n" +
      source + "return __p;\n";

    var render = new Function(settings.variable || 'obj', '_', source);
    if (data) return render(data, _);
    var template = function(data) {
      return render.call(this, data, _);
    };

    // Provide the compiled function source as a convenience for build time
    // precompilation.
    template.source = 'function(' + (settings.variable || 'obj') + '){\n' +
      source + '}';

    return template;
  };

Not any real surprises there - it basically does a series of string replaces on the template you pass in to interpolate any variables you want outputted, converts blocks you want evaluated to regular Javascript, and then injects the object you passed in into the local scope.

Then, the “magic” happens at “var render = new Function(settings.variable || ‘obj’, ‘_’, source);” where a new function is created with a “print” function locally defined and your template converted to a Javascript function.

Here is the “source” string that is passed into “new Function”:

var __p = '';
var print = function () {
    __p += Array.prototype.join.call(arguments, '')
};
with(obj || {}) {
    __p += '\n';
    if (depth) {;
        __p += '\n<div class=\'fib-box\' data-depth=\'' + (depth) + '\' style=\'width: ' + (val) + 'px; height: ' + (val) + 'px;\'></div>\n';
        print(template(getFibObj(depth - 1)));
        __p += '\n';
    };
    __p += '\n';
}
return __p;

That is the dynamic function generated for the following template:

<% if(depth){ %>
<div class='fib-box' data-depth='<%= depth %>' style='width: <%= val %>px; height: <%= val %>px;'></div>
<% print(template(getFibObj(depth-1))) %>
<% } %>

Since its looking for the “template” function in the global scope it looks like everything should work fine. To test it, I decide to take the Fibonacci sequence and generate boxes for each of the numbers in the sequence up to some N.

Theres a live demo running at http://twitlabs.net/us/ and a Gist of the code is replicated below:

https://gist.github.com/3223733.js

Just for fun, you can also change how the boxes are arranged by toggling between “Stacked” and “Cascaded”.

Anyway, fair warning - I imagine there are some performance implications regarding doing this as well as computational limits. You could potentially cause a stack overflow by consuming your entire stack via the recursion.

As always, questions, comments, concerns welcome!

BackboneJS: JSON over REST

We’ve recently started using BackboneJS on a couple of projects to help create more responsive UIs and avoid having to deal with a tangled mess of jQuery callbacks, ad-hoc Javascript templating, and difficult code re-use. On the whole, I’ve been impressed with Backbone and I think it’ll make a valuable asset in our toolbox.

Anyway, one of the core concepts of Backbone is that client side models will be automatically synced with their server side counterparts via RESTful AJAX calls. Stripping away the buzzword bingo, what that means is that when a Backbone model is created in the UI a AJAX POST request is initiated to create that model, updates trigger a PUT, and destroying a model will cause a DELETE.

Conceptually, this system makes a lot of sense and it fits nicely with how data is structured in most apps. Unfortunately, it falls apart when there isn’t a one to one correspondence between frontend Backbone models and your backend persistence layer.

For example, say on the frontend you wanted to allow a user to enter and edit the various languages she spoke. Naturally, you could define a Backbone “Language” model, managed by a “LanguageList” Collection, and finally drawn with a “LanguageView” View. This would work great but unfortunately on the backend you’re storing the list of languages the user speaks in a single row - say as a CSV list for arguments sake. At this point, you’d be stuck because Backbone would be pushing AJAX requests per model and your backend would have no way to combine the languages into a CSV list.

In this instance it seems like you really want “JSON over REST”, in the sense that you want to push update events for several models on a single request. The Backbone FAQ mentions this technique but unfortunately doesn’t elaborate on specifically how to achieve this behavior.

Batch operations on Models are common, but often best handled differently depending on your server-side setup. Some folks don’t mind making individual Ajax requests. Others create explicit resources for RESTful batch operations: /notes/batch/destroy?ids=1,2,3,4. Others tunnel REST over JSON, with the creation of “changeset” requests:

Google’ing around I stumbled across this StackOverflow answer - http://stackoverflow.com/questions/11298152/how-to-do-batch-operations-in-backbone-js-via-tunneling-rest-over-json which seems to have been removed. The answer basically suggested embedding a Collection within a Model so that Backbone effectively passed a JSON array to your REST endpoint.

Unfortunately, the answer didn’t provide a concrete example either so here is a stripped down version of what I ended up using.

var Language = Backbone.Model.extend({
    defaults: function() {
        return {
            languageName: "",
        };
    },
    initialize: function() {

    },
    save: function() {
        LanguagesCollection.at(0).save();
    },
    destroy: function(options) {

        options = options ? _.clone(options) : {};
        var model = this;
        var success = options.success;

        model.trigger('destroy', model, model.collection, options);

        LanguagesCollection.at(0).save();
    }
});

var LanguageList = Backbone.Collection.extend({
    model: Language,
});

var LanguageCollection = Backbone.Model.extend({
    defaults: function() {
        return {
            languages: new LanguageList()
        };
    },
    parse: function(response) { }
});

var LanguageCollectionList = Backbone.Collection.extend({
    model: LanguageCollection,
});

var LanguageView = Backbone.View.extend({

    tagName: "div",
    template: _.template($('#language-item-template').html()),
    initialize: function() {
        this.model.bind('change', this.render, this);
        this.model.bind('destroy', this.remove, this);
    },

});

var LanguagesAppView = Backbone.View.extend({

    el: $("#languageList"),

    initialize: function() {
        LanguagesCollection.add([{}]);
        LanguagesCollection.at(0).get("languages").bind('add', this.addOne, this);
        LanguagesCollection.at(0).get("languages").bind('all', this.render, this);
    },

    addOne: function(loc) {
        var v = new LanguageView({
            model: loc
        });
        this.$el.append(v.render().el);
    }

});

LanguagesCollection = new LanguageCollectionList();
var LanguagesAppView = new LanguagesAppView;

Basically, “Language” is the “real” model which is managed by the “LanguageList” collection. Then, “LanguageCollection” is the container model which holds a single “LanguageList” collection and is subsequently managed by the “LanguageCollectionList”.

If you look through the code, basically the Views are bound to events on ‘LanguagesCollection.at(0).get(“languages”)’ which is a LanguageList object.

The other thing to note is that Language.destroy() function is updated to not make an AJAX DELETE call but instead trigger a save on the collection.

Also, LanguageCollection.parse() is defined as an empty function because ‘LanguagesCollection.at(0).get(“languages”)’ needs to remain a Backbone object and not be converted into the regular JSON which would come back from the server. The caveat here is that if you need to set an ID after a POST, you’ll need to update parse() to process the JSON and then update the corresponding models within the collection.

Anyway, I’m still a Backbone novice so any input or insight is more than welcome. I’d also love to know if theres a more straightforward way to address this. A couple of posts mentioned overriding Backbone.sync but that wasn’t feasible for me because I had other Backbone models on the page which needed to update using the regular REST pattern.

Update 7/20/2012

As pointed out by Aria below, it’s actually possible to override the “sync” method per model to achieve this JSON over REST behavior. Overall, its probably a cleaner solution since it avoids introducing the complexity of collections contained in models simply to get the JSON over REST behavior.

Here’s an example implementation of a sync function that produces the desired results:

var Language = Backbone.Model.extend({
    defaults: function() {
        return {
            languageName: "",
        };
    },
    initialize: function() {

    },
    url: "echoBackbone.php",
    sync: function(method, model, options){
    	
    	var methodMap = {
    		    'create': 'POST',
    		    'update': 'PUT',
    		    'delete': 'DELETE',
    		    'read': 'GET'
    	};
    	
    	var type = methodMap[method];

        // Default options, unless specified.
        options || (options = {});

        // Default JSON-request options.
        var params = {type: type, dataType: 'json'};

        // Ensure that we have a URL.
        if (!options.url) {
          params.url = this.url || urlError();
        }
    	
        // Don't process data on a non-GET request.
        if (params.type !== 'GET' && !Backbone.emulateJSON) {
          params.processData = false;
        }

        // NEW - the juicy bits to combine all the models in this collection
        var collection = [];
        model.collection.each( function(e, i, l){
            collection.push( e.toJSON() );
        });
        
        params.data = JSON.stringify(collection);
        
        // Make the request, allowing the user to override any Ajax options.
        return $.ajax(_.extend(params, options));        
    }
});

NOTE: Almost all of that code is copied directly out of Backbone.js, the exception is the following block which collects up all the models in the current model’s collection and then adds it into the “data” field:

  var collection = [];
        model.collection.each( function(e, i, l){
            collection.push( e.toJSON() );
        });
        
  params.data = JSON.stringify(collection);

Happy Backbone’ing!