Category: General

Recently I was working on a project where part of it was doing data exports. Exports on the surface are quick and easy - query the database, put it into the export format, send it over to the user. However, as a data set grows, exports become more complicated. Now processing it in real time no longer works as it takes too long or too much memory to export. This is why I’ll almost always use a background process (notified via Gearman) to process the data and notify the user when the export is ready for download. On separate background threads you can have different memory limits and not worry about a request timeout. I suggest trying to not use Doctrine’s objects for the export, but get the query back in array format (via getArrayResult). Doctrine objects are great to work with, but expensive in terms of time to populate and memory usage; if you don’t need the object graph results in array format are much quicker and smaller memory wise.

On this specific export I was exporting an entity which had a foreign key to another table that needed to be in the export. I didn’t want to create a join over the entire data set as it was unnecessary. For example, a project which has a created by user as a relation. If I simply did the following:

// In ProjectRepository
public function getBaseExportQuery()
{
  return $this->createQueryBuilder('p')
              ->getQuery()
              ->getArrayResult();
  
}

I’d end up with an array which had all the project columns except any that are defined as a foreign key. This means in my export I couldn’t output the “Created by user id” as it wasn’t included in the array. It turns out that Doctrine already has this exact situation accounted for. To include the FK columns you need to set a hint on the query to include meta columns to true. The updated query code would look similar to:

// In ProjectRepository
public function getBaseExportQuery()
{
  return $this->createQueryBuilder('p')
              ->getQuery()
              ->setHint(\Doctrine\ORM\Query::HINT_INCLUDE_META_COLUMNS, true)
              ->getArrayResult();
  
}

Now you can include the foreign key columns without doing an joins on a query that returns an array result set.

As a team last week we decided to work through a “Coding Kata” as a learning experiment. For those unfamiliar with the concept, a Kata is a Japaneese word that refers to a choreographed series of movements. In the context of programming, a Kata refers to repeatedly solving a problem with incrementally improving technique. The challenge we decided to look at was a somewhat contrived Bank OCR problem described on codingdojo.org.

We only looked at “User Story 1” which describes a process for translating lines of ASCII text into the decimal numbers which they represent. The task isn’t particularly hard but what we were interested in was how different people approached the problem, what considerations they had, and where they got hung up. As a group, the consensus approach was to decompose the problem into pieces, decide on data structures, and then sketch out some pseudo code.

• Breaking down the problem is fairly straightforward. We're basically looking to read in the file, separate the lines into the "bank numbers", recognize individual numbers, and then combine the numbers we've recognized. An additional benefit of approaching the Kata like this was that individual team members could craft solutions in different programming styles - imperative, functional, etc.
• Once we had the "big" pieces of the solution, the next thing we flushed out was which data structures to use. After some discussion it was clear that reading the file and recognizing the numbers were related and the data structures used would need to be related. Interestingly, there was a decent amount of debate around what to use to represent the read lines and then how to match them. The team was split between using a 3x3 array versus using a 9 character string.
• The final step we took was jotting down some pseudo code on a whiteboard to actually perform all the steps. I've always found it interesting to watch people code on a whiteboard and this time wasn't any different. Our team seemed to struggle with visualizing what the data looked like and also how much detail to convey in the pseudo code.

After we wrapped up on the whiteboard we ended up implementing 3 solutions. A functional solution using a 3x3 array for the numbers, an imperative solution also using a 3x3 array, and an imperative solution using a 9 character string to represent the numbers. The solutions are similar but the style of the original author still definitely comes through.

Anyway, how would you have approached a problem like this? We’d love to hear in the comments!

Last week we officially did a very quiet launch of HotelSaver.io. The concept is fairly simple: You submit your existing hotel reservation, we constantly monitor for price drops, if we find one we notify you immediately and you save money. I had the idea for this site a few months ago when I had made reservations in New Orleans for a bachelor party, then noticed the next day the prices dropped and I managed to save over 40% of the reservation by getting it price matched and discounted. At this point I thought “wow, that was incredibly easy, took little time and saved a ton of money”. Shortly thereafter and shooting it around with everyone here, it was decide we’re going to launch a MVP product and see what the reception is.

With this concept it is really easy to quickly blow it up into a massive product with complex algorithms, payment types, etc. however trying to follow our own advice to our clients we launched with the minimal features to make it useful to the end user: simple reservation monitoring and payment processing to get us paid. We knew the first version of the product would be far from finished in terms of design and feature complete, but we wanted to see if others thought the idea had legs. Here are a few things we did to cut down on the time to launch even more:

• The initial design is based on a free template we found which allowed us to spend near no time on design. It works on mobile devices and doesn't look terrible. None of us are great designers, so we figured this was well worth it for the first release.
• Not over thinking user management. Over time we plan to add accounts to the site so you can see your existing reservations from a dashboard, however for this first version we opted to go with a simple "email" to link together accounts. Users submit their existing hotel reservation with their email which we use later if you need to retrieve it. From there you can retrieve your "active" (reservations that have not yet past) reservations in an email we email to you.
• Payment processing. This one was a no-brainer for us. We wanted to be PCI compliant and also have a good user experience. Stripe we had worked with in the past and knew it was incredibly easy to use. We went with the checkout feature so we never would have any of their credit card information and it never hit our servers, making us PCI compliant.

We also wanted to get feedback from a small group of users. We posted it on HackerNews and immediately started getting great feedback. We knew posting this on the day we were traveling for the Holidays wasn’t optimal as we couldn’t respond to feedback immediately, we wanted to get this launched. We managed to make it to the front page of HackerNews for a while and instantly had 2,500 unique visitors that day, up from zero the day before! The feedback was great the main points were:

• Everyone loved the idea and thought if executed properly it'd be great!
• People didn't like that we wanted to charge $19.99 regardless of if we could find a lower cost reservation. It was too risky.
• Some of the design could use some love.
• Pricing would be more interesting/better if it was a percentage saved or a money back guarantee.

Today we revamped our pricing strategy after the feedback. We knew the upfront cost was most likely a turn away for many users but didn’t know what percentage would hate it. After reading the feedback on the post and numerous emails, we’ve switched to a 20% of the amount saved. This makes it 100% risk free to the user. We won’t make money unless you save money. If we save you $100 dollars, you get $80 of it. We’ll be next week working on promoting the revised pricing strategy to see what additional feedback we can get as well as addressing the other parts of the feedback.

We’ll be trying to keep everyone updated on our adventures of launching our own product in house. We’re excited to try some techniques we’ve seen over the years and testing them out ourselves as well as trying some new ideas. If you have any feedback let us know!

With the year coming to a close we decided to take a look back at how the Boston startup community did. It was another awesome year for the community and the numbers definitely echo that sentiment. We’d like to extend a big congratulations to the companies that went public or were acquired in 2014 and here’s to a great 2015!

A couple of months ago I was at Firebrand Saints and started wondering how the effects that they run on their videos work. At Firebrand, their main bar has a few flat screen TVs showing cable channels you’d expect but every now and then one of the TVs will start displaying the video through a filter. So as you’re sitting at the bar you’ll notice Sportcenter go from regular video to what looks like Sportcenter put through a pixelated filter. Unfortunately, the fall got a bit busy and I forgot to jot this down but it recently came back to me while watching the Family Guy - Take On Me skit.

So how do you programmatically apply effects to video? After some searching around, it seems like the preferred way to do this is to convert the video to a series of images, apply the effects, and then encode the images back to a video. Since we’re on Linux the weapons of choice for this are libav (ffmpeg fork) and ImageMagick. Additionally, I used youtube-dl to grab some source video from YouTube.

Playing around with manipulating videos and images is pretty CPU intensive so I decided to do this on a c3.xlarge. Once you have a machine up, just run the following to get everything setup:

sudo apt-get update
sudo apt-get install libav-tools imagemagick python-pip python-dev build-essential
sudo pip install --upgrade youtube_dl
sudo mkdir /mnt/videos
sudo chown ubuntu:ubuntu /mnt/videos
cd /mnt/videos/

Now, snag yourself a video from YouTube:

youtube-dl https://www.youtube.com/watch?v=nfWlot6h_JM

Next, you’ll want to extract the audio and then the individual frames from the video:

mkdir frames
avconv -i Taylor\ Swift\ -\ Shake\ It\ Off-nfWlot6h_JM.mp4 -c:a copy -vn -y audio.aac
avconv -i Taylor\ Swift\ -\ Shake\ It\ Off-nfWlot6h_JM.mp4 -r 30 -f image2 frames/%05d.png

And now for the fun part - time to apply some effects to the image! As an fun first pass I ran the “paint” transform across the images. PS. Remember how we launched on that c3.xlarge? Well now we can run the transforms in parallel with xargs:

cd frames
find *.png | xargs -n 1 -P 4 -i mogrify -resize 50% {}
cd ../
mkdir -p target/frames
find frames/*.png | xargs -n 1 -P 4 -i convert {} -paint 5 target/{}

Finally, you’ll need to encode the images back together into a video format of your choice:

avconv -f image2 -i frames/%05d.png -c:v h264 -crf 1 -r 24 -i audio.aac painedVideo.mp4

Here’s a clip of the video I ended up with:

So what else can you do with this? Well that’s the awesome part! An image manipulations you can do programmatically (ImageMagick, NodeJS, etc.) you can apply to your video.

Perhaps some Mystery Science Taylor Swift?

Unfortunately this adventure has left me with more questions than answers:

• How do you apply filters to a video in real time?
• Would it be possible to integrate this into a rooted (or stock) Chromecast?
• Could you build something to support user interactivity? Maybe with Canvas/nodeJS?

I’d love any thoughts, input, or ideas!