Including Foreign Keys in Doctrine2 Array Results

By Matt Daum | 1 Comment on Including Foreign Keys in Doctrine2 Array Results

Recently I was working on a project where part of it was doing data exports. Exports on the surface are quick and easy – query the database, put it into the export format, send it over to the user. However, as a data set grows, exports become more complicated. Now processing it in real time no longer works as it takes too long or too much memory to export. This is why I’ll almost always use a background process (notified via Gearman) to process the data and notify the user when the export is ready for download. On separate background threads you can have different memory limits and not worry about a request timeout. I suggest trying to not use Doctrine’s objects for the export, but get the query back in array format (via getArrayResult). Doctrine objects are great to work with, but expensive in terms of time to populate and memory usage; if you don’t need the object graph results in array format are much quicker and smaller memory wise.

On this specific export I was exporting an entity which had a foreign key to another table that needed to be in the export. I didn’t want to create a join over the entire data set as it was unnecessary. For example, a project which has a created by user as a relation. If I simply did the following:

I’d end up with an array which had all the project columns except any that are defined as a foreign key. This means in my export I couldn’t output the “Created by user id” as it wasn’t included in the array. It turns out that Doctrine already has this exact situation accounted for. To include the FK columns you need to set a hint on the query to include meta columns to true. The updated query code would look similar to:

Now you can include the foreign key columns without doing an joins on a query that returns an array result set.

Taken Audio MadLibs

By Jared Clapp | Leave a Comment on Taken Audio MadLibs

Because sometimes it’s just fun to make something absurd: http://taken.setfive.com/.

taken

If you aren’t familiar, the Taken movie series (http://www.imdb.com/title/tt0936501/) is a set of ridiculous action dramas that have acquired a strong cult following over the years (deservedly or not). All of the movies basically involve someone close to a retired CIA agent, played by Liam Neeson, being “Taken” and Neeson subsequently raining hell over anyone involved in wronging him. As a result of the movie series, Liam Neeson has made a formidable run at Kiefer Sutherland’s title of today’s Chuck Norris. Neeson, who plays the main character in the movie employs his signature throat punch in place of Norris’ roundhouse kick.

Our team seems to share a fondness for laughable action movies and actually took the afternoon off to watch the latest Taken 3 film when it debuted a couple weeks ago. With a little bit of vacation downtime prior to the debut and the urge to develop a web application as preposterous as the film series, we came up with the idea of creating Taken Audio MadLibs (http://taken.setfive.com/). For those of you not familiar MadLibs (http://en.wikipedia.org/wiki/Mad_Libs) is a phrasal template word game where one player prompts others for a list of words to substitute for blanks in a story, before reading the – often comical or nonsensical – story aloud.

So our take on the MadLibs program works something like this:

-There are four different “story lines” which all involve dialog between Liam Neesons character and a second character
-Neesons lines are actual lines from the Taken movies
-The other character’s lines are transformed into audio clips by Google’s unofficial Text To Speech (TTS) API and are based on the words you enter into a simple webform
-The lines are combined together into a hilarious back and forth dialog between Neeson and your character

Technically, there really isn’t too much “magic” going on with the program. It’s built on the Symfony2 framework and employs a simple one page parallax scrolling design for the web forms. Once the user submits the web form for their story we send it off to the controller where the user entered words are inserted into a set of templated lines. These text lines are then sent off to the Google TTS API which returns an .mp3 audio file with the audio representation of the text. We then splice together the Google TTS mp3 lines with the Taken audio files that we have stored on the server and combine into one audio file. The audio file is returned to the UI in the form of a HTML5 audio tag where the user can play or download the file. We also provide the user with the option of emailing the audio file to a friend if they would like to.

There were two problems that we ran into worth mentioning for those of you playing around with Google’s TTS api or combining multiple audio files of different formats.

1. Google’s TTS API only accepts 100 characters per call so you’ll have to split a given line or sentence into 100 character chunks and then combine the multiple mp3s back into one. This isn’t too difficult to do but worth mentioning if you ever plan to play around with this API.

2. We did run into a bit of trouble trying to combine the .mp3 files that Google returned with the Taken audio .mp3 files we got from (http://www.soundboard.com/sb/Taken_sound_clips). The problem is that the frame rate of the Google .mp3 files is different than the Taken files so when we tried to combine them into one some audio players would not render the resulting file. To get around these issues we took the following steps the combine and massage the audio files via a couple different server-side Linux-based Audio programs (avconv and oggCat):

  • combine the “chunked” Google .mp3s into one .ogg representing a single line in the dialog: “avconv -i [mp3_input_file] -acodec libvorbis -q:a 5 [ogg_output_file]”
  • convert the Taken “line” .mp3 files to individual .ogg files: “avconv -i [mp3_input_file] -acodec libvorbis -q:a 5 [ogg_output_file]”
  • combine the final Google “line” .ogg files with the Taken “line” .ogg files: “oggCat [ogg_output_file] [ogg_input_file_1] [ogg_input_file_2] [ogg_input_file_3] ….”
  • convert the final .ogg file to .wav so all browser types will play nice (shakes fist at safari): “avconv -i [ogg_input_file] [wav_output_file]”

Anyways, if you haven’t checked out the final program yet it can be seen here http://taken.setfive.com/ and if you have any questions, comments, or feedback feel free to leave them below.

Scala: Fun with Parser Combinators

By Ashish Datta | Leave a Comment on Scala: Fun with Parser Combinators

A couple of months ago I decided to use Scala with the Play Framework for a Bitcoin related project. The decision to use Play was motivated primarily by the goal of implementing a “pure” Bitcoin application, leveraging bitcoinj to interact with the Bitcoin network as opposed to a third party service. Overall, everything was pretty straightforward but one thing that stuck out was how the Play framework handles parsing JSON.

Coming from loosely typed PHP I knew that handling serializing and unserializing of JSON data was going to be different but Play’s approach is a completely new paradigm. If you use Play with Scala, you can handle parsing JSON input back into objects using Scala’s Parser Combinators syntax. I’m going to butcher this description so check the wikipedia entry but the idea is that parser combinators let you “build up” increasingly complex parsers by combining functions that recognize smaller inputs. If you’ve taken a compilers or programming language class, Scala’s parser combinators end up looking a lot like Backus–Naur Form for the input you want to recognize.

Anyway in an effort to learn Scala a bit better and take Parser Combinators for a spin I decided to build out a small project. What I ended up building is a really simple implementation of a Turtle Graphics system. You basically feed it a series of “turtle” commands and it’ll move the “turtle” around on a Swing window drawing some graphics.

Here’s an examples of the input and output:

Which was generated by:

Overall, parser combinators seem to be a really powerful Scala feature that would make developing domain specific languages relatively straightforward. Compared to messing around with a parser generator, using parser combinators seems to more closely mirror what the formal grammar of a language would be.

The entire project is available on GitHub. If you clone that project, there’s a runnable JAR which you can run with:

java -jar logoparser.jar /home/ashish/workspace_java/logo-parser/samples/face.txt

You’ll need to provide an absolute path for the “filewatcher” to work. Once the app starts, if you modify the file you specify it’ll repaint the canvas with your updates. Note: I’m not sure why but certain text editors don’t seem to register in the Java “filewatcher” interface so if your updates aren’t showing up try using a different editor.

Anyway, as always I’d love any feedback!

Coding: How do you approach an engineering problem?

By Ashish Datta | Leave a Comment on Coding: How do you approach an engineering problem?

As a team last week we decided to work through a “Coding Kata” as a learning experiment. For those unfamiliar with the concept, a Kata is a Japaneese word that refers to a choreographed series of movements. In the context of programming, a Kata refers to repeatedly solving a problem with incrementally improving technique. The challenge we decided to look at was a somewhat contrived Bank OCR problem described on codingdojo.org.

We only looked at “User Story 1” which describes a process for translating lines of ASCII text into the decimal numbers which they represent. The task isn’t particularly hard but what we were interested in was how different people approached the problem, what considerations they had, and where they got hung up. As a group, the consensus approach was to decompose the problem into pieces, decide on data structures, and then sketch out some pseudo code.

  • Breaking down the problem is fairly straightforward. We’re basically looking to read in the file, separate the lines into the “bank numbers”, recognize individual numbers, and then combine the numbers we’ve recognized. An additional benefit of approaching the Kata like this was that individual team members could craft solutions in different programming styles – imperative, functional, etc.
  • Once we had the “big” pieces of the solution, the next thing we flushed out was which data structures to use. After some discussion it was clear that reading the file and recognizing the numbers were related and the data structures used would need to be related. Interestingly, there was a decent amount of debate around what to use to represent the read lines and then how to match them. The team was split between using a 3×3 array versus using a 9 character string.
  • The final step we took was jotting down some pseudo code on a whiteboard to actually perform all the steps. I’ve always found it interesting to watch people code on a whiteboard and this time wasn’t any different. Our team seemed to struggle with visualizing what the data looked like and also how much detail to convey in the pseudo code.

After we wrapped up on the whiteboard we ended up implementing 3 solutions. A functional solution using a 3×3 array for the numbers, an imperative solution also using a 3×3 array, and an imperative solution using a 9 character string to represent the numbers. The solutions are similar but the style of the original author still definitely comes through.

Anyway, how would you have approached a problem like this? We’d love to hear in the comments!

Friday Links: RDMS, Bitcoin, and Facebook

By Ashish Datta | Leave a Comment on Friday Links: RDMS, Bitcoin, and Facebook

Congrats! You’ve survived the first full work week of 2015. Hopefully your resolutions are still intact and if they’re not there’s always next year. Shockingly informational links ahead so hold on to your hats.

Are there any reasons to use PostgreSQL DB over MySQL DB?

PostgreSQL vs. MySQL is as close to a religious war as you’ll get in the database world. If you’re wondering about the pros and cons of PostgreSQL then check out this Quora thread. Read More >

Efficient Marketing Research for Startups Using Facebook Advertising

Market research is hard, especially when you’re on a budget. Luckily for you Facebook is a great underutilized tool to perform research and validate some hypotheses. Learn More >

Bitcoin: Looking forward to 2015

The price of Bitcoin might be falling off a cliff but there’s still big things ahead. adatta02 expands on a pmarca Tweetstorm to see what’s in store. Keep reading >