On one of our projects that I am working on I had the following problem: I needed to create an aggregate temporary table in the database from a few different queries while still using Doctrine2. I needed to aggregate the results in the database rather than memory as the result set could be very large causing the PHP process to run out of memory. The reason I wanted to still use Doctrine to get the base queries was the application passes around a QueryBuilder object to add restrictions to the query which may be defined outside of the current function, every query in the application goes through this process for security purposes.
After looking around a bit, it was clear that Doctrine did not support (and shouldn’t support) what I was trying to do. My next step was to figure out how to get an executable query from Doctrine2 without ever running it. Doctrine2 has a built in SQL logger interface which basically lets you to listen for executed queries and to see what the actual SQL and parameters were for the executed query. The problem I had was I didn’t want to actually execute the query I had built in Doctrine, I just wanted the SQL that would be executed via PDO. After digging through the code a bit further I found the routines that Doctrine used to actually build the query and parameters for PDO to execute, however, the methods were all private and internalized. I came up with the following class to take a Doctrine Query and return a SQL statement, parameters, and parameter types that can be used to execute it via PDO.
In the ExampleUsage.php file above I take a query builder, get the runnable query, and then insert it into my temporary table. In my circumstance I had about 3-4 of these types of statements.
If you look at the QueryUtils::getRunnableQueryAndParametersForQuery function, it does a number of things.
Overall this was the best solution I could find at the time for what I was trying to do. If I was ok with running the query first, capturing the actual SQL via an SQL Logger would have been the proper and best route to go, however I did not want to run the query.
Hope this helps if you find yourself in a similar situation!
On many of our projects we use Gearman to do background processing. One of problems with doing things in the background is that the web debug toolbar isn’t available to help with debugging problems, including queries. Normally when you want to see your queries you can look at the debug toolbar and get a runnable version of the query quickly. However, when its running in the background, you have to look at the application logs to see what the query is. The logs don’t contain a runnable format of the query, for example they may look like this:
Problem is you can’t quickly take that to your database and run it to see the results. Plugging in the parameters is easy enough, but it takes time. I decided to quickly whip up a script that will take what is in the gist above and convert it to a runnable format. I’ve posted this over at http://code.setfive.com/doctrine-query-log-converter/ . This hopefully will save you some time when you are trying to debug your background processes.
It should work with both Doctrine 1.x/symfony 1.x and Doctrine2.x/Symfony2.x. If you find any issues with it let me know.
Good luck debugging!
The first step is to create a script that will be executed by PhantomJS. This script will do the following:
Next, we want to create a PHP function that actually executes the above script and converts the html to a SimpleXmlElement object.
Finally, running the function from step 3 should result in something like this.
Recently I was working on a project where part of it was doing data exports. Exports on the surface are quick and easy – query the database, put it into the export format, send it over to the user. However, as a data set grows, exports become more complicated. Now processing it in real time no longer works as it takes too long or too much memory to export. This is why I’ll almost always use a background process (notified via Gearman) to process the data and notify the user when the export is ready for download. On separate background threads you can have different memory limits and not worry about a request timeout. I suggest trying to not use Doctrine’s objects for the export, but get the query back in array format (via getArrayResult). Doctrine objects are great to work with, but expensive in terms of time to populate and memory usage; if you don’t need the object graph results in array format are much quicker and smaller memory wise.
On this specific export I was exporting an entity which had a foreign key to another table that needed to be in the export. I didn’t want to create a join over the entire data set as it was unnecessary. For example, a project which has a created by user as a relation. If I simply did the following:
I’d end up with an array which had all the project columns except any that are defined as a foreign key. This means in my export I couldn’t output the “Created by user id” as it wasn’t included in the array. It turns out that Doctrine already has this exact situation accounted for. To include the FK columns you need to set a hint on the query to include meta columns to true. The updated query code would look similar to:
Now you can include the foreign key columns without doing an joins on a query that returns an array result set.
Last week we officially did a very quiet launch of HotelSaver.io. The concept is fairly simple: You submit your existing hotel reservation, we constantly monitor for price drops, if we find one we notify you immediately and you save money. I had the idea for this site a few months ago when I had made reservations in New Orleans for a bachelor party, then noticed the next day the prices dropped and I managed to save over 40% of the reservation by getting it price matched and discounted. At this point I thought “wow, that was incredibly easy, took little time and saved a ton of money”. Shortly thereafter and shooting it around with everyone here, it was decide we’re going to launch a MVP product and see what the reception is.
With this concept it is really easy to quickly blow it up into a massive product with complex algorithms, payment types, etc. however trying to follow our own advice to our clients we launched with the minimal features to make it useful to the end user: simple reservation monitoring and payment processing to get us paid. We knew the first version of the product would be far from finished in terms of design and feature complete, but we wanted to see if others thought the idea had legs. Here are a few things we did to cut down on the time to launch even more:
We also wanted to get feedback from a small group of users. We posted it on HackerNews and immediately started getting great feedback. We knew posting this on the day we were traveling for the Holidays wasn’t optimal as we couldn’t respond to feedback immediately, we wanted to get this launched. We managed to make it to the front page of HackerNews for a while and instantly had 2,500 unique visitors that day, up from zero the day before! The feedback was great the main points were:
Today we revamped our pricing strategy after the feedback. We knew the upfront cost was most likely a turn away for many users but didn’t know what percentage would hate it. After reading the feedback on the post and numerous emails, we’ve switched to a 20% of the amount saved. This makes it 100% risk free to the user. We won’t make money unless you save money. If we save you $100 dollars, you get $80 of it. We’ll be next week working on promoting the revised pricing strategy to see what additional feedback we can get as well as addressing the other parts of the feedback.
We’ll be trying to keep everyone updated on our adventures of launching our own product in house. We’re excited to try some techniques we’ve seen over the years and testing them out ourselves as well as trying some new ideas. If you have any feedback let us know!