Category: PHP

Doctrine 1.2: To many columns causes findBy* to fail

By Ashish Datta | Leave a Comment on Doctrine 1.2: To many columns causes findBy* to fail

Last week, one of our projects hit a pretty odd limit that I’d never expected to reach. The project is an analytics platform that allows admins to “pull” data from another, third party application. To accomplish this, the application allows admin users to dynamically add and remove columns from SQL tables and then dynamically chart these columns. Because of this, one of the tables had gotten over 350 columns which had all been created dynamically at runtime.

Anyway, things were working fine until last week when the application started throwing the following fatal error: “Fatal error: Uncaught exception ‘Doctrine_Table_Exception’ with message ‘Invalid expression found: ‘ in /usr/share/php/symfony/plugins/sfDoctrinePlugin/lib/vendor/doctrine/Doctrine/Table.php:2746″ Looking at the error, I noticed a warning was actually getting thrown right before the fatal error: “Warning: preg_replace(): Compilation failed: regular expression is too large at offset 32594 in /usr/share/php/symfony/plugins/sfDoctrinePlugin/lib/vendor/doctrine/Doctrine/Table.php on line 2745″ Looking through the code of Table.php, its clear that because “preg_replace” fails the $expression is subsequently blank which causes Doctrine to throw an error. I wanted to see how bad the regex was so I updated the Table.php to dump the expression. Here is what Doctrine was trying to run:

/(lsc_calculated_promoterscore_delta_innovativeness_formula|Lsc_calculated_promoterscore_delta_innovativeness_formula|lsc_calculated_promoterscore_delta_innovativeness_order|lsc_calculated_promoterscore_delta_favorability_formula|Lsc_calculated_promoterscore_delta_favorability_formula|Lsc_calculated_promoterscore_delta_innovativeness_order|Lsc_calculated_promoterscore_delta_favorability_order|

[Lots of columns...]

|Grid_mac_total|GridTotalTotal|GridTotalWinpc|Grid_total_mac|GridWindowsMac|grid_both_mac|grid_mac_ipad|audience_type|GridBothWinpc|GridBothTotal|Grid_both_mac|GridTotalIpad|Audience_type|WindowsosFy13|Grid_mac_ipad|grid_mac_mac|Grid_mac_mac|Program_type|program_type|AudienceType|GridMacTotal|GridBothIpad|GridTotalMac|GridMacWinpc|venue_child|ProgramType|GridMacIpad|Venue_child|GridBothMac|Program_id|program_id|VenueChild|GridMacMac|Fiscalyear|fiscalyear|is_locked|Is_locked|ProgramId|IsLocked|Country|country|venue|Venue|os|OS|id|Os|Id)(Or|And)?/

Looking at the php.net documentation for preg_replace and preg_match neither actually mention a hard limit on the size of a regex that can be compiled. Obviously there is a limit though and I imagine it must depend on the underlying RegExp library that your PHP is compiled against so it might be platform dependent.

As for solutions for this problem? The best solution for an extreme case like this is probably to just manually fill those in with real methods in the Doctrine_Table classes:

PHP: Quick and dirty CLI tasks

By Ashish Datta | Leave a Comment on PHP: Quick and dirty CLI tasks

Something that comes up every so often in a sufficiently large PHP project is having to write helper scripts that run on the command line to complete various tasks. It might be periodically processing some images, updating cached analytics, etc. If the project is a Symfony project, it’s usually easy enough to add a Symfony task and be able to leverage the Symfony infrastructure to manage the individual “scripts” as tasks. This is equally true with Drupal, using Drush tasks to manage the individual scripts works well and lets you have a single, central spot for all your “helpers”. But what if its a vanilla PHP project or WordPress?

A technique I’ve started using is to create a class and then add each of the tasks as static functions. This allows you to keep all the tasks in one place, reuse code and configurations, and generally mimic how Symfony tasks and Drush work. From there, the file pulls off $argv to figure out what function to call and just passes $argv in as an argument as well.

Here’s a stub of a class to set something like this up:

SwiftMailer: Expected response code 250 but got code 421

By Ashish Datta | 7 Comments on SwiftMailer: Expected response code 250 but got code 421

Last week we deployed a background script for a client which was used intermitently to batch send a couple of hundred emails. We were using SwiftMailer but weren’t able to use the “Spool” strategy to send because the messages contained Unicode characters which was breaking the serialization. Anyway, we ended up with code that looked something like the following:

Nothing to crazy going on. We were also sending the emails through Amazon SES which is why we introduced the sleep(..) to keep ourselves below the sending limits.

Things seemed like they were fine but then we’d seemingly randomly get the following exception:

PHP Fatal error:  Uncaught exception 'Swift_TransportException' with message 'Expected response code 250 but got code "421", with message "421 Timeout waiting for data from client.
"' in /usr/share/php/symfony/vendor/swiftmailer/classes/Swift/Transport/AbstractSmtpTransport.php:406
Stack trace:
#0 /usr/share/php/symfony/vendor/swiftmailer/classes/Swift/Transport/AbstractSmtpTransport.php(290): Swift_Transport_AbstractSmtpTransport->_assertResponseCode('421 Timeout wai...',$
#1 /usr/share/php/symfony/vendor/swiftmailer/classes/Swift/Transport/EsmtpTransport.php(197): Swift_Transport_AbstractSmtpTransport->executeCommand('MAIL FROM: <adm...', Array, Arra$
#2 /usr/share/php/symfony/vendor/swiftmailer/classes/Swift/Transport/EsmtpTransport.php(267): Swift_Transport_EsmtpTransport->executeCommand('MAIL FROM: <adm...', Array)
#3 /usr/share/php/symfony/vendor/swiftmailer/classes/Swift/Transport/AbstractSmtpTransport.php(441): Swift_Transport_EsmtpTransport->_doMailFromCommand('admin@chatthrea...')
#4 /usr/share/php/symfony/ve in /usr/share/php/symfony/vendor/swiftmailer/classes/Swift/Transport/AbstractSmtpTransport.php on line 406

After doing some digging around, it turns out Amazon’s SES service has a connection timeout which SwiftMailer was tripping up on. I couldn’t actually find an official published timeout limit but looking at the SwiftMailer code it seemed like it was possible to set a timeout inside Swift. We added a “timeout: 5” setting to our Symfony factories.yml file inside the SwiftMailer settings and it seemed to fix our issues.

Drupal 7: Fancy exposed view filters

By Ashish Datta | Leave a Comment on Drupal 7: Fancy exposed view filters

A couple of days ago, we were looking to theme a set of Drupal 7 View filters that were also using AJAX to load the new View content. By default, the filters were rendering as a set of radio buttons so by default we were getting HTML that looked like this:

The problem was that given the design mockups we had, styling the filters to match with only CSS was going to be impossible. Writing a jQuery plugin to dynamically replace the filters with custom HTML was pretty straightforward but the tricky part was getting a notification that an AJAX request had completed so that we could reload the filters.

We managed to do this using the Drupal.behaviors Javascript object using code that looks similar to what is below:

Anyway, nothing to crazy but there doesn’t seem to be a ton of documentation about the Drupal.behaviors object.

Symfony 1.4 partial model rebuilds

By Ashish Datta | Leave a Comment on Symfony 1.4 partial model rebuilds

A couple of months ago we started building out a Symfony 1.4 project for a client that involved allowing a “super admin” to add Doctrine models and columns at runtime. I know, I know, crazy/terrible/stupid idea but it mapped so well in to the problem space that we decided that a little “grossness” to benefit the readability of the rest of the project was worth it. Since users were adding models and columns at runtime we had to subsequently perform model rebuilds as things were added. Things worked fine for awhile, but eventually there were so many tables and columns that a single model rebuild was taking ~1.5 minutes on an EC2 large.

Initially, we decided to move the rebuild to a background cron process but even that began to take a prohibitively long time and made load balancing the app impossible. Then I started wondering is it possible to partially rebuild a Doctrine model for only the pieces that have changed?

Turns out it is possible. Looking at sfDoctrineBuildModelTask it looked like you could reasonably just copy the execute() method out and update a few lines.

Then, the next piece was just building the forms for the corresponding models. Again, looking at sfDoctrineBuildFormsTask it looked like it would be possible to extract and update the execute() method.

Anyway, without further ado here is the class I whipped up:

Using it is pretty straightforward, just call FastModelRebuild::doRebuild( array(“sfGuardUser”, “sfGuardUserProfile”) ); and thats it!

Anyway, fair warning I’d only do something like this if you “Know what you are doing” ™

As always, questions and comments are welcome.