Last week, one of our projects hit a pretty odd limit that I’d never expected to reach. The project is an analytics platform that allows admins to “pull” data from another, third party application. To accomplish this, the application allows admin users to dynamically add and remove columns from SQL tables and then dynamically chart these columns. Because of this, one of the tables had gotten over 350 columns which had all been created dynamically at runtime.
Anyway, things were working fine until last week when the application started throwing the following fatal error: “Fatal error: Uncaught exception ‘Doctrine_Table_Exception’ with message ‘Invalid expression found: ‘ in /usr/share/php/symfony/plugins/sfDoctrinePlugin/lib/vendor/doctrine/Doctrine/Table.php:2746″ Looking at the error, I noticed a warning was actually getting thrown right before the fatal error: “Warning: preg_replace(): Compilation failed: regular expression is too large at offset 32594 in /usr/share/php/symfony/plugins/sfDoctrinePlugin/lib/vendor/doctrine/Doctrine/Table.php on line 2745″ Looking through the code of Table.php, its clear that because “preg_replace” fails the $expression is subsequently blank which causes Doctrine to throw an error. I wanted to see how bad the regex was so I updated the Table.php to dump the expression. Here is what Doctrine was trying to run:
/(lsc_calculated_promoterscore_delta_innovativeness_formula|Lsc_calculated_promoterscore_delta_innovativeness_formula|lsc_calculated_promoterscore_delta_innovativeness_order|lsc_calculated_promoterscore_delta_favorability_formula|Lsc_calculated_promoterscore_delta_favorability_formula|Lsc_calculated_promoterscore_delta_innovativeness_order|Lsc_calculated_promoterscore_delta_favorability_order| [Lots of columns...] |Grid_mac_total|GridTotalTotal|GridTotalWinpc|Grid_total_mac|GridWindowsMac|grid_both_mac|grid_mac_ipad|audience_type|GridBothWinpc|GridBothTotal|Grid_both_mac|GridTotalIpad|Audience_type|WindowsosFy13|Grid_mac_ipad|grid_mac_mac|Grid_mac_mac|Program_type|program_type|AudienceType|GridMacTotal|GridBothIpad|GridTotalMac|GridMacWinpc|venue_child|ProgramType|GridMacIpad|Venue_child|GridBothMac|Program_id|program_id|VenueChild|GridMacMac|Fiscalyear|fiscalyear|is_locked|Is_locked|ProgramId|IsLocked|Country|country|venue|Venue|os|OS|id|Os|Id)(Or|And)?/
Looking at the php.net documentation for preg_replace and preg_match neither actually mention a hard limit on the size of a regex that can be compiled. Obviously there is a limit though and I imagine it must depend on the underlying RegExp library that your PHP is compiled against so it might be platform dependent.
As for solutions for this problem? The best solution for an extreme case like this is probably to just manually fill those in with real methods in the Doctrine_Table classes: