Symfony2 and Ordering of Relations and Collection Form Fields

Recently I was working on a project where I kept finding myself ordering a relation over and over by other than something than ID order (ie id= 1,2,3,4,5). For example, I always wanted my relation to be ordered by the ‘name’ field, rather than the ID or order it was inserted into the DB. Let’s take this schema as an example:

CREATE TABLE IF NOT EXISTS `post` (
`id` INT NOT NULL AUTO_INCREMENT ,
`name` VARCHAR(32) NULL ,
PRIMARY KEY (`id`) )
ENGINE = InnoDB
CREATE TABLE IF NOT EXISTS `post_attachment` (
`id` INT NOT NULL AUTO_INCREMENT ,
`name` VARCHAR(32) NULL ,
`url` VARCHAR(32) NULL ,
`post_id` INT NOT NULL ,
PRIMARY KEY (`id`) ,
INDEX `fk_post_attachment_post_idx` (`post_id` ASC) ,
CONSTRAINT `fk_post_attachment_post`
FOREIGN KEY (`post_id` )
REFERENCES `post` (`id` )
ON DELETE CASCADE
ON UPDATE CASCADE)
ENGINE = InnoDB

The issue is each time I attempted:

<?php
$attachments = $post->getPostAttachments();
foreach($attachments as $attachment)
{
echo $attachment->getId().' '.$attachment->getName()."\n";
}
//Output
// 1 d name
// 2 c name
// 3 a name
// 4 b name

I wanted the output to be in alphabetical order for example. To make this the default for that relation you can add the following annotation to your ‘Post’ entity:

<?php
//MyBundle/Entity/Post.php
...
class Post
{
...
/**
* @ORM\OneToMany(targetEntity=PostAttachment",mappedBy="post")
* @ORM\OrderBy({"name"="ASC"})
*/
private $post_attachments;
...
}

Now if you do “$post->getPostAttachments()” they’ll be automatically in order. The ‘@ORM\OrderBy’ column takes care of the ordering automatically. You can specify as many columns on the relation as you’d like there. In addition, this will make it so that all form collections on post with post_attachments are also ordered by name, rather than ID. This affects the relation call every time. If you are only looking into having it some of the time, look into using the repository to do the ordering for those calls.

Doctrine 1.2: To many columns causes findBy* to fail

Last week, one of our projects hit a pretty odd limit that I’d never expected to reach. The project is an analytics platform that allows admins to “pull” data from another, third party application. To accomplish this, the application allows admin users to dynamically add and remove columns from SQL tables and then dynamically chart these columns. Because of this, one of the tables had gotten over 350 columns which had all been created dynamically at runtime.

Anyway, things were working fine until last week when the application started throwing the following fatal error: “Fatal error: Uncaught exception ‘Doctrine_Table_Exception’ with message ‘Invalid expression found: ‘ in /usr/share/php/symfony/plugins/sfDoctrinePlugin/lib/vendor/doctrine/Doctrine/Table.php:2746″ Looking at the error, I noticed a warning was actually getting thrown right before the fatal error: “Warning: preg_replace(): Compilation failed: regular expression is too large at offset 32594 in /usr/share/php/symfony/plugins/sfDoctrinePlugin/lib/vendor/doctrine/Doctrine/Table.php on line 2745″ Looking through the code of Table.php, its clear that because “preg_replace” fails the $expression is subsequently blank which causes Doctrine to throw an error. I wanted to see how bad the regex was so I updated the Table.php to dump the expression. Here is what Doctrine was trying to run:

/(lsc_calculated_promoterscore_delta_innovativeness_formula|Lsc_calculated_promoterscore_delta_innovativeness_formula|lsc_calculated_promoterscore_delta_innovativeness_order|lsc_calculated_promoterscore_delta_favorability_formula|Lsc_calculated_promoterscore_delta_favorability_formula|Lsc_calculated_promoterscore_delta_innovativeness_order|Lsc_calculated_promoterscore_delta_favorability_order|

[Lots of columns...]

|Grid_mac_total|GridTotalTotal|GridTotalWinpc|Grid_total_mac|GridWindowsMac|grid_both_mac|grid_mac_ipad|audience_type|GridBothWinpc|GridBothTotal|Grid_both_mac|GridTotalIpad|Audience_type|WindowsosFy13|Grid_mac_ipad|grid_mac_mac|Grid_mac_mac|Program_type|program_type|AudienceType|GridMacTotal|GridBothIpad|GridTotalMac|GridMacWinpc|venue_child|ProgramType|GridMacIpad|Venue_child|GridBothMac|Program_id|program_id|VenueChild|GridMacMac|Fiscalyear|fiscalyear|is_locked|Is_locked|ProgramId|IsLocked|Country|country|venue|Venue|os|OS|id|Os|Id)(Or|And)?/

Looking at the php.net documentation for preg_replace and preg_match neither actually mention a hard limit on the size of a regex that can be compiled. Obviously there is a limit though and I imagine it must depend on the underlying RegExp library that your PHP is compiled against so it might be platform dependent.

As for solutions for this problem? The best solution for an extreme case like this is probably to just manually fill those in with real methods in the Doctrine_Table classes:

public static function findOneById( $id ){
return self::getInstance()->createQuery("u")->where("u.id =?", $id)->execute()->getFirst();
}

Symfony 1.4 partial model rebuilds

A couple of months ago we started building out a Symfony 1.4 project for a client that involved allowing a “super admin” to add Doctrine models and columns at runtime. I know, I know, crazy/terrible/stupid idea but it mapped so well in to the problem space that we decided that a little “grossness” to benefit the readability of the rest of the project was worth it. Since users were adding models and columns at runtime we had to subsequently perform model rebuilds as things were added. Things worked fine for awhile, but eventually there were so many tables and columns that a single model rebuild was taking ~1.5 minutes on an EC2 large.

Initially, we decided to move the rebuild to a background cron process but even that began to take a prohibitively long time and made load balancing the app impossible. Then I started wondering is it possible to partially rebuild a Doctrine model for only the pieces that have changed?

Turns out it is possible. Looking at sfDoctrineBuildModelTask it looked like you could reasonably just copy the execute() method out and update a few lines.

Then, the next piece was just building the forms for the corresponding models. Again, looking at sfDoctrineBuildFormsTask it looked like it would be possible to extract and update the execute() method.

Anyway, without further ado here is the class I whipped up:

<?php
class FastModelRebuild {
// Just return some standard Symfony configs
private static function getConfig(){
return array(
"data_fixtures_path" => array( sfConfig::get("sf_data_dir") . "/fixtures" ),
"models_path" => sfConfig::get("sf_lib_dir") . "/model/doctrine",
"migrations_path" => sfConfig::get("sf_lib_dir") . "/migration/doctrine",
"sql_path" => sfConfig::get("sf_data_dir") . "/sql",
"yaml_schema_path" => sfConfig::get("sf_config_dir") . "/doctrine"
);
}
/*
* This is the main function. Just pass in an array of names of models to rebuild.
*/
public static function doRebuild( $modelArray ){
// Grab the schema file and convert it to an array
$schemaYAML = sfYaml::load( sfConfig::get("sf_config_dir") . "/doctrine/schema.yml" );
$tempYAML = array();
// Pull out only the models that need to get rebuilt
foreach( $modelArray as $key ){
$tempYAML[ $key ] = $schemaYAML[ $key ];
}
// Create a temporary file for the partial schema
// For some reason it needs a ".yml" extension (don't know why)
$schema = tempnam( sys_get_temp_dir() , "sf" ) . ".yml";
file_put_contents( $schema, sfYaml::dump($tempYAML, 10) );
self::doModelRebuild( $schema );
sfAutoload::getInstance()->reloadClasses(true);
self::doFormRebuild( $schema );
unlink( $schema );
// Clear the cache to reset the autoloader
exec( "/usr/bin/php " . sfConfig::get("sf_root_dir") . "/symfony cc --app=frontend" );
}
private static function doFormRebuild( $schema ){
$config = sfContext::getInstance()->getConfiguration();
$config->fastRebuildSchema = $schema;
$databaseManager = new sfDatabaseManager( $config );
$generatorManager = new sfGeneratorManager( $config );
$generatorManager->generate("sfDoctrineFormGeneratorFast", array(
'model_dir_name' => "model",
'form_dir_name' => "form",
));
$properties = parse_ini_file(sfConfig::get('sf_config_dir').DIRECTORY_SEPARATOR.'properties.ini', true);
$constants = array(
'PROJECT_NAME' => isset($properties['symfony']['name']) ? $properties['symfony']['name'] : 'symfony',
'AUTHOR_NAME' => isset($properties['symfony']['author']) ? $properties['symfony']['author'] : 'Your name here'
);
// customize php and yml files
$finder = sfFinder::type('file')->name('*.php');
$filesystem = new sfFilesystem();
$filesystem->replaceTokens($finder->in(sfConfig::get('sf_lib_dir').'/form/'), '##', '##', $constants);
}
private static function doModelRebuild( $schema ){
$config = self::getConfig();
$builderOptions = sfContext::getInstance()->getConfiguration()->getPluginConfiguration('sfDoctrinePlugin')->getModelBuilderOptions();
$stubFinder = sfFinder::type('file')->prune('base')->name('*'.$builderOptions['suffix']);
$before = $stubFinder->in($config['models_path']);
$import = new Doctrine_Import_Schema();
$import->setOptions($builderOptions);
$import->importSchema($schema, 'yml', $config['models_path']);
// markup base classes with magic methods
foreach (sfYaml::load($schema) as $model => $definition)
{
$file = sprintf('%s%s/%s/Base%s%s', $config['models_path'], isset($definition['package']) ? '/'.substr($definition['package'], 0, strpos($definition['package'], '.')) : '', $builderOptions['baseClassesDirectory'], $model, $builderOptions['suffix']);
$code = file_get_contents($file);
// introspect the model without loading the class
if (preg_match_all('/@property (\w+) \$(\w+)/', $code, $matches, PREG_SET_ORDER))
{
$properties = array();
foreach ($matches as $match)
{
$properties[$match[2]] = $match[1];
}
$typePad = max(array_map('strlen', array_merge(array_values($properties), array($model))));
$namePad = max(array_map('strlen', array_keys(array_map(array('sfInflector', 'camelize'), $properties))));
$setters = array();
$getters = array();
foreach ($properties as $name => $type)
{
$camelized = sfInflector::camelize($name);
$collection = 'Doctrine_Collection' == $type;
$getters[] = sprintf('@method %-'.$typePad.'s %s%-'.($namePad + 2).'s Returns the current record\'s "%s" %s', $type, 'get', $camelized.'()', $name, $collection ? 'collection' : 'value');
$setters[] = sprintf('@method %-'.$typePad.'s %s%-'.($namePad + 2).'s Sets the current record\'s "%s" %s', $model, 'set', $camelized.'()', $name, $collection ? 'collection' : 'value');
}
// use the last match as a search string
$code = str_replace($match[0], $match[0].PHP_EOL.' * '.PHP_EOL.' * '.implode(PHP_EOL.' * ', array_merge($getters, $setters)), $code);
file_put_contents($file, $code);
}
}
$properties = parse_ini_file(sfConfig::get('sf_config_dir').'/properties.ini', true);
$tokens = array(
'##PACKAGE##' => isset($properties['symfony']['name']) ? $properties['symfony']['name'] : 'symfony',
'##SUBPACKAGE##' => 'model',
'##NAME##' => isset($properties['symfony']['author']) ? $properties['symfony']['author'] : 'Your name here',
' <##EMAIL##>' => '',
"{\n\n}" => "{\n}\n",
);
// cleanup new stub classes
$after = $stubFinder->in($config['models_path']);
$filesystem = new sfFilesystem();
$filesystem->replaceTokens(array_diff($after, $before), '', '', $tokens);
// cleanup base classes
$baseFinder = sfFinder::type('file')->name('Base*'.$builderOptions['suffix']);
$baseDirFinder = sfFinder::type('dir')->name('base');
$filesystem->replaceTokens($baseFinder->in($baseDirFinder->in($config['models_path'])), '', '', $tokens);
}
}

Using it is pretty straightforward, just call FastModelRebuild::doRebuild( array(“sfGuardUser”, “sfGuardUserProfile”) ); and thats it!

Anyway, fair warning I’d only do something like this if you “Know what you are doing” ™

As always, questions and comments are welcome.

Getting an extra ‘Invalid’ or other error on your symfony form?

On a project I’m working on I came across the following problem: we had a email field that we needed to be unique in our system, but we also made sure that it matched a confirm email field. A snippet of our form looks like this:

<?php
public function configure()
{
$this->setWidgets( array(
'email' => new sfWidgetFormInputText()
'confirm_email' => new sfWidgetFormInputText()
));
$this->setValidators(array(
'email'=> new sfValidatorAnd(
array(
new sfValidatorEmail( array('required' => true) ),
new sfValidatorDoctrineUnique(
array('throw_global_error' => true, 'model' => 'sfGuardUser', 'column' => 'username'),
array('invalid' => 'Sorry! A user with that email address already exists.')
)
)),
'confirm_email' => new sfValidatorEmail( array('required' => true) )
));
$this->validatorSchema->setPostValidator(
new sfValidatorSchemaCompare('password', '==', 'confirm_password')
);
}

When we submitted an email that was already in the system we got back two errors:

  • Sorry! A user with that email address already exists.
  • Invalid.

For a while I thought is there some extra validator somewhere that I left on? Where is this invalid coming from? It ended up being due to the way the validators work. If a validator throws an error it doesn’t return that validator’s value. So by the time it gets to the sfValidatorSchemaCompare post validator the value of `email` is null and `confirm_email` has the value you input, thus the seemingly extra ‘Invalid’ message.

This can be fixed easily with a sfValidatorCallback instead of the sfValidatorSchemaCompare. Here is the fix:

<?php
public function validateConfirmEmail( $validator, $values ){
if($values['email']&&$values['email']!=$values['confirm_email'])
{
throw new sfValidatorError($validator, 'Please confirm your email, currently they do not match!.');
}
return $values;
}

This way if the email is blank it doesn’t both making sure that the `email` matches the `confirm_email`. You don’t need to worry about a person just passing two blank emails as the earlier validator(the sfValidatorEmail requires it to be there and valid).

If you are getting an extra validation error, check your postValidators and how the values get to them.

Changing a Doctrine connection with the same name in single instance

On one of our projects that we use multiple connections that are defined at run time we recently were generating reports that required us to change a specific connection multiple times in a single run.    We noticed that even though we would define a new connection, it would not throw any errors but just continue to use the originally defined connection.  Here is how we were doing the connections:

<?php
$databaseManager = sfContext::getInstance()->getDatabaseManager();
$manager=Doctrine_Manager::getInstance();
$newConn = new sfDoctrineDatabase(array('dsn'=>'XXX','name'=> 'ExampleName'));
$newConn->connect();
$databaseManager->setDatabase('ExampleName',$newConn);

If you called the code above once, it would connect properly to the given DSN. However if you then called it a second time with a new DSN, it would not error and would simply just remain connected to the first DSN. After hunting around a bit it was the problem that Doctrine wasn’t assigning the new connection as the old connection was still open. To get around this we updated the code to the following:

<?php
$databaseManager = sfContext::getInstance()->getDatabaseManager();
$manager=Doctrine_Manager::getInstance();
// If the Doctrine Manager has the connection already close it so the new connection can be established
if($manager->contains('ExampleName'))
$manager->closeConnection($manager->getConnection('ExampleName'));
$newConn = new sfDoctrineDatabase(array('dsn'=>'XXX','name'=> 'ExampleName'));
$newConn->connect();
$databaseManager->setDatabase('ExampleName',$newConn);

You need to first check to see if the Doctrine Manager has the connection, as if you try to get a connection that doesn’t exist, it will throw an exception.

Hope this saves you some time!