Category: General

Joins, for better and for worse

By Matt Daum | 1 Comment on Joins, for better and for worse

Recently I’ve been having one hell of a time with SQL Joins. It isn’t that I don’t understand how to use them, its that they have came to haunt me in completely opposite instances.

First: An application I had written a while ago that was suppose to be kept small, and very simple which continually was enlarged and became very big was running slow. I upgrade the person to our faster server believing it was just that our server that is meant for static sites couldn’t handle it. While it did greatly reduce the load time of the site(1/6) it was still slow for me. I looked into it and found that after all the alterations of the application which we had done, that one page had 1600 queries at its current state.

Before you think down upon our development abilities and our code efficiency let me put everything in perspective. The project was started as a very simple CRUD only database, that we did basically free for the a client who was a friend. Shortly after it continually was being expanded upon, and the friend could only afford minimal costs, so he asked us to just “hack” it together as it was a prototype. Well about a month later after development, we had possible one of the most hacked together prototypes for him. He never wanted to redo it from scratch so we could architect it correctly due to the cost. For the prototype, it worked fine and very fast. However the prototype turned into a beta test for the friend. He added much more information to the database, and the inefficient queries began to show. Now the application is loaded with information, and the inefficient queries are terrible. We have told him that if he plans to use this we should rewrite it from scratch as many other cool features could be used.

So back to the problem. Well we attempted to add JOINs to the program which should cut the number of queries by at least 1/10th. However when I added the left joins via Criteria/Propel I found that it was adding the joins twice. The reason they needed to be left joins is because the foreign keys were not required,

dv madness – Holy Shit Batman!

By Ashish Datta | Leave a Comment on dv madness – Holy Shit Batman!

So a few days ago Daum and I decided to try and “hackathon” our way through one of our symfony projects.

We were developing on our (mt) gs server which is usually a stable box (until Mediatemple’s whole cluster goes down). Things were going fine until our Symfony controller started to hang and not serve any requests. The gs machine is a shared box so we couldn’t do anything invasive to try and diagnose our problems.

We opened a support ticket and then I tried switching the enviroment off “dev” to “test”, things got better for awhile and then promptly began crashing again. The hackathon was looking more like a crashathon so we decided to try and switch development over to our (mt) dv machine.

To try and salvage the day, we used the DNS rollover delay to discuss our living situation Anyway, so we finally had the new DNS set up and the symfony project migrated over to the dv. And this is where the chaos began.

So first problem, everything we symlinked to “httpdocs” return 403 FORBIDEN errors. We figured it must be the symlinks right? We checked httpd.conf and everything was fine. Daum was getting antsy so finally called (mt) support who informed us that they would open a ticket and get back to us. We tried everything we could think of, symlinks in httpdocs, symlinking to httpdocs, symlinks inside the directory. Symlinks were only working for targets INSIDE httpdocs…

After this fracas, Daum stumbled across a KB ticket that described how to configure specific vhosts on a dv. It turns out PLESK places additional httpd.conf files inside the new vhost directory!?

Ok fine, so we edited up the Plesk http conf to enable our symlinks and mess with the directory structure a bit. Now Symfony was vomitting on require() failing – great. Back to httpd.conf and it turns out Plesk enabled Open_Base_Dir restrictions for php inside the vhost to restrict it to only files inside the document root…

WOW – so after something like 3 hours of yelling at support and messing around we finally transitioned a project from our gs machine to our dv. Things learned:

1. You can’t have a local PEAR install of Propel and Symfony with a different version of Propel

2. On dv servers Plesk installs custom http.conf files PER vhost that customizes Apache and PHP behavior – none of these settings are editable from within Plesk.

3. Moving DNS records within Mediatemple takes time – switching the domain from the gs to dv took at least an hour.

Propel and Primary keys

By Matt Daum | Leave a Comment on Propel and Primary keys

Today I was writing an administrator backend for a project we have. I had the code:

$old_feeds=RssFeedRelationPeer::doSelect($c);
if(count($old_feeds)>0){
   foreach ($old_feeds as $old) $old->delete();
}

I was trying to remove some old foreign constraints before I deleted the main object, however, for the longest time the “$old” objects were not deleting, but no errors were being thrown. I did the usual debugging, added a die() statement inside the count, it was going there. I added a die statement in the foreach, it went there. I thought, “It must not be pulling the right ‘old’ objects.” I then added a $old->getName(); to see exactly what objects it was going through-they were the right ones. It made no sense. I next tired after the $old->delete();to add a $old->save();. The error I received was “You cannot save an object that has been deleted.” This didn’t make sense, since the object was still in the database. It hit me then, I had this problem in the past. Propel hates tables without primary keys. I quickly just added a simple primary key to the model, and the delete statements worked.

Moral: If dealing with Propel and you are getting some unexpected behavior, with zero errors, check to see if you have a primary key; it may save you hours of head banging.

Launched Wikia Evolution!

By Ashish Datta | Leave a Comment on Launched Wikia Evolution!

So I spent the summer working over at Wikia Inc as an engineer on their search project – http://re.search.wikia.com

Last month I wrote a Firefox 3.0 extension which launched yesterday – https://addons.update.mozilla.org/en-US/firefox/addon/8267

From the official Wikia blog:

One of our core values at Wikia Search is Community. We want everyone to be able to participate in the Wikia Search project. That’s why we are proud to introduce Wikia Evolution, our new Firefox toolbar. You can download it here, or via Mozilla’s Firefox Add-On Library.

The mission of Wikia Evolution: To empower users to interact with search.

We want to make it dead-simple for you to add URLs into our index under appropriate keywords. Already, we’re the cutting edge when it comes to incorporating user feedback into our search results, so much so that Google is experimenting with eerily similar features. Wikia Evolution pushes the envelope even further. It allows you to quickly and easily add the web page you are on into Wikia Search, directly from your browser, for whatever keyword is appropriate. Instant indexing! Then, you can modify the search result to make it really killer, all without leaving the page you’re on.

The release was even covered by Techcrunch!

http://www.techcrunch.com/2008/08/06/wikia-evolution-to-help-suck-search-data-from-google-yahoo/

Like everything at Wikia, the toolbar is open source. A public SVN is up at http://people.swlabs.org/~bartek/websvn/listing.php?repname=SearchUI&path=%2Fcool%2Ftoolbar%2F and you can download the code from the Mozilla add-ons site.