Congrats to Tēvolution

Our friends over at Tēvolution were written up on the incouraged communications blog today. Check out the post at http://www.incouraged.com/2011/06/08/tevolution-authentici-tea/

The post outlines their business and how the application we built supports their business goals. There are also some screenshots of the application in action.

If you haven’t already, check out their site at http://www.trackyourimpact.com/

Scaling HerCampus.com

Over the past month we’ve been working with Her Campus (HerCampus.com) to help them with issues they were having. When we started talking with Her Campus, we learned that they had different types of issues ranging from some Drupal based ones to actual server level problems. They were having some trouble keeping up to traffic demands, and if a traffic spike occurred their current infrastructure wasn’t sufficient.

After looking at their setup, we noted quite a few areas in which we could improve performance. The old setup was a fairly standard setup, a frontend server using Apache to handle HTTP requests and then a second server which was their MySQL database server. The servers were a 4 gig and 8 gig server respectively.

It was clear Apache was adding unneeded overhead, and wasn’t the best solution for them. We revamped their setup significantly. We switched them to Nginx + PHP-FPM. We immediately saw great improvement on from this change alone. However we wanted to get them to a single server, and to be able to handle traffic spikes with a single server.

We ended up doing the following:

  • Switch from Apache -> Nginx+PHP-FPM
  • Update all MyISAM tables to INNODB tables, and upgrade their MySQL to 5.1
  • Tune MySQL settings to fit their requirements
  • Update several tables adding indexes, reducing query time from in one case 34 seconds to 0.02 seconds.
  • Add the Boost Module to their setup
  • Update several of the view queries to be better written, added caching to each query.
  • Use ImageCache and sub-domains to load assets

After these updates we were able to move them from their two servers (8 gig and 4 gig) to a single server(4 gig). We have also reduced load times significantly. Their server loads dropped from 4-5 on average to 0.25. Recently they had an article on the Huffington Post and didn’t have any problems handling the 4x traffic spike they saw. At points we were seeing according to ChartBeat over 600 people on the site at once. The single server handled this without problems.

The updated infrastructure will give them a savings of about 75% from their previous setup. It also gives the users on the site a much faster and reliable experience.

We look forward to helping Her Campus with their continuing expansion of their site and user base!

Received problem 2 in the chunky parser

I was using cURL in PHP to POST some data to a URL earlier tonight and ran into this problem.

With VERBOSE on cURL was erroring with the following error:

"Received problem 2 in the chunky parser"

After some Googling it turns out this is a problem with how some servers respond with chunked encoding.

A simple fix for this is to set the HTTP version cURL is using to 1.0:

curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_0 )

It’s not pretty but hey it works!

Security questions, re-imagined

Earlier today we were discussing implementing “security questions” for a client of ours. The client felt that we should implement security questions so that users would have to answers one or more questions before taking certain actions on the site.

For those who aren’t familiar with the concept, some applications will ask users “security questions” at certain touchpoints in the application. The questions have been previously answered by the user and usually ask somewhat personal information like “what street did you grow up on?”, “what is the name of your favorite pet?”, “what was your high school’s mascot?”

As several security researchers have pointed out, the answers to these types of questions can be easily derived from a mix of a user’s social profiles and some social engineering. One of the most famous examples of this was the compromise of Sarah Palin’s email account during the lead up to the 2008 presidential campaign.

At the gym earlier, I started wondering about this problem and stumbled across what might be the basis of a novel solution.

The issue with the current solution is that the lexicon of questions asked are always pieces of personal information that users typically will share with the world. The obvious solution would be asking extremely personal questions like “who was your first kiss?”, “have you ever stolen something”, and so on. Unfortunately, these will undoubtedly make users uncomfortable and force the application to store extremely sensitive information.

What we’re really looking for is innocuous personal questions that users will not typically broadcast via social networks and also difficult to social engineer. With this in mind, my solution would be to ask questions that users don’t normally think about but when taken together, are identifiable enough to prove that a user is in fact genuine.

Here’s a few I thought of:

– Do you signal a “3” with your three index fingers or with your thumb and two index fingers? (For those who haven’t seen it, this is discussed at length in The Inglorious Bastards)
– Do you tie your shoes with “bunny ears” or with a loop?
– What knot do you use to tie your necktie?
– What type of seafood are you allergic to?
– What brand of refrigerator do you currently own?

Obviously, some of these are multiple choice questions which makes a probabilistic attack easier but by using a combination of multiple choice and open ended I think you could end up with a pretty strong solution.

Anyway, I’d love to hear any feedback and other good questions if anyone comes up with them.

MySQL and System Time

Recently for a client we had a very peculiar problem: a nightly script which checked if a person has done a certain action that day was always flagging everyone. We tested the script multiple times on our servers and it always worked fine. The query had something similar to this:

SELECT * FROM action_table WHERE DATE(action_table.time)=DATE(NOW())

Well after a while of trouble shooting we found out that the system clock on the clients server was skewed and in the wrong time zone. We synced the machines clock and update its time zone. Before it had been in UTC time and we switched it over to EST. What is interesting is that MySQL did not respect the new time zone of the clock, it was still reporting as if the system was set to UTC. We then did a soft restart(reload) on the MySQL service, but it still maintained that it was in a UTC timezone and not EST. It took a hard restart of the MySQL service to have it respect the EST timezone.

All in all, as far as we can tell you need to restart the MySQL service to have it respect a new timezone.