I’ve worked with a number of start-ups and young companies over the years and one thing I’ve noticed is that it is quite common that a smaller company does not think much about their IT. They are not insuring that it is properly structured, safe, and reliable. The smaller companies can become so focused on their product/business that they forget to make sure their underlying infrastructure is solid. Companies place all their energies on their code and their code quality; however, often overlook equally important setups: such as, the servers that run the code. Here are some of tips I frequently give companies.
Where is your code?
Small companies are often focused on making sure that their product is bug free, or doesn’t crash in certain browsers, etc. However, if tomorrow their outsourced developer disappeared or their server crashed, they might not have access to their code. I recommend that companies keep a hard copy document stating how to recover the code, which is in turn backed up across multiple company computers in which multiple people have access.
Knowing how to recover the code without outside help is critical. If the outsourced development firm/developer disappears, there is a conflict, or any other reason they are unreachable or will not cooperate, it is important to be able to have access to the code. Far too often outside contractors disappear and I’ve seen companies stuck being unable to get their own code.
Making sure there is always access to the up-to-date codebase, will save the headaches later. Also, the current developer won’t be needed to ask for the code if you want another developer to work on it or do a code review.
Where is your database?
Similar to above, what happens if for any reason (server crash, hackers, an act of God, etc.) your production database disappears? Do you and your colleagues know where to recover the information?
Often data is the most valuable possession of your business. Being able to recover it is critical. If you can’t recover any of your data, it is very possible that will be the end of your company. How often should you back your data up? That depends on your business. Some companies a daily backup will be plenty, however for others such as companies which pay people to take surveys, losing a day of data could equate to tens of thousands of dollars lost. This is something you’d need to discuss with your colleagues.
Another important part of database backups is making sure that they are stored for long enough. If you only keep one backup which happens at midnight each night what happens in the following situation? At 11:59 PM one night your database is compromised and most of the data is corrupted. That night when your backup runs, it will erase all your data.
Nowadays, data storage is very cheap (under 9 cents per month per GB), so keeping backups for plenty of time is well worth it. At a bare minimum you should make sure that your database backups are kept long enough that you’d notice any problem with the data before the oldest backup is removed. For example, for a forum you may want to keep at least 2 weeks of database backups. If someone deletes data from your forums, you’d most likely notice it within 2 weeks and can recover it. Again, data storage is cheap so keep backups for plenty of extra time (or forever).
Server Configurations
Using the cloud? Your own private hardware? Either way, an often overlooked backup is the backup of how your servers run. Without a server to run your code and serve your data, the other two become insignificant. Keeping up to date backups of your server configurations are critical. You need to make sure that you can always recover a failed server.
I’ve seen several Amazon EC2 instances fail. With the failure, sometimes companies are left scrambling to figure out how to get their site or product back online. How long can your company afford to be offline? If over an hour is too long, make sure that you always have up to date server images that you can immediately boot.
If you don’t have any documentation of how your server is setup, it’s likely that when it crashes you may not be able to get back online quickly. It’s even more likely you’ll forget different settings that will continue to cause the product or site to not fully function correctly.
Aside from keeping a backup of your server configuration, having access to the server permissioned correctly is equally important. An example of what I’ve seen: A company hires a new contractor who they want to work on a development version of the product while the main developer works on a different feature. The contractor mistakenly runs the wrong command on the server and wipes out the entire site. This happens all too often and is narrowly avoided at other times. Making sure that different users only have access to specific environments (such as the development environment above) on the server is very important. Everyone should have their own logins to the server so that you can remove any user without requiring everyone to know the latest password. I also recommend always using SSH Keys, they make weak passwords irrelevant and the server more secure.
Have a Written Disaster Plan
I’ve listed out some key tips above for how to keep your business running properly. However, all these should be summarized in one document. The document should be backed up itself and accessible to multiple people in your company which you trust. If something happens to you, you don’t want it to ruin your entire company, so distribution of the disaster recovery plan is critical.
Making sure you understand the plan and that the plan is effective is equally important. At least once a month it makes sense to go through the plan, make sure it is up to date, that all the different parts are actually working (you aren’t backing up a blank database due to a typo), and doing the steps to do a recovery.
While planning for disaster, failure, and unforeseen events takes time, it will pay for itself when something goes wrong.
Have any questions? Shoot me an email with any questions!
