Strategies and Tips When Upgrading Legacy Systems

Part of the big plan in october was to migrate our web servers from rackspace to EC2 and to upgrade our websites to PHP5 and MYSQL5 (was PHP4 and MYSQL4 before). I had one month to create the magic and was the lead developer for the project – It was a huge challenge for me. Of course, it was a team effort. The migration wouldn’t be a success without the help from our system administrator and various departments.

Here, I would like to touch more on the strategies and things to take note while upgrading the code. Whether to dump your old application and buy/build a new one from scratch or to upgrade your legacy code to be compatible with the new environment has always been a tough business decision. If your business chose the later, this article may be of interest to you.

Upgrade Strategies

Nothing can be successful without a good plan. Project of such a scale requires even more planning.

* Knowing the scale of this project

I needed to identify what applications needed to be migrated to the new environment and the complexity of the code. I spend about 2 days drilling into code and identifying the problematic areas. The one month duration sounded tight but doable with great effort.

* Breaking down the projects into sub-projects

All big problems are made up of many small problems. If you solve all the small problems, you solve the big problem. I told myself that if I could break the code into pieces, I could potentially get more developers involved and speed up the whole upgrading process. I have identified the possibility of upgrading the forums and ecommerce site as individual smaller projects.

* Identifying critical and non critical areas

I listed down some features of the upgrade as critical and some as non-critical. Critical features were mainly customer related, ie user functionality and GUI. Non critical upgrades were mainly back end administration task – things like sales graph, newsletter archive and so on. In the event that we could not do a perfect migration, getting ticks for all the critical areas would still imply success. If not, we would need to rollback all the changes.

* Focusing on Objectives

The objective of the migration was to “Get it to work first, improve later”. I noticed how easy developers would have the urge to changes things and be side-tracked while going through the legacy code. There were many ideas of how a function could be improved to perform more efficiently, or how there might be security issues needed to be fix. Fixing them would be cool but then we would also need to test them and that would take time. I had to practice restraint – wrote all of them down and put them as a post-migration tasks.

* Checking if we were on track

We had a quick scrum everyday to make sure there were no show stoppers, In addition, We came up with a checklist and checkpoints to ensure that we were all progressing (A gantt chart would be nice). This was a rough example of what we had:

1. Redirect all outbound emails to local account – by sep 27 – sys admin
2. Sync code and DB to new server – by sep 28 – sys admin
3. upgrade forum – by 3 oct – dev
4. upgrade ecommerce site – by 7 oct – dev
5. upgrade admin area – by 12 oct – dev
6. Upgrade other websites – by 15 oct -dev
7. UAT for forum – by 10 oct – whole team
8. UAT for ecommerce site – by 12 oct – marketing
9. UAT for admin area – by 16 oct – marketing
10. UAT for all other websites – by 20 oct – marketing
11. Bug fixes for UAT Failures – by 25 Oct – dev
11. Migrate all Cron job and Testing – by 23 Oct – sys admin
12. EC2 configuration, disaster recovery process and Testing- by 25 Oct – sys admin
13. Add network monitoring for new server – by 20 oct – sys admin
14. Server maintenance announcement – on 25 Oct – marketing and publishing

* Preventing split brain situation

We had a code freeze 1 week prior the actual migration. I manually merged all the code changes from the old server to the new server. Luckily for me, there weren’t a lot of changes. On the final day, we pulled the old site down for maintenance and synced the database over.

* Check list for the migration day

This is probably the most important bit. We manually went through the process in our mind and tried to envisage where things could go wrong. Everyone knew what to expect and how to react if things went wrong. This was a rough example of what we had:

1. Switch on website maintenance mode in old and new servers.
2. Change email server config to all all outbound emails to go through.
3. Update DNS entries.
4. Dump databases from old to new server.
5. Update php.ini to use production settings.
6. Once DNS propagated, do quick testing on all critical areas.
7. Revert whole process if testing on critical areas fail.
8. Switch off website maintenance mode.
9. Live Running. Fight fire as they come.

Legacy Code Upgrade

We not only upgraded PHP, we also upgraded MYSQL and APACHE at the same time. PHP upgrade was more time consuming for us. There were no major dramas with MYSQL and APACHE. We had xdebug turned on and that saved us a lot of time in debugging.

* PHP 5 Syntax Upgrade

There were a few areas worth taking note of when doing the PHP4 to PHP 5 upgrade. The official PHP website actually provides a cool resource for those who are interested – http://us2.php.net/manual/en/migration5.incompatible.php (I should have googled around prior doing the migration).

Among the list of incompatible changes from the link above, I like to single out “array_merge” and “get_class” functions as the most commonly faced issue for us.
array_merge() syntax change was not so bad because if a non-array variable was passed, an a E_WARNING error was thrown and xdebug would be able to pick that up elegantly.

get_class() errors were harder to pick up because they often caused logic error in an if statement. For example, the setIdentity function would always return false in PHP5 since the class name was “Person” instead of “person”. This function worked in PHP4 though.


function setIdentity(&$identity)
{
if (get_class($identity) != "person")
{
return false;
}
else
{
$this->identity = &$identity;
return true;
}
}

* PHP.INI and APACHE MODULES

We compared the PHP and APACHE configuration files and ensure that both of them had all the required modules installed in the new server. All the PHP modules we installed were backwards compatible so far.

* MYSQL 5 Join Syntax

I noticed the sequence of the table in the joins was particularly important in MYSQL5 but not in MYSQL4. I had major issues with joins not working in MYSQL5 in quite a few places. This was a problematic query that worked in MYSQL4 and not MYSQL5.


SELECT thread.threadid, thread.forumid
FROM thread AS thread, subscribethread AS subscribethread,
LEFT JOIN deletionlog AS deletionlog
ON (deletionlog.primaryid = thread.threadid AND type = 'thread')
WHERE subscribethread.threadid = thread.threadid

MYSQL5 demanded deletionlog to LEFT JOIN with the immediate table before, ie thread. By changing the sequence of the table, vbulletin was able to run the query successfully.


SELECT thread.threadid, thread.forumid
FROM subscribethread AS subscribethread,
thread AS thread LEFT JOIN deletionlog AS deletionlog
ON (deletionlog.primaryid = thread.threadid AND type = 'thread')
WHERE subscribethread.threadid = thread.threadid

* Fixing PHP 4 Hacks
There were some interesting tricks that PHP4 could do but PHP5 couldn’t. One trick was to overwrite the $this variable completely. Running the MyTable function in this PHP 4 code will produce a fatal error.


class MyDB
{
// The database object, instance of Database
var $db;

function setDB(&$db, $tablename)
{
// What a sensational PHP 4 hack! Forces use of our static instance factory
// This will not work in php5 (Bernard Peh, 10 oct 2010)
$this = MySQLDB::factory($db, $tablename);
}
}

* Custom made libraries

Someone had manually created some custom pear packages and put them together with the pear libraries. We installed all the pear packages from scratch and manually moved all the custom libraries over.

* Hardcoded Paths or System Variables

We had hardcoded paths like /var/www/ or /mounted/images everywhere in the code base. We had to do a mass find and replace to get those paths to work in the new environment. Sed or Perl is extremely helpful in situations like this.

There were more issues and fixes but I thought I pretty much covered the main areas to look out for. Hopefully, this can reduce some sweat for people who wanted to upgrade their PHP4 system.

Conclusion

Considering the scale the project, the company considered the migration of the core websites to EC2 a success despite us needing to fight some fire on the first 2 days after the migration. Upon reflection, I would like to point out a few areas that we could have done in the past which would make our lives a lot easier when we did our system upgrade many years later.

* Version Control

If possible, try to version control everything before starting any project – from PHP code to system scripts to system config files. Version control provides an excellent way to track history and revert changes. Life would be so much easier for the developers if they had version control for the legacy sites.

* Documentation

I wasn’t only talking about code level documentation, but all business rules and rationale behind certain hacks should be documented as well. People tend to take shortcuts in times of emergency and forget to fix the hack later. Spending that extra mins can save a lot of headaches for future developers looking at that the same piece of code. Its all karma.

* Follow Up

Everyone went onto other projects after the migration and new priorities emerged. Ideally, we should have a proper debrief and talked about the follow up items. We still have a lot of fixes and ways to make our code more robust. Should they be prioritised as well?

Like it.? Share it:

One Response to Strategies and Tips When Upgrading Legacy Systems

  1. Disaster Recovery

    Very thoughtful and exciting publish! I’ve been looking round for the most effective Disaster Recovery and found a fantastic site for them and your website regarded like your readers might worth it so I’ve linked to it here!

    Check it out!

    – All one of the best