Christopher J. Buckley

Upgrading part of your back-end application is a task that must be planned and implemented meticulously. This site has recently under-gone a code change to its Gallery application. Here’s how this site was able to perform a full code upgrade, as well as still achieving full availability across both servers.

Considerations

• Backup, backup, backup! We must make sure that our backups are held securely and cannot be corrupted. This database is small enough to use mysqldump, but incremental backups of the SQL binaries is a serious consideration in large scale enterprises
• Gallery is clustered across two Apache webservers: Any changes to my document root, will be quickly replicated across to the other webserver. This must not be allowed to happen until regression testing has taken place across one production server.
• MySQL replication: any code changes rolled out will immediately be replicated across the MySQL cluster. If the application upgrade fails, it will be a lot easier to roll back only one database. We must be careful to not push this change across our databases until regression tested across one production box.
• What role does our Load Balancer play? What rules do we need to implement to make sure traffic is directed accordingly…
• For file replication, this site uses rsync. To ensure code consistency during the maintenance window, this replication must be disabled.

Maintenance Actions

1. Perform a full backup of both the original Gallery code and the MySQL DB Binaries
2. Stop the replication slave running on Webserver 02. The code change will be rolled out across Webserver 01, so it’s important no changes are replicated within our maintenance window.
3. How to deal with our traffic across our load balancers? This will be covered in the next section. Ultimately, we want all requests that do not contain /gallery/ in the path to be spread across all webservers; all requests for /gallery/ must be directed across the node not under maintenance (webserver 02)
4. Stop the rsync file replication process on both servers.

Load Balancer specifics

• Create a rule that distributes requests for all requests that do not equal /gallery/ across all available webservers

if( ! string.contains( $path, "gallery" ) ) break;

• Make sure that my IP address can reach the maintenance pool

   if ( ! string.ipmaskmatch( $ip, "my.ip.address/32" ) ) {
        pool.use( "Sticky Pool" );
}     else {
        pool.use( "Maintenance" ); }

This ensures that my IP address reaches a pool containing one specific node – webserver01. All other requests are load-balanced to the ‘Sticky Pool’ which includes two nodes – webserver01 & 02, however 01 is put in a draining node (no further connections being sent to it).

So, both quickly and easily, using ZXTM’s trafficscript we’re able to create rules to efficiently distribute traffic appropriately.

End Result: All requests are still load-balanced across both webservers, with the exception of requests for $path == /gallery/. These requests are sent to one specific node (not under maintenance). This allows the webmaster (me!) to upgrade the code base, and regression test, across webserver 01.

Code Upgrade

Load Balancer rules enabled, we can safely proceed with our code upgrade. This is simply a matter of uploading the new codebase across the existing one. We can then proceed to the Gallery site knowing that we will go to webserver 01; able to both test and perform the application upgrade as applicable. This completed, and tested, we can now restart our rsync daemons, and allow the MySQL slave to start again on webserver 02. Within seconds the code change has been pushed out across the remaining webserver.

End Result: A complete success.