Folks,

I just wanted to let you know, it looks like San Francisco is having a MAJOR power event, with outages from the Financial district all the way down to Daly City. One of our colos at 365 Main Street has experienced a power outage (never mind that they always swear up and down that this kind of event can’t possibly happen, oh no, they have multiple redundant systems and they charge us up the wazoo to make sure that we’ll have business continuity, so of course, this isn’t really happening, oh yes) however, our other data centers are all up and running, so we hope to be back up and running as quickly as possible.

I’ll keep you all updated on progress, and I appreciate you bearing with us as we work our way through this…

Dave

Sites affected include Technorati, Netflix (these display nice “We’re Dead” pages), Typepad, LiveJournal, Sun.com, and Craigslist (these just time out).

I, personally, am amazed that none of these significant websites employ global server load balancing. Major load-balancer vendors such as F5, Cisco, NetScaler and Zeus all offer first-class GSLB offerings.

Using TrafficScript you can prevent hot linking like so:

$referrer = string.lowerCase( http.getHeader( "Referer" ));
$host     = http.getHostHeader();
$path     = http.getPath();
$redirect = "/blocked.png";

# Never block requests with no Referer:
if( $referrer != "" ) {
   # Referer header should contain the hostname of the site
   if( !string.contains( $referrer, $host )) {
      # Only block images
      $path = http.getPath();
      if( string.regexMatch( $path, "\\.(jpe?g|png|gif)" )) {
         # OK, block this image (but allow our special blocked image)
         if( $path != $redirect ) {
            # Redirect them to our 'Go away' image
            http.redirect( $redirect);
         }
      }
   }
}

..and if you’re quick enough, you can catch the cheeky so and so’s at altplustab who decided to use an image of mine on their front-page. I wonder how long it’ll be before they remove it?

Here’s a screenshot for when they stop stealing my bandwidth and remove the hotlink.
 

Here’s another person abusing my bandwidth, not a surprise from a MySpace user, mind.
 

This paper has taken some time coming, but the make up is clear:

For me, there are two notable aspects of this paper:

1. Firstly, F5 have pushed further out ahead of their competition. Having had first hand experience of their products, I would agree with Gartner’s assessment, but also at the same time have some reservations (largely involving the iRules scripting language). Something which did irk me was Gartner’s assertion that:
   

   The Big-IP product is so feature-rich it can be intimidating to some customers

   No – the product’s GUI just isn’t that intuitive. Really – is an overview page of global health status too much to implement? [1]

2. Secondly, as anyone who has an understanding of ADC products has realised for quite a while (or, should have done), Cisco is not a lead vendor in this space. They are a niche vendor for ADC. Having had personal struggles with Cisco’s kit in ADC, it’s refreshing to see them (finally) pushed to the appropriate market segment.

   Ultimately, enterprise’s should not be deploying Cisco equipment in ADC. Look to the visionary vendors for the right solution.

3. A personal one this, but it’s nice to see Zeus Technology still holding firm in the market-leading visionary space. This site uses ADC from Zeus, and very fine it is too.

[1] In response to the comment left by Lori I feel I must clarify my comments and also raise some of my own.

1. To clarify: I am not asserting that BigIP’s feature-set isn’t as rich as Gartner, and F5, claim. Indeed, it is – and more! My two major issues with BigIP are, as I blogged previously:
   • The clunky iRules scripting language which Gartner highlights for BigIP, but fails to mention the perl-like, and intuitive, TrafficScript language of ZXTM. iRules is not an easy language to pick up in my opinion. I would love to hear other sysadmin and programmers’ opinions on this.
   • The non-intuitive interface makes using the feature richness of the product that little bit harder. As a ardent supporter of the command line, I’m the first person to appreciate GUI intuitiveness does not necessarily equal feature richness. However, F5 has a feature rich product, but they can do better with their GUI. That was the point I was making. Apologies if the original article was written in haste.

     Yes, as Lori notes:
     

     Intuitiveness is subjective, not objective, and not a good measuring stick. if intuitiveness was a primary buying criteria no one would ever purchase anything Cisco

     However, I feel this doesn’t address the real reason people buy Cisco. It’s not because it’s ‘better’ than their competitors’ products, but simply because you don’t get fired for buying Cisco. A depressing statement, but true.

     Ask yourself, would you deploy Cisco’s CSS over ZXTM/Radware/Akamai in an SME DC? Well, lots of firms do. Why? Because you won’t get fired for doing so. If you use a smaller player, you’re perceived to be taking a business risk. Lots of firms don’t suffer from this thinking, but some do. This is only my personal experience, I would appreciate other people’s experiences.

2. Interestingly, Lori writes that
   

   BIG-IP and ZXTM are not competitors in the same market segment, primarily because BIG-IP is over-kill for the SMB market in which ZXTM continues to dominate and target. These two products don’t compete directly, and both companies acknowledge that willingly…

   I have no issue with this comment. It’s a pleasure to read, in fact. I have been struggling for months now to set different expectations between these two different products in my day-to-day role.

Lori, your post has highlighted that I wrote my original article in haste. I hope that the above clarifies and appreciates the concerns you raised.

As a result of it’s ease of customisations, the developer Daniel B. Cid, with a little bit of help from myself, have implemented supported rule-sets for my former employers’ products: Zeus WebServer and ZXTM. Both products are widely deployed across many enterprise environments; adding specific rulesets for their software is one which I hope assists all fellow sysadmins tasked with running infrastructure using Zeus software.

Download

You can download the latest snapshot of OSSEC from this page.

Implementation

After running install.sh you can add the Zeus rulesets as you would any other supported rule set.

Example:

<localfile>
<log_format>syslog</log_format>
<location>/usr/local/zeus/zxtm/zxtm/log/errors</location>

<localfile>
<log_format>syslog</log_format>
<location>/usr/local/zeus/zws/web/log</location>

Alerts

Below you can clearly see the ruleset alerting you, quickly and efficiently, to a failure.
 

Help..

Any questions, just ask!

Considerations

• Backup, backup, backup! We must make sure that our backups are held securely and cannot be corrupted. This database is small enough to use mysqldump, but incremental backups of the SQL binaries is a serious consideration in large scale enterprises
• Gallery is clustered across two Apache webservers: Any changes to my document root, will be quickly replicated across to the other webserver. This must not be allowed to happen until regression testing has taken place across one production server.
• MySQL replication: any code changes rolled out will immediately be replicated across the MySQL cluster. If the application upgrade fails, it will be a lot easier to roll back only one database. We must be careful to not push this change across our databases until regression tested across one production box.
• What role does our Load Balancer play? What rules do we need to implement to make sure traffic is directed accordingly…
• For file replication, this site uses rsync. To ensure code consistency during the maintenance window, this replication must be disabled.

Maintenance Actions

1. Perform a full backup of both the original Gallery code and the MySQL DB Binaries
2. Stop the replication slave running on Webserver 02. The code change will be rolled out across Webserver 01, so it’s important no changes are replicated within our maintenance window.
3. How to deal with our traffic across our load balancers? This will be covered in the next section. Ultimately, we want all requests that do not contain /gallery/ in the path to be spread across all webservers; all requests for /gallery/ must be directed across the node not under maintenance (webserver 02)
4. Stop the rsync file replication process on both servers.

Load Balancer specifics

• Create a rule that distributes requests for all requests that do not equal /gallery/ across all available webservers

if( ! string.contains( $path, "gallery" ) ) break;

• Make sure that my IP address can reach the maintenance pool

   if ( ! string.ipmaskmatch( $ip, "my.ip.address/32" ) ) {
        pool.use( "Sticky Pool" );
}     else {
        pool.use( "Maintenance" ); }

This ensures that my IP address reaches a pool containing one specific node – webserver01. All other requests are load-balanced to the ‘Sticky Pool’ which includes two nodes – webserver01 & 02, however 01 is put in a draining node (no further connections being sent to it).

So, both quickly and easily, using ZXTM’s trafficscript we’re able to create rules to efficiently distribute traffic appropriately.

End Result: All requests are still load-balanced across both webservers, with the exception of requests for $path == /gallery/. These requests are sent to one specific node (not under maintenance). This allows the webmaster (me!) to upgrade the code base, and regression test, across webserver 01.

Code Upgrade

Load Balancer rules enabled, we can safely proceed with our code upgrade. This is simply a matter of uploading the new codebase across the existing one. We can then proceed to the Gallery site knowing that we will go to webserver 01; able to both test and perform the application upgrade as applicable. This completed, and tested, we can now restart our rsync daemons, and allow the MySQL slave to start again on webserver 02. Within seconds the code change has been pushed out across the remaining webserver.

End Result: A complete success.

The articles i have written are contained on Zeus’ Knowledgehub. Any questions, just ask!

The below script is written in Trafficscript language. What’s really useful about this script is that a client request for any URL not pre-fixed by www. is catenated and 301′d back to the server, prepended with the original path request.

# Cache Host header being issued by the client
$host = http.getHeader( "Host" );

# This line passes both the path and query string to $url,
# meaning that a request for /path/to/output?is-here.php
# is placed into the below variable.
$url = http.getRawURL();

# log.info( "url: ".$url ); #  Optional log output
# Use a Perl Compatible regular expression to match our domain name.
if( string.regexmatch( $host, "^domain-name$" ) ) {

# Re-write the variable cached in $host to include a prefix of 'www.'
$location = string.regexsub( $host, "domain-name.com", "www . ".$host );
http.sendResponse( "301 Moved Permanently", "text/html", "",
"Location: http://".$location . $url);
}