Notable pieces from Bruce’s essay are outlined herein:

And that brings me to our first mistake: for a time, there was a conflict between Open Source and Free Software evangelism. My intent has always been for Open Source to simply be another way of talking about Free Software, tailored to the ears of business people, and that it would eventually lead them to a greater appreciation of Richard Stallman’s arguments. This has come to pass, and I hope you’ll continue to make it so. One only had to witness the attendance of the GPL 3 committees to see that the importance of FSF’s work was appreciated by the largest of corporations.

For me, this is the single most important message to make people aware of: Open Source is a ‘business-friendly’ term to spread the philosophy and principles of Free-Software. The GNU foundation does tremendous work in enabling, and preserving, software freedom’s for users. It is of great sadness to me that people I talk to are at times completely unaware of the term ‘Free-Software’, instead thinking that this software is ‘Free as in no-cost’, rather than ‘Free as in Freedom’. Richard Stallman is owed an extremely large debt for his vision in creating the FSF and ensuring software freedom’s for all.

In contrast, we have not yet achieved the penetration that we might have desired on user desktop systems, at least if you don’t count the fact that Free Software provides a large part of Apple’s MacOS today, and critical elements of Microsoft Windows as well. Both companies have been forced to develop strategies to live with us, some of them less comfortable than others. Today we are seeing much of the value of software move from the desktop to the network, an area in which we are already entrenched. This can only lead to the expansion of Open Source on the systems in individual user’s hands.

This is a very realistic point: GNU+Linux penetration on the desktop is growing daily; users are realising that proprietary software and vendor lock-in is something that they no longer have to be part of – they can possess software freedom with GNU+Linux – primarily first on Server, but now on Desktop. The next decade will be pivotal in ensuring GNU+Linux on the Desktop penetrates to the masses.

One recent phenomenon has been the appearance of government officials openly on the stage at conferences concerning Free Software. Of late, it’s my turn to speak when the minister has finished his greeting, and they are always announcing some national government initiative concerning Open Source. OK, I speak outside of the U.S. a lot, but even in the U.S. we are seeing Linux (and presumably the GNU system)

Free-Software is here to stay: it is used both by Enterprises and Governments. Ignore the FUD from Proprietary vendors who tell you that Free and Open Source Software bears a heavier risk than using proprietary software.
Remember also, when you are using a ‘Linux’ machine, you are more than likely using the GNU System, be your distribution from Red Hat, Debian, Mandriva, Gentoo, Slackware et al. Go and tell people about the GNU System!

Microsoft remains a problem, as the bastion of the old way of thinking about software, and as the epitome of the old school of dirty corporate fighting. Their current strategy seems to be to poison us with money, most recently by making patent agreements with a number of Linux distributions. These agreements go against the spirit of the software licenses used by our developers, and were perhaps intended to dissuade developers from contributing their work. To this end, Microsoft poured more money into Novell last year than Novell’s annual profit – indeed Novell would have had no annual profit without Microsoft.

A very articulate observation from Perens.

Perens closes by stating that:

So, you can see that the future will present its challenges for Open Source. We could never have forecast how big we would become during Decade Zero of Open Source. But we’ve built tremendous strength, to the point that we can consider much larger tasks. Join us now, as we enter Decade One.

Please do take time to read the original article located at Perens’ personal website.

When using a separated tier network of postfix servers, ie – a cluster at the front, which receive from a cluster in a segmented network, you will need to use the relayhost variable to daisy link the two clusters together. Usually, you may be tempted to simply submit on port 25. This was previously acceptable, but now a concerted drive is being made towards the correct port – 587. When using DKIM implementations such as DKIM proxy mail will not be signed on port 25 without some hackery in master.cf [this is strongly advised against]. As Jason states:

. . .The point is we don’t want to sign mail from untrusted sources, and that’s what could happen if you direct that mail through dkimproxy.out.

So how do we ensure our chain of servers signs mail submitted? Easy!

relayhost = [smtp.domain.tld]:587

You can now sit back and watch as all your mail is digitally signed by what-ever DKIM implementation you have chosen.

• PDF & FDF: usually a blank e-mail with a PDF attachment.
• Greeting Card: invitations from an ‘old friend’ to go to a website with a numerical http address, usually containing malware.
• Image Spam: A gif or png attached to a, usually – though not always, blank e-mail ready to sell the latest software or stock.
• Obfuscate Words: Lines of text take this format, N o*t o,n-l-y d o-e's t,h+i s f i+r*m h+a'v_e grea_t fundamenta,ls*,.

I’m going to show you how using the free-software package, SpamAssassin, you can successfully neuter these 4 major spam attacks.

• Greeting Card: Can be easily defeated using postcards.cf.
• Image Spam: These quickly become extremely popular, but have now decreased in prevalence after very successful methods were implemented to combat them. For SA, use the module Imageinfo: Imageinfo.pm and Imageinfo.cf supplied by SARE Rules Emporium.
• PDF & FDF: Can be successfully discarded by using PDFInfo.pm and PDFInfo.cf again supplied by SARE Rules Emporium.
• Obfuscate Words: These have recently hit, and hit hard. Spammers, seemingly bewildered by their inability to get through current filters using the above popular methods, have now resorted to the old way of securing spam delivery: obfuscation. The good news is that this, again, is easily defeatable. The ruleset(s) originally written by Jennifer Wheeler are mirrored locally by this site.
  1. chickenpox.cf: [words obfuscated by non word characters] Th1s|s a v3ry h4ndy se7 t0 c@tch th!s 50rt 0f (rap.
  2. backhair.cf: [words obfuscated by nonsense html tags] 
     Y<oivugriub>oucou<iuqgheriugv9h>ld rea<y>llyu<owiuer88>se this.

Game on, spammers!

The RT wiki has instructions on setting up your MTA to ‘pipe’ the e-mails into RT:

You need to tell your Mail Transfer Agent (ex sendmail, postfix, or qmail) how to forward messages to RT’s mail gateway. To do this, create an aliases in your system’s mail aliases file. Here’s an example, which routes mail to the mailbox [=rt@example.com] (and [=rt-comment@example.com]) into new tickets in the RT queue named General. Note that the queue name is case-insensitive.

Add the following lines to /etc/aliases (or your local equivalent such as /etc/mail/aliases):

rt: "|/opt/rt3/bin/rt-mailgate --queue general --action correspond --url http://localhost/rt"
rt-comment: "|/opt/rt3/bin/rt-mailgate --queue general --action comment --url http://localhost/rt"

These instructions are accurate, but rely on Postfix not having implemented Virtual Domains. Here’s an example configurationfrom main.cf where virtual domains have been implemented:

virtual_transport = virtual
virtual_uid_maps = static:5000
virtual_mailbox_maps = mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf
virtual_alias_maps = mysql:/etc/postfix/mysql_virtual_alias_maps.cf

This configuration will invoke the virtual transport within Postfix. A virtual transport is unable to perform the necessary pipe (“|”) to the rt-mailgate binary. Only the local transport is able to perform a pipe. So, the question is: how do we run virtual domain(s), but still invoke the local transport delivery method to successfully perform the pipe into rt-mailgate?

A Virtual to Local Rewrite Solution

Here’s a quick walkthrough on what steps you need to put in place to ensure that a mail to the (virtual) domain of rt-test@domain.tld is successfully piped as per the RT wiki instructions above.

1. Create an /etc/postfix/aliases file.
2. Within this file add entries that follow this format:

   rt-test                       rt@rt.domain.tld
   support                       support@rt.domain.tld
   abuse                         abuse@rt.domain.tld
   # and so on...
   

3. postmap /etc/postfix/aliases
4. Within /etc/aliases create the pipe aliases referred to in the RT wiki:

   support: "|/opt/rt3/bin/rt-mailgate --queue 'Support' --action \
   correspond --url http://rt.domain.tld/"
   rt: "|/opt/bin/rt-mailgate --queue 'General' --action \
   correspond --url http://rt.domain.tld/"
   abuse: "|/opt/rt3/bin/rt-mailgate --queue 'Abuse'  --action \
   correspond --url http://rt.domain.tld/"
   # Ensure there are no line breaks..
   

5. Run the newaliases command.
6. Insert the appropriate configuration amendments to main.cf

   # To ensure local delivery, rt.domain.tld must be added to
   # $mydestination
   mydestination = localhost localhost.localdomain rt.domain.tld
   # /etc/postfix/aliases is added:
   virtual_alias_maps = hash:/etc/postfix/aliases
   mysql:/etc/postfix/mysql_virtual_alias_maps.cf
   # alias_maps is what is READ by delivery agents etc.
   alias_maps = hash:/etc/aliases
   # alias_databases is what is WRITTEN by newaliases
   alias_database = hash:/etc/aliases
   # masquerade as @rt.example.com unless also on this list,never root
   masquerade_domains = rt.domain.tld
   masquerade_exceptions = root
   

7. Save the file, then reload Postfix’s configuration: /etc/init.d/postfix reload
8. Send an e-mail to support@domain.tld and observe Postfix working its wonders..
   
   Aug 19 21:12:18 solo postfix/local[2972]: 2B91710EB92: to=<support @rt.domain.tld>, orig_to=<support@cjbuckley.net>, relay=local, delay=17, delays=17/0.05/0/0.54, dsn=2.0.0, status=sent (delivered to command: /opt/rt3/bin/rt-mailgate --queue 'Support' --action correspond --url http://rt.domain.tld/)

I wrote this article mainly because I see the question oft repeated on the Postfix Users Mailing List. Any comments and (especially) improvements welcome.

I know from knowledge of MailScanner that the default conf file blocks password protected archives by default. However, upon looking in the conf file I did not see the entry to over-ride this. Knowing that this over-ride entry exists I checked the conf file on my home e-mail server from MailScanner 4.60 .

chris@solo> grep 'Password' /etc/MailScanner/MailScanner.conf
Allow Password-Protected Archives = no

Interestingly, this attribute is not present in the Ubuntu package of MailScanner

How To Add Exceptions To Password Protected Archives

1. Within /etc/MailScanner/MailScanner.conf add entry:
   Allow Password-Protected Archives = %rules-dir%/password.protected.rules
2. Create the file /etc/MailScanner/rules/password.protected.rules
3. Within this file add entries for IP’s or e-mail addresses you wish to allow password protected archives from. An example of this file could be:
   
   From: 152.78. yes
   From: 130.246. yes
   From: chris.buckley@domain.tld yes
   FromorTo: allow-zips@domain.tld yes
   FromOrTo: default no
   
4. Save this file, then restart MailScanner.

NB: Post updated, see Julian’s comment below.

If, after setting up your new extensions to the SMTP protocol, you wish to verify everything is working correctly then I thoroughly recommend a blank e-mail to this address: check-auth@verifier.port25.com

Let’s run through the results of this e-mail for my domain, cjbuckley.net:

 Summary of Results
SPF check:          pass
DomainKeys check:   neutral
DKIM check:         pass
Sender-ID check:    pass

Immediately, we can see that the implementations have been successful – this is a good sign!

SPF check details
Result:         pass
ID(s) verified: smtp.mail=example@cjbuckley.net
DNS record(s):
cjbuckley.net. 300 IN TXT "v=spf1 redirect=_spf.cjbuckley.net"
_spf.cjbuckley.net. 300 IN TXT "v=spf1 ip4:87.127.106.176/29 ip4:84.45.189.64/29 include:ukfsn.org -all"

The detail outputted from this SPF check is impressive; as we can see, my domain initially does a redirect to _spf.cjbuckley.net (note the underscore – these _are_ valid in TXT records). This allows many domains I am the hostmaster for (corporate ones, usually) to be easily managed.

For example, I use redirect: _spf.mycorporatedomain.com for all my corporate domains, this cleans up the SPF records and allows easy management of the record database – all other domains can be redirected to _spf.mycorporatedomain.com. It works very well, and is something I noticed Google taking advantage of, initially.. :-)

Next up – DomainKeys:

DomainKeys check details:
Result:         neutral (message not signed)
ID(s) verified: header.From=example@cjbuckley.net
DNS record(s):

The checker confirms that I do not have DK implemented – indeed, this is correct. DKIM replaces DK, and is now being driven forward to replace Yahoo! DomainKey’s. Note: gmail.com only verifies DK signatures, not DKIM yet – a bit disappointing, though they _do_ sign both with DKIM and DK.

 DKIM check details:
Result:         pass
ID(s) verified: header.From=example@cjbuckley.net
DNS record(s):
beta._domainkey.cjbuckley.net. 300 IN TXT "k=rsa; t=y;p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDiIfv9vqSRUo9L8
ztX/C4gfCD/Ivt8eAGyQbjJ2g4Rq764NPwauj5/sY2AfMrFPqhA0ieXWtmJy2gFS
c4ZlxT8KYaFsJATOpJfYAXUtzmmQ8+RcioyeN3LjzNhm1gUYyJI1Lw0yD2y+k
dN3YxY4NZ0esMXrKbsngTl3pNcNCNxXwIDAQAB"

We can see that our DKIM policy has been verified successfully. As dkim.org states:

DKIM lets an organization take responsibility for a message. The organization taking responsibility is a handler of the message, either as its originator or as an intermediary. Their reputation is the basis for evaluating whether to trust the message for delivery.

Given this statement, e-commerce, financial and banking organisations should have either implemented or be on their way to planning an implementation of DKIM. Personally, I find SPF flawed – DKIM is the wiser choice.

I’m hoping to blog a dkimproxy implementation guide shortly, as the published guide has a few advisories I have issue with and would like clarified.

Comments, welcome!

[Munin] The monitoring tool surveys all your computers and remembers what it saw. It presents all the information in graphs through a web interface. Its emphasis is on plug and play capabilities. After completing a installation a high number of monitoring plugins will be playing with no more effort.

Using Munin you can easily monitor the performance of your computers, networks, SANs, applications, weather measurements and whatever comes to mind. It makes it easy to determine “what’s different today” when a performance problem crops up. It makes it easy to see how you’re doing capacity-wise on any resources.

So, i’ve implemented Munin across my platform of servers – as I write this I have implemented monitoring of my load-balancer and web/mailserver box. There is a secondary web-server to be bought online soon.

Of special interest to me in this program is the ability to track the replication of MySQL, I can now easily see (and be alerted to via Nagios) any lag in my replication ’seconds behind master’ time.

Please take a look at the monitoring information at: monitors.cjbuckley.net

As a result of it’s ease of customisations, the developer Daniel B. Cid, with a little bit of help from myself, have implemented supported rule-sets for my former employers’ products: Zeus WebServer and ZXTM. Both products are widely deployed across many enterprise environments; adding specific rulesets for their software is one which I hope assists all fellow sysadmins tasked with running infrastructure using Zeus software.

Download

You can download the latest snapshot of OSSEC from this page.

Implementation

After running install.sh you can add the Zeus rulesets as you would any other supported rule set.

Example:

<localfile>
<log_format>syslog</log_format>
<location>/usr/local/zeus/zxtm/zxtm/log/errors</location>

<localfile>
<log_format>syslog</log_format>
<location>/usr/local/zeus/zws/web/log</location>

Alerts

Below you can clearly see the ruleset alerting you, quickly and efficiently, to a failure.
 

Help..

Any questions, just ask!

debug1: Local version string SSH-2.0-OpenSSH_4.5
debug1: An invalid name was supplied
Cannot determine realm for numeric host address

..but was unable to find the fix. I tried UseDNS no within sshd_config, no luck.

However, I’ve stumbled across the answer! On your client Mac, add your server to your /etc/hosts file. With that done, try connecting via SSH again – all fixed!

Considerations

• Backup, backup, backup! We must make sure that our backups are held securely and cannot be corrupted. This database is small enough to use mysqldump, but incremental backups of the SQL binaries is a serious consideration in large scale enterprises
• Gallery is clustered across two Apache webservers: Any changes to my document root, will be quickly replicated across to the other webserver. This must not be allowed to happen until regression testing has taken place across one production server.
• MySQL replication: any code changes rolled out will immediately be replicated across the MySQL cluster. If the application upgrade fails, it will be a lot easier to roll back only one database. We must be careful to not push this change across our databases until regression tested across one production box.
• What role does our Load Balancer play? What rules do we need to implement to make sure traffic is directed accordingly…
• For file replication, this site uses rsync. To ensure code consistency during the maintenance window, this replication must be disabled.

Maintenance Actions

1. Perform a full backup of both the original Gallery code and the MySQL DB Binaries
2. Stop the replication slave running on Webserver 02. The code change will be rolled out across Webserver 01, so it’s important no changes are replicated within our maintenance window.
3. How to deal with our traffic across our load balancers? This will be covered in the next section. Ultimately, we want all requests that do not contain /gallery/ in the path to be spread across all webservers; all requests for /gallery/ must be directed across the node not under maintenance (webserver 02)
4. Stop the rsync file replication process on both servers.

Load Balancer specifics

• Create a rule that distributes requests for all requests that do not equal /gallery/ across all available webservers

if( ! string.contains( $path, "gallery" ) ) break;

• Make sure that my IP address can reach the maintenance pool

   if ( ! string.ipmaskmatch( $ip, "my.ip.address/32" ) ) {
        pool.use( "Sticky Pool" );
}     else {
        pool.use( "Maintenance" ); }

This ensures that my IP address reaches a pool containing one specific node – webserver01. All other requests are load-balanced to the ‘Sticky Pool’ which includes two nodes – webserver01 & 02, however 01 is put in a draining node (no further connections being sent to it).

So, both quickly and easily, using ZXTM’s trafficscript we’re able to create rules to efficiently distribute traffic appropriately.

End Result: All requests are still load-balanced across both webservers, with the exception of requests for $path == /gallery/. These requests are sent to one specific node (not under maintenance). This allows the webmaster (me!) to upgrade the code base, and regression test, across webserver 01.

Code Upgrade

Load Balancer rules enabled, we can safely proceed with our code upgrade. This is simply a matter of uploading the new codebase across the existing one. We can then proceed to the Gallery site knowing that we will go to webserver 01; able to both test and perform the application upgrade as applicable. This completed, and tested, we can now restart our rsync daemons, and allow the MySQL slave to start again on webserver 02. Within seconds the code change has been pushed out across the remaining webserver.

End Result: A complete success.