Christopher J. Buckley

Plusnet in cock-up shocka - again! el reg brings word of another spectacular cockup success from plusnet;this time implementing a spam filter appliance.

PlusNet, the terminally scatterbrained ISP, has permanently deleted customers’ email again after it bungled the installation of an anti-spam appliance, it admitted this morning.

The Sheffield firm said it had blackholed about 10 per cent of the backlog of mail which built up yesterday daytime while its engineers worked on the appliance. The same proportion of legitimate mails between 5pm and 11pm was also wrongly earmarked and binned.

An update this morning said: “These mails will not be recoverable and we recommend customers who have not received email that was sent yesterday to ask the sender to re-send where possible.” So, if you you were sent an email that you never received by an unknown person about an unknown subject, give them a bell and tell them to send it again.

The system is now working again, it adds.

We’ve lost count of the number of times we’ve had to write about PlusNet screwing up their email system in the last year, and imagine customers have lost patience. Let us know your questions for the company in our comments section.

PlusNet say they don’t want to answer any of The Register’s own questions, and instead emailed us a pre-prepared statement…

I mean, really! How can they be so totally useless? And why do they still have any customers?

As summer draws nearer to a close, and as the sunlight comes in shorter supply, i’ve noticed how beautiful our airport here in Cambridge looks as I drive past it every evening. So, I did something today that I don’t often do - I took a walk! Yes readers, I walked to the airport to get some pictures of the runway as dusk fell. I ideally needed a tripod, but I don’t have one, so it was best efforts to keep the lens still.

I still don’t think the pictures describe how gorgeous the runway looks with the differing sets of lights at each juncture in the setting sun.

Currently, there are four major spam attacks postmasters are being faced with. They include, but are not limited to:

• PDF & FDF: usually a blank e-mail with a PDF attachment.
• Greeting Card: invitations from an ‘old friend’ to go to a website with a numerical http address, usually containing malware.
• Image Spam: A gif or png attached to a, usually - though not always, blank e-mail ready to sell the latest software or stock.
• Obfuscate Words: Lines of text take this format, N o*t o,n-l-y d o-e’s t,h+i s f i+r*m h+a’v_e grea_t fundamenta,ls*,.

I’m going to show you how using the free-software package, SpamAssassin, you can successfully neuter these 4 major spam attacks.

• Greeting Card: Can be easily defeated using postcards.cf.
• Image Spam: These quickly become extremely popular, but have now decreased in prevalence after very successful methods were implemented to combat them. For SA, use the module Imageinfo: Imageinfo.pm and Imageinfo.cf supplied by SARE Rules Emporium.
• PDF & FDF: Can be successfully discarded by using PDFInfo.pm and PDFInfo.cf again supplied by SARE Rules Emporium.
• Obfuscate Words: These have recently hit, and hit hard. Spammers, seemingly bewildered by their inability to get through current filters using the above popular methods, have now resorted to the old way of securing spam delivery: obfuscation. The good news is that this, again, is easily defeatable. The ruleset(s) originally written by Jennifer Wheeler are mirrored locally by this site.
  1. chickenpox.cf: [words obfuscated by non word characters] Th1s|s a v3ry h4ndy se7 t0 c@tch th!s 50rt 0f (rap.
  2. backhair.cf: [words obfuscated by nonsense html tags] 
     Y<oivugriub>oucou<iuqgheriugv9h>ld rea<y>llyu<owiuer88>se this.

Game on, spammers!

Request Tracker is an enterprise grade ticketing system developed by Best Practical. RT is used by Fortune 100 companies, government agencies, educational institutions, and development organizations worldwide. Many implementations of RT run behind the Postfix mail-server.

The RT wiki has instructions on setting up your MTA to ‘pipe’ the e-mails into RT:

You need to tell your Mail Transfer Agent (ex sendmail, postfix, or qmail) how to forward messages to RT’s mail gateway. To do this, create an aliases in your system’s mail aliases file. Here’s an example, which routes mail to the mailbox [=rt@example.com] (and [=rt-comment@example.com]) into new tickets in the RT queue named General. Note that the queue name is case-insensitive.

Add the following lines to /etc/aliases (or your local equivalent such as /etc/mail/aliases):

rt: “|/opt/rt3/bin/rt-mailgate –queue general –action correspond –url http://localhost/rt”
rt-comment: “|/opt/rt3/bin/rt-mailgate –queue general –action comment –url http://localhost/rt”

These instructions are accurate, but rely on Postfix not having implemented Virtual Domains. Here’s an example configurationfrom main.cf where virtual domains have been implemented:

virtual_transport = virtual
virtual_uid_maps = static:5000
virtual_mailbox_maps = mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf
virtual_alias_maps = mysql:/etc/postfix/mysql_virtual_alias_maps.cf

This configuration will invoke the virtual transport within Postfix. A virtual transport is unable to perform the necessary pipe (”|”) to the rt-mailgate binary. Only the local transport is able to perform a pipe. So, the question is: how do we run virtual domain(s), but still invoke the local transport delivery method to successfully perform the pipe into rt-mailgate?

A Virtual to Local Rewrite Solution

Here’s a quick walkthrough on what steps you need to put in place to ensure that a mail to the (virtual) domain of rt-test@domain.tld is successfully piped as per the RT wiki instructions above.

1. Create an /etc/postfix/aliases file.
2. Within this file add entries that follow this format:

   rt-test                       rt@rt.domain.tld
   support                       support@rt.domain.tld
   abuse                         abuse@rt.domain.tld
   # and so on...
   

3. postmap /etc/postfix/aliases
4. Within /etc/aliases create the pipe aliases referred to in the RT wiki:

   support: "|/opt/rt3/bin/rt-mailgate --queue 'Support' --action \
   correspond --url http://rt.domain.tld/"
   rt: "|/opt/bin/rt-mailgate --queue 'General' --action \
   correspond --url http://rt.domain.tld/"
   abuse: "|/opt/rt3/bin/rt-mailgate --queue 'Abuse'  --action \
   correspond --url http://rt.domain.tld/"
   # Ensure there are no line breaks..
   

5. Run the newaliases command.
6. Insert the appropriate configuration amendments to main.cf

   # To ensure local delivery, rt.domain.tld must be added to
   # $mydestination
   mydestination = localhost localhost.localdomain rt.domain.tld
   # /etc/postfix/aliases is added:
   virtual_alias_maps = hash:/etc/postfix/aliases
   mysql:/etc/postfix/mysql_virtual_alias_maps.cf
   # alias_maps is what is READ by delivery agents etc.
   alias_maps = hash:/etc/aliases
   # alias_databases is what is WRITTEN by newaliases
   alias_database = hash:/etc/aliases
   # masquerade as @rt.example.com unless also on this list,never root
   masquerade_domains = rt.domain.tld
   masquerade_exceptions = root
   

7. Save the file, then reload Postfix’s configuration: /etc/init.d/postfix reload
8. Send an e-mail to support@domain.tld and observe Postfix working its wonders..
   
   Aug 19 21:12:18 solo postfix/local[2972]: 2B91710EB92: to=<support @rt.domain.tld>, orig_to=<support@cjbuckley.net>, relay=local, delay=17, delays=17/0.05/0/0.54, dsn=2.0.0, status=sent (delivered to command: /opt/rt3/bin/rt-mailgate –queue ‘Support’ –action correspond –url http://rt.domain.tld/)

I wrote this article mainly because I see the question oft repeated on the Postfix Users Mailing List. Any comments and (especially) improvements welcome.

DynDNS are a major player in both e-mail and DNS outsource solutions, both for home and corporate customers. The DNS for cjbuckley.net, for instance, is outsourced to DynDNS. Previously, DynDNS refused to publish their IP list for their backup MX services. Existing mail-servers that check (be it at MTA or Content-Filter level) SPF TXT record validity, and employ DynDNS’s backup MX service were currently unable to SPF whitelist their backup MX 100% accurately. But why would you wish to white-list? I think a quick history over-view would be prudent…

Overview

Since SPF become a permanent extension to the SMTP RFC, domain’s wishing to have their e-mail accepted by both the ‘big three’ (Yahoo, Hotmail, GMail) and other well-run mail-servers must be sure to publish their SPF record(s).

SPF is not without it’s critics, however. SPF, currently, breaks more than it fixes. From a personal view-point, I agree entirely with these points:

1. SPF breaks pre-delivery forwarding
2. SPF relies upon DNS for security, but DNS isn’t a security service
3. SPF doesn’t actually address unsolicited bulk mail at all

Point 1 is significant.

The SPF FAQ states that:

Checking SPF On Forwarded Mail

Mail forwarding is set up by the receiver and so for forwarded mail, the border mail server (at which SPF should be checked) the the forwarder’s mail server. If you check SPF on your mail server it is coming from your forwarder and not from a mail server authorized by the sending domain. Technically this is similar to checking SPF against mail relayed from your secondary MX like discussed in the previous item. Authorized forwarders should be whitelisted against SPF checks to avoid this problem.

DynDNS addresses this problem

If you are using a backup MX service, you will need, as per instructions above, to SPF white-list the backup MX (ie, all IP addresses from which it may send e-mail to your primary MX record). To not do so creates the situation that -all- mail going to your backup MX will be subject to a negative score upon receipt to your primary MX.

As talked about at the start of this article, DynDNS previously refused to publish their IP list for backup MX, stating categorically that they would ‘not publish’ this list. I am glad to see they have finally seen sense and allowed postmasters responsible for corporate (and personal) mail-servers the ability to white-list this important MX record.

Enough talk - how do I whitelist within Postfix?

If you’re running Postfix, you will almost certainly have implemented postfix-policyd-spf-perl.

1. Edit /usr/lib/postfix/postfix-policyd-spf-perl
2. Ensure the relevant line(s) read the same as:

use constant relay_addresses => map(
NetAddr::IP->new($_),
qw( 204.13.249.0/24 208.78.69.71/32 208.78.69.72/32 208.78.69.73/32 208.78.69.74/32 ) );
# add addresses to qw ( ) above separated by spaces using CIDR notation.

You’ve now successfully white-listed DynDNS’s backup servers, and will not suffer from SPF’s ability to break pre-delivery forwarding.

Final Points

Before anyone mentions SRS: it’s broken. It doesn’t work, as i write, within Postfix. Plus, it’s a -huge- hack.

I’ve resisted the temptation to mention DKIM, until now. DomainKeys does not suffer from the same problem i’ve written above that SPF does. There are explanations of how DomainKeys work both at Wikipedia and Yahoo! This domain, and all others I am responsible for, both sign and check e-mails for DKIM/DomainKey signatures.

Today at work I encountered an interesting ‘problem’ on our Ubuntu 6.06 LTS mail-server. I deploy MailScanner 4.51.5 across our cluster, which after a good initial configuration sits 24/7/365 unattended. Today a user reported that password protected archives were being removed before Postfix attempted delivery.

I know from knowledge of MailScanner that the default conf file blocks password protected archives by default. However, upon looking in the conf file I did not see the entry to over-ride this. Knowing that this over-ride entry exists I checked the conf file on my home e-mail server from MailScanner 4.60 .

chris@solo> grep 'Password' /etc/MailScanner/MailScanner.conf
Allow Password-Protected Archives = no

Interestingly, this attribute is not present in the Ubuntu package of MailScanner

How To Add Exceptions To Password Protected Archives

1. Within /etc/MailScanner/MailScanner.conf add entry:
   Allow Password-Protected Archives = %rules-dir%/password.protected.rules
2. Create the file /etc/MailScanner/rules/password.protected.rules
3. Within this file add entries for IP’s or e-mail addresses you wish to allow password protected archives from. An example of this file could be:
   
   From: 152.78. yes
   From: 130.246. yes
   From: chris.buckley@domain.tld yes
   FromorTo: allow-zips@domain.tld yes
   FromOrTo: default no
   
4. Save this file, then restart MailScanner.

NB: Post updated, see Julian’s comment below.

Word reaches me that multiple major websites are down as a result of a major power outage in San Francisco, CA. Valleywag initially claimed that a drunken person had gotten in and damaged 40 racks, but an update from Technorati’s Dave Sifry says the problem is a widespread power outage.

Folks,

I just wanted to let you know, it looks like San Francisco is having a MAJOR power event, with outages from the Financial district all the way down to Daly City. One of our colos at 365 Main Street has experienced a power outage (never mind that they always swear up and down that this kind of event can’t possibly happen, oh no, they have multiple redundant systems and they charge us up the wazoo to make sure that we’ll have business continuity, so of course, this isn’t really happening, oh yes) however, our other data centers are all up and running, so we hope to be back up and running as quickly as possible.

I’ll keep you all updated on progress, and I appreciate you bearing with us as we work our way through this…

Dave

Sites affected include Technorati, Netflix (these display nice “We’re Dead” pages), Typepad, LiveJournal, Sun.com, and Craigslist (these just time out).

I, personally, am amazed that none of these significant websites employ global server load balancing. Major load-balancer vendors such as F5, Cisco, NetScaler and Zeus all offer first-class GSLB offerings.

Whilst recently implementing DKIM and SPF for both my personal domain e-mail and for the corporate mail-servers I am responsible for, I found a great verification e-mail box from the DKIM mailing list.

If, after setting up your new extensions to the SMTP protocol, you wish to verify everything is working correctly then I thoroughly recommend a blank e-mail to this address: check-auth@verifier.port25.com

Let’s run through the results of this e-mail for my domain, cjbuckley.net:

 Summary of Results
SPF check:          pass
DomainKeys check:   neutral
DKIM check:         pass
Sender-ID check:    pass

Immediately, we can see that the implementations have been successful - this is a good sign!

SPF check details
Result:         pass
ID(s) verified: smtp.mail=example@cjbuckley.net
DNS record(s):
cjbuckley.net. 300 IN TXT "v=spf1 redirect=_spf.cjbuckley.net"
_spf.cjbuckley.net. 300 IN TXT "v=spf1 ip4:87.127.106.176/29 ip4:84.45.189.64/29 include:ukfsn.org -all"

The detail outputted from this SPF check is impressive; as we can see, my domain initially does a redirect to _spf.cjbuckley.net (note the underscore - these _are_ valid in TXT records). This allows many domains I am the hostmaster for (corporate ones, usually) to be easily managed.

For example, I use redirect: _spf.mycorporatedomain.com for all my corporate domains, this cleans up the SPF records and allows easy management of the record database - all other domains can be redirected to _spf.mycorporatedomain.com. It works very well, and is something I noticed Google taking advantage of, initially.. :-)

Next up - DomainKeys:

DomainKeys check details:
Result:         neutral (message not signed)
ID(s) verified: header.From=example@cjbuckley.net
DNS record(s):

The checker confirms that I do not have DK implemented - indeed, this is correct. DKIM replaces DK, and is now being driven forward to replace Yahoo! DomainKey’s. Note: gmail.com only verifies DK signatures, not DKIM yet - a bit disappointing, though they _do_ sign both with DKIM and DK.

 DKIM check details:
Result:         pass
ID(s) verified: header.From=example@cjbuckley.net
DNS record(s):
beta._domainkey.cjbuckley.net. 300 IN TXT "k=rsa; t=y;p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDiIfv9vqSRUo9L8
ztX/C4gfCD/Ivt8eAGyQbjJ2g4Rq764NPwauj5/sY2AfMrFPqhA0ieXWtmJy2gFS
c4ZlxT8KYaFsJATOpJfYAXUtzmmQ8+RcioyeN3LjzNhm1gUYyJI1Lw0yD2y+k
dN3YxY4NZ0esMXrKbsngTl3pNcNCNxXwIDAQAB"

We can see that our DKIM policy has been verified successfully. As dkim.org states:

DKIM lets an organization take responsibility for a message. The organization taking responsibility is a handler of the message, either as its originator or as an intermediary. Their reputation is the basis for evaluating whether to trust the message for delivery.

Given this statement, e-commerce, financial and banking organisations should have either implemented or be on their way to planning an implementation of DKIM. Personally, I find SPF flawed - DKIM is the wiser choice.

I’m hoping to blog a dkimproxy implementation guide shortly, as the published guide has a few advisories I have issue with and would like clarified.

Comments, welcome!