Troubleshooting Exchange Server Antispam: Disable Outlook Junk Mail

When troubleshooting antispam issues, particularly false positives (legitimate email incorrectly tagged as spam), frequently you run into scenarios where Exchange Server antispam features seem to be working correctly but you still see messages being delivered to the Junk Mail folder instead of the Inbox.

For instance, you whitelist a sender or sender domain, or add the sending IP address to the IP Allow List. You find message(s) from the whitelisted sender, domain or IP address still being delivered to the Junk Mail folder. You open the message and check the antispam headers - as expected, Exchange has stamped the message with a SCL of -1.

When troubleshooting Exchange antispam issues, it's best to turn off Microsoft Outlook's own antispam filtering. This is something Outlook does in addition to Exchange's server-side antispam agents. By default, Outlook's Junk E-mail Filter is turned on and set to Low.

To disable Junk Mail filtering in Outlook 2007:
1. Go to Tools menu | Options | Preferences tab | under E-mail section -> click the Junk Mail button.
2. In Junk E-mail Options, on the Options tab, select No Automatic Filtering...


Figure 1: Disable Outlook's own Junk E-mail Filtering when troubleshooting Exchange server antispam features

Related posts and links:

• About the Junk E-mail Filter
  

Disable Antispam agents on a Receive Connector

Exchange 2007's antispam agents are enabled for all Receive Connectors on a transport server. Is there a way to disable the agents on a particular Receive Connector?

Although not as simple as turning an agent off for each IP address or Receive Connector, Exchange 2007's new transport permissions model allows you to do this just as easily.

The ms-Exch-Bypass-Anti-Spam permission is what allows a sender to bypass antispam agents. By default, mail from authenticated senders is not filtered.

To allow unauthenticated/anonymous senders to bypass antispam filters on a particular Receive Connector, use the following command:

Get-ReceiveConnector "My Receive Connector" | Add-ADPermission -User "NT Authority\Anonymous Logon" -AccessRights ExtendedRight -ExtendedRights ms-exch-bypass-anti-spam

Related Posts and Links:

• Removing internal host names and IP addresses from message headers
  
• Why Get-TransportAgent doesn't agree with the Exchange console
  
• Exchange 2007 Content Filter: How to move messages to Junk Mail folder
  
• Exchange 2007 Content FIlter: The Whitelist Is Here!
  
• Exchange 2007 Transport Permissions Model
  

HOW TO: Prevent annoying spam from your own domain

One of the more annoying types of spam is the one that seems to be coming from your own domain; or worse— from your own email address! Of course, users from your own domain don't generally spam each other— unless you're using one of the free web-based email services. And most of us don't spam ourselves.

Obviously, this is coming from a spammer who has spoofed your email address, or that of someone else from your domain. Unfortunately, SMTP— the protocol that allows mail clients and servers to exchange email, allows headers to be spoofed easily.

In Exchange Server 2007, Accepted Domains tell Exchange which domains to accept email for. If a domain - e12labs.com in this example, exists as an Accepted Domain, there is no reason external senders should use that domain in the MAIL or FROM headers.

You may have remote POP3/IMAP4 users who use SMTP to send mail. However, such sessions should be authenticated, and preferably use a separate Receive Connector.

Thanks to the extensive Transport Permissions model in Exchange 2007, we can easily prevent such spam. Receive Connectors have the ms-exch-smtp-accept-authoritative-domain-sender permission which dictates whether an Accepted Domain can be used in the MAIL or FROM headers. External/internet hosts submit mail to your server without authentication, as anonymous senders. To prevent anonymous senders from sending mail using your domain(s), we need to remove the ms-exch-smtp-accept-authoritative-domain-sender permission assigned to them.

Use the following command to remove the ms-exch-smtp-accept-authoritative-domain-sender permission from NT Authority\Anonymous Logon on internet-facing Receive Connector(s):

Get-ReceiveConnector "My Internet ReceiveConnector" | Get-ADPermission -user "NT AUTHORITY\Anonymous Logon" | where {$_.ExtendedRights -like "ms-exch-smtp-accept-authoritative-domain-sender"} | Remove-ADPermission

Once this permission is removed, when anonymous senders try to submit mail using your Accepted Domain(s), here's how the SMTP conversation goes:

220 E12Postcard.e12labs.com Microsoft ESMTP MAIL Service ready at Wed, 3 Sep 2008 06:22:43 -0700
helo
250 E12Postcard.e12labs.com Hello [172.31.0.170]
mail from:jadams@e12labs.com
550 5.7.1 Client does not have permissions to send as this sender

Exchange stopped spoofing of P1/envelope headers. Let's continue the session and try to spoof the P2 headers (the ones in the DATA part of the message) — maybe that'll work!

mail from:someone@someotherdomain.com
250 2.1.0 Sender OK
rcpt to:jadams@e12labs.com
250 2.1.5 Recipient OK
data
354 Start mail input; end with .
from:jadams@e12labs.com
subject: Header spoofing

This is how we spoof headers, spoof headers.

.
550 5.7.1 Client does not have permissions to send as this sender
quit
221 2.0.0 Service closing transmission channel

As you can see, removing the ms-exch-smtp-accept-authoritative-domain-sender permission stops spoofing of your domains in both envelope (P1) and message (P2) headers.

When not to remove the permission?
Is there a scenario where one should not remove the ms-exch-smtp-accept-authoritative-domain-sender permission from NT Authority\Anonymous Logon? Yes, on Receive Connectors used by internal or trusted SMTP hosts (such as copiers/scanners and application servers) that submit mail without authentication.

But you do have these internal/trusted hosts submitting to a separate Receive Connector, don't you?

Related posts:

• Exchange Server 2007: How To Allow Relaying
  
• Removing internal host names and IP addresses from message headers
  
• Header Firewall: Why spammers can't insert fake SCL (and other Exchange Organization) X headers
  

Quarantine Mailbox: Displaying original SCL, sender and recipients

In Exchange Server 2007, messages delivered to the quarantine mailbox show up as DSNs sent by the postmaster address of the default domain. In HOW TO: Expose original senders and recipients of quarantined messages, we modified the QTNE.cfg form for Microsoft Outlook to reveal original senders and recipients.

Although the original sender and recipient fields were added, the original SCL stamped on the quarantined message wasn't visible. The OriginalScl property was exposed in Exchange 2007 SP1, and is now included in the updated form in that post. Installing the updated form exposes the original SCL for messages in the quarantine mailbox, as seen in Figure 1.


Figure 1: The original SCL for messages in the quarantine mailbox can be displayed using the updated Outlook form

Exchange Server Antispam Updates: Do I have the latest update?

Not sure if the Exchange 2007 or 2003 (IMF) updates on your system are the latest and greatest? Doubt if the automatic update process is working?

You can use the Microsoft Update Catalog web site to search for these. You can also subscribe to the RSS feeds for each update (search result on the web site).

• Exchange Server 2007 Antispam updates  
  
• Exchange 2003 Intelligent Message Filter updates  
  

You can also use the site to search for other Microsoft products and subscribe to the feeds.

Quick antispam report or status check?

Having received an annoyingly higher proportion of spam in my Inbox this morning, I wanted to quickly check what the antispam agents are doing. Here's a quick cmdlet (besides the ones to check whether the antispam agents are enabled, checking the Content Filter SCL thresholds, etc.).

Get-AgentLog -StartDate "6/16/2008" | group action | ft name,count -Autosize

What you get back:

Name Count
---- -----
RejectCommand 520
AcceptMessage 39
RejectMessage 163
QuarantineMessage 11
DeleteMessage 21

The filters are still working. Perhaps it's one of those days when you wake up to high volume of spam.

Note to self: Create a quick monitoring script that provides more information from agent logs, antispam configs, and perfmon counters.

Related posts:
- Keeping tabs on Antispam filters: A few handy scripts in Exchange Server 2007
- Exchange Server 2007: How are RBLs performing?
- Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs

Errata: Edge Subscription and Synchronization white paper

The white paper on Edge Subscription and Synchronization has the following error:

Under Recipient Information:

Distribution groups are not replicated to ADAM.

Distribution Groups are in fact replicated to ADAM using EdgeSync. In Exchange Server 2007 SP1, Distribution Group membership (the member attribute) is also replicated.

On Windows Server 2008, ADAM is replaced by Active Directory Lightweight Directory Services (AD LDS).

Note that new Distribution Groups created in Exchange Server 2007 are set to receive mail only from authenticated senders by default— preventing them from receiving internet mail. Any Exchange recipients set to receive mail only from authenticated senders are not replicated by EdgeSync.

Related Posts:
- Configuring firewalls and name resolution for Edge Transport servers
- New Distribution Groups do not receive internet email by default

Assigning SCL to messages scanned by 3rd-party antispam filters

Here's a problem I had a hard time resolving on Exchange Server 2003. Exchange Server 2007's Transport Rules resolve this within minutes.

Pretend you're taking a Microsoft exam:
Scenario: "You are the Exchange administrator for your organization... ".

Exchange has the Content Filter Agent (CFA), and the Edge Transport Server role designed to be a non-domain-joined mail gateway, located in perimeter networks. However, the Unix/Linux/Security folks in your organization don't trust Exchange to do the filtering (or act as the mail gateway). They insist on using open source anti-spam software, such as SpamAssasin (yes Eric, I remember CRM114 - the Controllable Regex Mutilator... and all the cool stuff you can command it to do. Just to set the record straight, I was suitably impressed back then, and continue to be so till this day.. :) on the non-Exchange SMTP gateways. After tweaking it for a number of weeks, they are able to make it work the way they want it to, or are close to it. It's implementation time!

Their solution is to insert an X-header in messages that looks like this:

X-Spam-Status:yes

That's it. Their job ends there.

As the Exchange team/administrator, your job is to ensure messages with that header end up in users' Junk Mail folder.

Exchange Server 2003 did not provide any way, out-of-the-box, to be able to inspect message headers and stamp SCL. To add some contextual fun - back then, our "solution" was a doc that showed the end-users how to create an Outlook rule to move such messages to Junk Mail. Only if Exchange/Windows admins could code... I can hear you think.

 Creating a Transport Rule: Exchange Server 2007's Transport Rules functionality allows you to accomplish this easily. Here's how:
1. Fire up EMC | Organization Config | Hub Transport | Transport Rules tab
2. Click on New Transport Rule in the Action pane
3. Give the new rule a name, add a comment if you wish
4. In the Conditions page, select the condition when a message header contains specific words
5. In the Step 2 edit box, click on the message header link

6. Type X-Spam-Status | click OK
7. In the edit box, click on the specific words link
8. Type yes | click OK | click Next
9. In the Actions page, select the action set the spam confidence level to value
10. In the rule description, click on the 0 link and add a value that's above your SCLJunkThreshold | click Next
11. On the Exceptions page, click Next if you do not want any exceptions to this rule
12. Click New | click Finish to close the wizard

Or you can use the following commands:

$condition = Get-TransportRulePredicate HeaderContains
$condition.MessageHeader = "X-Spam-Status"
$condition.words = @("yes")
$action = Get-TransportRuleAction SetSCL
$action.SCLValue = 5
new-TransportRule "Stamp SCL" -condition @($condition) -action @($action)

 Disabling the Content Filter agent: Since you have a 3rd-party filtering solution running on your non-Exchange SMTP host(s), you can disable the Content Filter Agent. Messages exceeding SCLJunkThreshold will still be moved to Junk Mail folder.

Disable-TransportAgent "Content Filter Agent"

Alternatively, you can leave the CFA enabled, but disable the Delete, Reject and Quarantine actions.

Set-ContentFilterConfig -SCLDeleteEnabled $false -SCLRejectEnabled $false -SCLQuarantineEnabled $false

Send a test message with the X-header X-Spam-Status:yes. The message has the SCL value set by the Transport Rule. If it is above the SCLJunkThreshold, it should be delivered to the Junk Mail folder.

Why this doesn't work with Delete, Reject or Quarantine thresholds enabled?

The Content Filter Agent fires on the EndOfData SMTP event. On Hub Transport servers, the Transport Rules Agent fires on the OnRoutedMessage event. If the CFA is left enabled, along with any of the above actions, messages may get deleted, rejected or quarantined. The Transport Rules Agent never gets to see deleted and rejected messages at all.

What about quarantined messages?

It's understandable if the message is already deleted or rejected - there's nothing left for the Transport Rule agent to act on! However, why doesn't the agent act on quarantined messages? With Pipeline Tracing turned on, I spent some precious cycles trying to unravel the mystery, with little success. As it turned out (thanks to helpful Exchange team folks for pointing out... ), I was overlooking the obvious - Quarantined messages are wrapped in an NDR "envelope". If you've ever logged into the quarantine mailbox and looked at quarantined messages, you know what these look like. (Related post: "HOW TO: Expose original senders and recipients of quarantined messages") The Transport Rules Agent does not see the original message headers, including our X-Spam-Status header. As a result, the rule never fires.

Life on the Edge

If using an Edge Transport server in the mix, things change. The Edge Rule agent fires on the EndOfData event, unlike Hub Transport servers. When two transport agents fire on the same SMTP event, you can set the priorities of each accordingly so one fires before the other. This is described in "How to Make the SCL Value Available to Edge Transport Rules" in Exchange Server 2007 documentation.

However, it's a lot simpler to just disable the Content Filter Agent. This meets the requirements of this scenario, where anti-spam filtering is done by 3rd-party anti-spam filters and Exchange isn't required to do any filtering at all.

Keeping tabs on Antispam filters: A few handy scripts in Exchange Server 2007

One of the more useful improvements in Exchange Server 2007 is the abundance of logging for different features and components (read previous post "Exchange Server 2007: How many logs hath thee?"). In particular, the antispam agent logs fill an important gap in monitoring, reporting and troubleshooting message flow as it relates to antispam agents (read previous post "Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs"). As a messaging/Exchange administrator, you want to be able to pin point what the antispam agents have been up to, and determine if particular messsages, or sending hosts, domains or email addresses have been blocked by any of the antispam agents. The antispam agent logs and the get-agentlog command allow you to do this quickly and efficiently.

In addition to the the get-agentlog command, Exchange Server 2007 also ships with a number of canned scripts that help you keep tabs on what the agents are doing. These scripts are found in the \Exchange Server\Scripts directory, where \Exchange Server is the path of the Exchange Server 2007 installation. Note, no documentation or support is available for these scripts - they are meant to be examples you can use to write your own scripts. Let's take a peek in the directory and see what we find.

Commandline parameters used by antispam scripts

Most of the following scripts take the same (optional) parameters.
-top n: Where n is the number of results to display. If not specified, the script defaults to (top) 10.
-StartDate: Start date/time
-EndDate: End date/time
-location: path of agent log files. If no path is specified, the agent works against the default agent log file location.

1. Get-AntispamFilteringReport.ps1: Takes one of the following values as a mandatory parameter: 1) connections 2) commands 3) messagesrejected 4) messagesdeleted 5) messagesquarantined, and displays statistics for each agent.
2. Get-AntispamSCLHistogram.ps1: Provides a breakdown of number of messages stamped with each SCL value.
3. Get-AntispamTopBlockedSenderDomains.ps1: Lists top ten sender domains from which mail was blocked.
4. Get-AntispamTopBlockedSenderIPs.ps1: Lists the top ten IP addresses blocked by antispam agents, and number of messages blocked from each.
5. Get-AntispamTopBlockedSenders.ps1: Lists the top ten blocked senders (SMTP email addresses) and number of messages blocked from each. The script can report on P1 (i.e. address in message envelope in the MAIL header) or P2 addresses (from headers in message body like FROM). Specify the option as a commandline argument (optional parameters -top n, -StartDate, and EndDate used in this example):

.\Get-AntispamTopBlockedSenders.ps1 P1 -top 20 -StartDate "12/1/2007" -EndDate "12/10/2007"

Replace the P1 in the above command with P2 to report on P2 senders.
6. Get-AntispamTopRBLProviders.ps1: Lists the top ten RBLs (aka "IP Block List Providers") and messages blocked by each (read previous post: "Exchange Server 2007: How are RBLs performing?").
7. Get-AntispamTopRecipients.ps1: Lists the top ten recipient addresses that receive spam. The addresses may or may not exist in your Organization - the reporting is based on actions taken by antispam agents on incoming messages. Recipient Filtering with Recipient Validation (dropping messages for recipients that do not exist in AD/GAL) is a great way to drop a large number of messages.

The following scripts are not used for reporting:
8. Reset-AntispamUpdates.ps1: Uninstalls any antispam updates and reinstalls the original "out-of-box" antispam data.
9. Install-AntispamAgents.ps1: By default, antispam agents are not installed on the Hub Transport server role (read previous post: "HOW TO: Install anti-spam agents on Hub Transport server"). This script installs them if you need to.
10. Uninstall-AntispamAgents.ps1: Uninstalls antispam agents.

Disabled mailboxes: Can they really receive email?

Some truths you live with for a lifetime, like Outlook users cannot send email using an alternate email address (with Outlook in MAPI mode - read previous post: "HOW TO: Send as alternate email address"). Others change as Microsoft Exchange evolves, either through new versions of Exchange server, or service packs and hotfixes.

Disabled mailboxes cannot receive email. Or rather, could not receive email. This has been true all this while, and hasn't changed in Exchange 2000, Exchange Server 2003, including SP1 and SP2.

The reason is described in Microsoft KBA 319047 - "You receive a non-delivery report when you send a message to a disabled account".

In addition to not being able to receive email, administrators face a few other operational issues when managing disabled mailboxes:

- A common scenario: An employee leaves. You disable the account, as part of a standard operating procedure followed in most organizations. You assign his/her manager/co-worker/replacement permissions to access the mailbox. If the mailbox is disabled, they can't access it!
- A disabled mailbox needs to be be enabled first before it can be moved (KBA 278966: "You cannot move or log on to an Exchange resource mailbox")
- The Application Event Log is flooded with annoying Event ID 9548s, informing you that the disabled account does not have a msExchMasterAccountSID attribute populated - something most Exchange administrators have probably gnashed their teeth at a few times a day.

Workarounds exist to populate the msExchMasterAccountSID attribute with the well-known SELF SID (KBA 322890 "How to associate an external account with an existing Exchange 2000 mailbox"), but not something you want to do on a regular interval after every account, or a bunch of them, are disabled.

The hotfix mentioned in KBA 903158: "A hotfix is available to modify the way that Exchange Server 2003 handles a disabled Active Directory user account that is associated with an Exchange Server 2003 mailbox" changes that behavior. It makes the Store act as if a disabled account with a null/empty msExchMasterAccountSID attribute actually has the SELF SID.

All the above actions (and more) complete successfully for disabled accounts. Yes, disabled accounts receive email if you have this hotfix applied - or any subsequent hotfix that updates Store.exe to version 6.5.7234.3 or later.

But what if I don't want disabled accounts to receive email? To prevent disabled accounts from receiving any email, setup Delivery Restrictions (in ADUC | user -> properties | Exchange General tab) to:


Figure 1: Setting Delivery Restrictions on a recipient to prevent receiving mail

1. Receive mail from authenticated users only: With Recipient Filtering enabled, this will drop internet mail at the gateway or the first Exchange server that receives inbound internet mail
2.Receive mail only from: a particular Distribution Group (use a Distribution Group with no members).

220 Welcome to a California mail server: Fighting spam with SMTP Banners

If you are interested in messaging and fighting spam, you probably watch the legal response to spam with some interest. Given the nature of email and the art of remaining anonymous or otherwise untraceable that spammers seem to have mastered, anti-spam laws were written off as largely ineffective, or even ridiculous. (The FTC begs to differ, in its report to Congress on the CAN-SPAM Act's effectiveness - PDF available here). Legislation and (its) enforcement can never be the sole approach to a problem mostly attributed to technology.

The much-talked about first conviction under the federal CAN-SPAM Act was handed out in California - to Nicholas Tombros in Los Angeles in 2004. Since then, quite a few spammers have had their date with the law, with the accompanying media brouhaha.

California anti-spam laws: California also has its own antispam laws - enacted as amendments to the Business and Professions Code. The rather lame one - Section 17538.4, allows California Attorney General to sue spammers, with a list of IFs that make it more like a "License To Spam". Section 17538.45 is the one of interest, aimed at protecting email service providers from spammers. Email service providers can file civil suits against spammers and claim greater of the following amounts:
- the actual monetary loss suffered by reason of that violation
- liquidated damages of $50 per message initiated or delivered, up to a maximum of $25,000 per day.

Any organization that provides the ability to send or receive email to registered users through equipment located in California (and is an "intermediary" in sending/receiving email) is an email service provider for the purpose of this law.

Interesting, as technology-related legislative action always is.

If your mail servers are located in California (and even if they are not... ), it makes sense to change the SMTP banner (SMTP 220 response) to give notice to a sender that your system should not be used to send unsolicited electronic mail advertisements.

The California Business and Professions Code Section 17538.45 states:

(3) (A) In any action brought pursuant to paragraph (1), the electronic mail service provider shall be required to establish as an element of its cause of action that prior to the alleged violation, the defendant had actual notice of both of the following:

(i) The electronic mail service provider's policy on unsolicited electronic mail advertising.

(ii) The fact that the defendant's unsolicited electronic mail advertisements would use or cause to be used the electronic mail service provider's equipment located in this state.

(B) In this regard, the Legislature finds that with rapid advances in Internet technology, and electronic mail technology in particular, Internet service providers are already experimenting with embedding policy statements directly into the software running on the computers used to provide electronic mail services in a manner that displays the policy statements every time an electronic mail delivery is requested. While the state of the technology does not support this finding at present, the Legislature believes that, in a given case at some future date, a showing that notice was supplied via electronic means between the sending and receiving computers could be held to constitute actual notice to the sender for purposes of this paragraph.

Let's leave the legal interpretation of laws to those who practice law. Regardless of whether your organization intends to take legal action against spammers or not, one of the important principles of information security is having policies in place and communicating them clearly. Not only does it demonstrate your organization's intent, it can be a potential deterrent for folks who may otherwise claim ignorance of such policies.

(While we're on the subject of SMTP banners, for historical context, also take a look at Paul Hobbes & John Levine's draft from 1998 titled "Anti-UBE and Anti-UCE Keywords in SMTP Banners". Much later, a SMTP extension was also proposed to include such information - RFC 3865 - A No Soliciting Simple Mail Transfer Protocol (SMTP) Service Extension.)

Changing the SMTP banner: Assuming the policy is in place, the SMTP banner can be changed easily to communicate it, as hosted email security and compliance services provider Postini (acquired by Google not too long ago) does.

220 Postini ESMTP 11 y6_12_2c0 ready. California Business and Professions Code Section 17538.45 forbids use of this system for unsolicited electronic mail advertisements.

On Exchange Server 2003/2000 or if using Windows Server's SMTP service component on SMTP gateways, the banner text can be changed by setting the IIS Metabase value ConnectResponse, using any Metabase editor or the adsutil.vbs script that ships with IIS (in the Inetpub\AdminScripts directory by default).

cscript adsutil.vbs set smtpsvc/1/ConnectResponse "220 California Business and Professions Code Section 17538.45 forbids use of this system for unsolicited electronic mail advertisements."

Note, the numeric value 1 after smtpsvc/ in the above command refers to the SMTP Virtual Server ID. Each SMTP VS has one, the default/first one starting at 1. To get a list of SMTP Virtual Servers using adsutil.vbs:
adsutil.vbs enum /p smtpsvc

Exchange Server 2007 makes changing the banner a relatively simple task. Note, the banner should start with the SMTP response code 220. Fire up the shell:

Set-ReceiveConnector "Connector Name" -banner "220 (Optional: Server.domain.com). California Business and Professions Code Section 17538.45 forbids use of this system for unsolicited electronic mail advertisements."

That wasn't too difficult, was it? If that simple banner change deters a a single spammer from victimizing your users, and in the process buys you the legal ammunition (that may some day help your organization take legal action, should it decide to.... ), is it worth it?

Connection Filtering, RBLs and SMTP logs

Exchange Server 2003's Connection Filtering feature allows you to block connections from IP addresses explicitly added to the Global Deny List, or drop messages from IP addresses listed on a RBL (Real-Time Blackhole List / Real-Time Block List).

Note: The term "RBL" is commonly used to describe DNS Black Lists (DNSBLs), but it's a trademark of Mail Abuse Prevention System (MAPS), one of the first RBLs. It was acquired by Trend Micro in 2005. Exchange Server 2007 uses the term IP Block List Provider.

When connections from IP addresses listed on the Global Deny List are dropped, you don't find any traces of those dropped connections in SMTP logs [for more details about SMTP logging in Exchange Server 2003, read previous post "Logging SMTP protocol activity"].This is understandable - the IP addresses are explicitly added to the deny list by an administrator, and a SMTP session is never initiated. It's not as important to log these (though many would prefer to capture information about connections so blocked). If in doubt about which hosts are getting blocked, all one needs to do is look at this list.

When it comes to DNSBLs, the behavior is different. Connections are not dropped. Messages are dropped on RCPT TO: command. This allows you to receive mail for addresses on the Exception List. RFC 2821 requires senders to be able to contact the postmaster of a domain. You can also add some of the generic addresses from your domain to the Exception List if you don't want email to these addresses blocked.

Here's a demo of the SMTP session. As seen in the session, the sending host gets a 550 5.7.1 error code because the IP address is listed on a DNSBL, and the recipient (in the RCPT TO: command) is not listed on the Exception List.

Here's what the SMTP log shows (some irrelevant fields removed):

2007-07-25 19:04:54 172.31.1.10 0 HELO - - 250 0 49 4 0 SMTP - - - -
2007-07-25 19:04:59 172.31.1.10 0 MAIL - +from:foo@mydomain.com 250 0 41 26 50 SMTP - - - -
2007-07-25 19:05:11 172.31.1.10 0 RCPT - +to:jadams@exchangelabs.net 550 0 0 31 5017 SMTP - - - -
2007-07-25 19:05:15 172.31.1.10 0 QUIT - - 240 22943 72 4 0 SMTP - - - -

SMTP logs the response code 550, but not the enhanced status code 5.7.1. Having said that, this enhanced status code doesn't by itself allow you to narrow it down to DNSBLs.

Exchange Server 2007's Connection Filtering agent behaves differently - connections from IP addresses explicitly listed in the IP Block List (equivalent of the Global Deny List in Exchange Server 2003) are not blocked. This change in behavior allows such connections to be logged. Detailed information is available from the agent logs [read previous post "Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs"] about why a message was accepted/rejected by anti-spam agents.

One example of the kind of reporting/troubleshooting capabilities this provides: you can search the agent logs for messages blocked by Connection Filtering agent based on DNSBL lookups:

Get-AgentLog -StartDate "7/27/2007" | where {$_.Reason -eq "BlockListProvider"}

The Get-AgentLog output:

Timestamp : 7/27/2007 6:42:08 AM
SessionId : 08C99DA727C8EF31
IPAddress : 68.45.233.59
MessageId :
P1FromAddress : dbuuxwxjlm@bobach.com
P2FromAddresses : {}
Recipients : {bf5b7b59@mydomain.com}
Agent : Connection Filtering Agent
Event : OnRcptCommand
Action : RejectCommand
SmtpResponse : 550 5.7.1 Recipient not authorized, your IP has been found on
a block list
Reason : BlockListProvider
ReasonData : Spamhaus SBL-XBL
Diagnostics :

SCRIPT: Turning on Filter Junk Email

OWA users who never logon using Outlook do not have their Junk Mail filtering option turned on by default. Exchange MVP Glen Scales has a script here that allows you to turn this on programmatically for Exchange Server 2007 users.

Why Get-TransportAgent doesn't agree with the Exchange console

You disable a particular anti-spam agent - let's say the Content Filtering Agent - from the Exchange console.



Next, you use the Get-TransportAgent command to get the status of transport agents - and surprisingly the Content Filter Agent shows up as Enabled!



Why doesn't Get-TransportAgent agree with the Exchange console about the status of an agent?

Fellow MCT and Exchange geek David Elfassy blogged about it recently.

The Anti-Spam tab is located under the Organization Configuration | Hub Transport node in the console. Disabling an agent from the console prevents the agent on all transport servers in the Organization from taking any action. It's an Organization-wide or global setting.

The Get-TransportAgent command is transport server-specific. It shows the status of agents on a particular server. With the Org-wide configuration set to disabled, an agent enabled on a particular transport server behaves like a soldier instructed to fire but not kill. (May not be the best analogy - it doesn't exactly scare spam, and it doesn't do much else).

To draw a parallel to Exchange Server 2003, you can enable settings on a particular filter - e.g. Recipient Filtering, in Global Settings. However, you still have the flexibility of being able to enable/disable it on a per-SMTP virtual server basis. This is somewhat similar.

To get the global/Org-wide status of a particular filter, you need to use the corresponding command for that filter.
- For Content Filter Agent: Get-ContentFilterConfig
- For Recipient Filter Agent: Get-RecipientFilterConfig
- For Sender Filter Agent: Get-SenderFilterConfig
- For SenderID Agent: Get-SenderIDConfig
- For Protocol Analysis Agent: Get-SenderReputationConfig

If you've carefully looked at the above list, you're probably wondering why there's no mention of Connection Filtering Agent in it. The Connection Filtering Agent handles multiple tasks which are listed individually in the console.

To check the global setting for configuration of each, use the corresponding config command:
- For IP Block List: Get-IPBlockListConfig
- For IP Block List Providers: Get-IPBlockListProvidersConfig
- For IP Allow List: Get-IPAllowListConfig
- For IP Allow List Providers: Get-IPAllowListProvidersConfig

To "disable" an agent in Organization Configuration (equivalent of what you do in the console) using the shell, use the corresponding Set command, e.g.:

Set-ContentFilterConfig -Enabled $false

Next, let's take a look at how you can disable an agent on a transport server. Generally, if the Get-Blah command returns a property like Enabled: True, you can use Set-Blah -Enabled $false to disable it (You probably know where we're going with this by now... :). Not so for Get-TransportAgent. Its "Set" counter-part - Set-TransportAgent allows you to modify only one property -Priority. You can change the order in which these agents fire by changing their Priority.

You use the Disable-TransportAgent command to disable an agent on a transport server:

Disable-TransportAgent "Connection Filtering Agent"

No points for guessing the command used to enable a transport agent - Enable-TransportAgent

Header Firewall: Why spammers can't insert fake SCL (and other Exchange Organization) X headers

After almost a year and a half of working with Exchange Server 2007, I still stumble into features I haven't tested before. When it happens, it blows holes into my theories about how well I know E12. :)

It started when investigating why a certain message - likely not from an authenticated source, would have a SCL of -1 (seen in message headers as X-MS-Exchange-Organization-SCL: -1).

One suggestion that got me thinking: it may have been a fake header, perhaps inserted by a spammer. I tested this a few times in different combinations (by inserting SCL and PCL headers before, after and in between other X-headers, like the ones below:


Note: Exchange uses the PCL and SCL headers seen above, but the QCL header in the above screenshot is fabricated/fake.

When the above messages came in, all the headers with X-MS-Exchange-Organization* were removed by Exchange. Nice work, Microsoft, I said to myself, and asked for an explanation.



Someone from the team immediately pointed me to Header Firewall - a supercool feature (not so much in what it does but the fact that it exists). It removes protected Exchange Organization and Forest headers from messages that enter or exit the Exchange Organization. This is the reason you're not likely to see fake SCL, PCL, SenderID, and other such headers in messages received by Exchange Server 2007 Edge or Hub Transport servers.

Preserving Organization and Forest X headers: If you want these to be preserved for inbound mail from particular sources, it is recommended to do so using a separate Receive Connector, just as you would create one if you needed to allow unauthenticated relaying [read previous post "Exchange Server 2007: How To Allow Relaying"], and assign MS-Exch-Accept-Headers-Organization and MS-Exch-Accept-Headers-Forest permissions. Similarly, to send Organization and Forest headers to particular external mail systems, it is recommended to create a separate Send Connector for those address spaces, and assign the MS-Exch-Send-Headers-Organization and MS-Exch-Send-Headers-Forest permissions.

Getting back to the original question - when do you see a SCL of -1? Either when the sender is authenticated, or the sending host is on the IP Allow List.

Read more about the Header Firewall feature in the Exchange Server 2007 documentation. A list of Exchange Organization and Forest x-headers can also be found there.

Exchange Server 2007 Enterprise CALs: Let Microsoft do the filtering

Microsoft introduced the concept of Standard and Enterprise Client Access Licenses (CALs) for Exchange Server 2007. In previous versions, only the server part of Exchange came in Standard or Enterprise flavors, the latter allowing you the use of enterprise features like more number of Storage Groups and Stores, no limits on Store sizes, clustering, etc.

With a lot more features and functionality in Exchange Server 2007, the Client Access Licenses come in both flavors as well.

If you're planning to buy Enterprise CALs for your Exchange Server 2007 deployment, these come with an interesting benefit. Yes, Microsoft's ForeFront for Exchange is part of the deal, and though it's a good product (its predecessor - Sybari Antigen for Exchange, was for long considered the best of breed anti-virus products for Exchange), that's not the interesting benefit.

That distinction goes to Microsoft Exchange Hosted Filtering, which is part of the Enterprise CAL offering. Yes, this is Microsoft's software product + services play - you can outsource your anti-spam and anti-virus filtering headaches to Microsoft. You may not have to deploy any Edge Transport servers (depending on your organization's security policies, of course.. ).

If you're wondering what the new Standard and Enterprise CALs offer, this page has it all. (Well, almost... - the nuances of Exchange Server 2007 licensing are lost to many, so always get a quote and details from your software vendor or Microsoft rep.).

Things to note:
- the Enterprise CAL is an add-on to the Standard CAL.
- Depending on Exchange Server features used, not all users need an Enterprise CAL.
- Standard Journaling, which offers the same granularity as Exchange Server 2003 - that of journaling at the Store level, does not require an Enterprise CAL
- Premium Journaling: Per user or per distribution list journaling - you need an Enterprise CAL for only the users that'll be subject to/use premium journaling.
- Simply because you require or have Entperprise CALs for users doesn't necessarily mean you require Enterprise Edition licenses for Exchange Server. The Enterprise Edition is required only if you want to scale up beyond 5 Storage Groups/Mailbox Stores, or if you want to use clustering using Exchange Server 2007's Single Copy Cluster (SCC) or Cluster Continuous Replication (CCR).
- As Paul Robichaux notes in WinITPro's "Exchange and Outlook UPDATE" newsletter, there has been a licensing change in CAL requirements: Managed Default Folders like Inbox, Deleted Items, Junk Mail, etc. no longer require an Enterprise CAL, restoring the equivalent of Exchange Server 2003/2000 Mailbox Manager policies (i.e. Recipient Policies with Mailbox Manager settings) without an additional licensing cost. Managed Custom Folders feature, which allows you to create and manage custom folders in user mailboxes, requires an Enterprise CAL.

Note: Mailbox and Public Folder Stores in Exchange Server 2007 Std. Edition do not have size restrictions like the 16 Gb. limit in previous versions. The limit was raised to a max of 75 Gb. in Exchange Server 2003 SP2, and removed completely from Stores in the Std Edition of Exchange Server 2007.

Moving items to Junk Mail folder doesn't do much

Scenario: Junk mail makes it to your Inbox, without getting filtered by any of Exchange's anti-spam filters.
Action: User moves item to Junk Mail folder.
Result: This accomplishes little more than having the piece of UCE moved out of the Inbox. Contrary to what many may think and like to believe, moving a message manually to the Junk Mail folder doesn't really have Outlook or Exchange do anything at all with that item. No, the sender or sending domain or host do not get blacklisted or added to Microsoft Outlook's Blocked Senders List. Yes, you're not likely to see a similar message from same sender, domain or host get treated any differently based on that action.

To make Microsoft Outlook filter messages from the same sender, the sender needs to be added to the Blocked Senders list manually (by right-clicking message | Junk E-mail | Add Sender to Blocked Senders List).



Does the Blocked Senders List make it to the Exchange Hub or Edge server using the Safelist Aggregation feature? No. Safelist Aggregation is only used for aggregating the safe list, as the name of this feature correctly suggests. The Blocked Senders List is only used by Microsoft Outlook, without any server-side action being taken on messages from senders on this list.

Related Posts:
- Exchange 2007 Content Filter: How to move messages to users' Junk Mail folder
- TechEd 2007: Notes from the BOF Session on Fighting Spam With Exchange Server 2007
- Exchange 2007 Content Filter: The Whitelist Is Here!
- HOW TO: Expose original senders and recipients of quarantined messages
- Connection Filtering, RBLs and SMTP logs

TechEd 2007: Notes from the BOF Session on Fighting Spam With Exchange Server 2007

Tuesday night was a Birds of a Feather (BOF) night, with plenty of BOF sessions and chalk talks scheduled. My session on Fighting Spam With Exchange Server 2007 (not a very creative title, I know... but it got the message across :) was scheduled for 7:45 PM. I was positive I'll probably be the only person in the room - 7:45 PM isn't exactly the time I'd want to attend a TechEd session on anything, with the exception of a few good speakers.

On a side note, keeping my expectations low also meant I could claim the attendance was more than I expected even if only one person showed up. Nevertheless, the room filled up as we got closer to the start time.

It was an interesting discussion on anti-spam in Exchange Server 2007, as well as Exchange 2003.

Some takeaways:

• Exchange's anti-spam features consist of more than just Intelligent Message Filter (IMF) - there's 1) Connection Filter: allows using IP Allow Lists, IP Block Lists, IP Block List Providers (RBLs) and IP Allow List Providers 2) Sender Filter: block sending domain or SMTP address 3) Recipient Filter: block internet messages to list of valid recipients, perform Recipient Validation to block messages for non-existent recipients 4) Sender ID Filter: looks up SPF records in purported sending domain's DNS zone, protects from address spoofing and phishing 5) Content Filter (aka "IMF"): Blocks messages based on content
• 3 Gateway Thresholds and Actions: Exchange Server 2007's Content Filter Agent supports 3 gateway thresholds and actions - 1) Delete 2) Reject and 3) Quarantine
• Quarantine functionality: Though the quarantine functionality in Exchange Server 2007 is a step forward compared to Exchange Server 2003's archiving feature, it's not as full-featured as that offered by many third-party products. End-users have no control over the quarantine - an administrator has to inspect quarantined messages and make the decision to deliver or delete. However, it's important to understand that a) it's not compulsory to use the quarantine and b) The end-user mechanism for accessing suspicious messages is the Junk Mail folder c) Quarantining is one more layer available to admins - between the gateway actions of Rejecting or Deleting mail and the Store action of delivering to Junk Mail
• Exchange Server 2007's Connection Filter agent does not immediately drop connections from IP addresses explicitly listed on the IP Block List. Exchange Server 2003 does this for IPs listed in its Global Deny List. This change in behaviour allows such "filtered" connections to be logged in the agent log. Some customers like to see these logged, whereas many want connections to be dropped immediately.
• Many customers prefer anti-spam appliances like Barracuda and IronPort in the perimeter, and consider these very effective, and generally lower cost compared to an Exchange 2007 Edge Transport server.
• No studies or test results are available yet to demonstrate how the Edge Transport server performs compared to such appliances. It would be great to have a feature and performance comparison of these.
• Many customers are interested in Greylisting - a technique that registers each connection's "triplet" of a) Sending host's IP address b) Sender's address c) Recipient's address and looks up a database. If not found, the connection is immediately dropped and the triplet added to the database. The technique is based on the idea that valid RFC-compliant SMTP hosts will attempt to resend the message after a set interval. The (receiving) SMTP server will find the triplet in the database upon the second attempt, and consequently accept the connection.
• Interested in learning more about Greylisting? Evan Harris has a whitepaper titled "The Next Step in Spam Control: Greylisting"
• Exchange Server 2007 SDK has the code for a Greylisting agent. This needs to be compiled into a DLL using Visual Studio 2005, and installed using the accompanying installation script. Since this is just a programming sample provided in the SDK, it makes sense to test this adequately in a test environment and not expect any production support
• Freeware Greylisting transport sinks/add-ins for Exchange Server 2003 are available on the web, like this one from Chris on Grynx.com
• Over a period of time, the effectiveness of techniques like Greylisting may decrease, but as of now it still eliminates 80-90% of spam.
• SMTP Tarpitting is a technique that inserts a delay in SMTP responses to SMTP clients that display suspicious protocol behavior. More details about tarpitting in a previous post titled "Enabled by default: SMTP Tarpit in Exchange Server 2007"
• Safelist Aggregation: This allows Exchange to aggregate senders marked as safe by (Outlook/OWA) users. If an EdgeSync subscription exists (in topologies with an Edge server), the safe list is replicated to the Edge as part of EdgeSync. Exchange's use of safe lists can reduce false positives significantly. It's not required to have an Edge server - Hub Transport servers with anti-spam agents installed can also use this feature.
  

TechEd 2007: Let's talk about fighting spam with Exchange Server 2007

If you're going to be at Microsoft TechEd 2007 in Orlando next week, and are interested in sharing your experiences in fighting spam using Exchange Server's anti-spam features, here's a Birds-of-a-feather (BOF) session just for you.

What: BOF session "Fighting Spam With Exchange Server 2007"
When: Tuesday, June 5th @ 7:45 PM
Where: Microsoft TechEd, Orlando, FL
What's a BOF session? : Birds-of-a-feather sessions are not presentations or lectures. These are meant to be interactive discussions, sans the PowerPoint slides and demos you typically see in breakout sessions.
Why so late?: Tuesday night is an all-BOF session night, with no other conference activities. Microsoft will be serving food and drinks.

Abstract:
“Exchange Server 2007 anti-spam features are a significant improvement over previous versions, and come with interesting deployment and topology questions. Are these effective in the fight against spam? This BOF session will be a round table for Exchange administrators to talk about their experiences with Exchange Server 2007 anti-spam features, including issues such as Edge servers, use of IP Block List Providers, and new features and techniques such as Sender Reputation, and Safelist Aggregation.”
Audience: IT Professional

Grimes Gripes About RBLs

InfoWorld columnist Roger Grimes gripes about Real-Time Block Lists (aka Real-Time Blackhole Lists, DNSBLs or DNS-based Block Lists. The Exchange Server 2007 term for RBLs is "IP Block List Providers").

The feelings towards RBLs are understandable - you won't be a fan of RBLs if your IP addresses are the ones getting listed, and your users breathing down your neck about bounced messages. It's important to monitor RBLs to determine if your IP addresses are listed in any. (Incidentally, Zenprise for Exchange does this automatically several times a day - configurable by the administrator - and alerts you if any of your IP addresses are listed on RBLs. Your external IP addresses are detected automatically based on messages sent to Zenprise's Zmail hosted service).

Once listed, some RBL providers make the process of getting IP addresses delisted a long and painful one. At times, and in spite of your best efforts and assurances by the provider that the IP address(es) will be delisted, they continue to live on in such lists. On the other end of the spectrum are RBL providers that make the delisting process as simple as entering the listed IP address in a form on their web site, without any checks in place. I would shy away from using the latter.

Regardless, if you are on the other side, as a victim of spam rather than a victim of RBLs, good/reputed RBL providers like Spamhaus (which has its detractors as well - read previous post: "Spamhaus, the E360Insight lawsuit and the future of RBLs"), can help in dramatically reducing spam at the mail gateway - by as much as 90+ percent. This further reduces the stress on mechanisms like content filters (like Intelligent Message Filter or IMF in Exchange Server 2003, and the Content Filter agent in Exchange Server 2007).

Whether you use RBLs to block inbound internet mail from black-listed addresses or not, it makes sense to monitor whether your IP addresses are listed or not on a regular basis.

Read Grimes' column titled "Why I hate RBLs" on InfoWorld.com.

HOW TO: Expose original senders and recipients of quarantined messages

The Content Filter Agent in Exchange Server 2007 allows you to quarantine messages above the SCLQuarantineThreshold. Messages so quarantined are delivered to a quarantine mailbox, specified in the Content Filter configuration. [Read previous post for more info, "Exchange 2007 Content Filter: How to move messages to users' Junk Mail folder"]

Messages arriving in the quarantine mailbox look like NDRs. They appear to be sent from the postmaster address of the default domain. As such, you can't easily see the original sender and recipient's addresses, as shown in the screenshot below:


Figure 1: By default, messages in the quarantine mailbox show the envelope sender— the postmaster address

"How to Recover Quarantined Messages from the Spam Quarantine Mailbox" in the Exchange Server 2007 documentation shows how to use a custom Outlook form to expose the original sender - the doc has code that can be copied to a text editor like Notepad and saved as a custom form (QTNE.CFG), which can be used in Outlook to expose the relevant MAPI property as a field that can be added to the view.

This exposes the original sender, but doesn't expose the original recipient of the message. To expose the recipient, you need to modify the form from the above link and add the relevant MAPI properties.

 Download the modified QTNE.CFG file
Download the modified form file and save it in the correct Forms folder for your locale.

For US English, the path is: \Program Files\Microsoft Office\Office12\Forms\1033. You can find out the locale ID (LCID) for your locale from KB 221435: List of supported locale identifiers in Word..

What was added in my Modified Quarantine Extension Form:
Under [Properties]

Property02=OriginalRecipient

After [Property.OriginalSender] section:

[Property.OriginalRecipient]
Type=30
NmidInteger=0x0E04
DisplayName=Original Recipient

 Install the form in Microsoft Outlook

1. Tools | Options | Other tab | Advanced Options | Custom Forms | Manage Forms | Install.
2. Navigate to the path where you saved the QTNE.CFG file, and select the file.


Figure 2: Installing the Modified Quarantine Extension form (QTNE.cfg) in Microsoft Outlook

3. Click OK a few times till all the dialog boxes are closed. The form is now installed.


 Add the fields
With the form installed in Outlook, it's time to make the fields show up!

1. In the Outlook messages pane, right-click the column heading on the top | select Custom.
2. In the Custom View dialog box, click Fields | from the "Select available fields from:" drop-down (which says "Frequently-used fields" by default), select Forms [screenshot].
3. Select the Modified Quarantine Extension Form | click Add
4. In the Show Fields dialog box, select Original Sender and Original Recipient | click Add.


Figure 3: Adding the Original Sender and Original Recipient fields to the view

5. Additionally, you can remove unnecessary fields and order the fields the way you want [screenshot of fields I have selected].

Once you're done, the Original Sender and Original Recipient fields are displayed as shown in the screenshot below:


Figure 4: The updated view after adding the Original Sender and Original Recipient columns

When the message list pane is expanded in Outlook, this is what it looks like.


Figure 5: Expanding the columns in the updated view

Exposing these fields makes managing the quarantine mailbox easier.

Update: Adding the Original SCL column
08/19/2008 7:39 PM

Exchange 2007 SP1 also exposes the original SCL. The Outlook form file (QTNE.cfg) can be updated by adding the following.

- In the [Properties] section:

Property03=OriginalSCL

- After the [Property.OriginalRecipient] section:

[Property.OriginalSCL]
Type=3
NmidPropset={41F28F13-83F4-4114-A584-EEDB5A6B0BFF}
NmidString=OriginalScl
DisplayName=Original SCL

You can download the updated QTNE.cfg form or manually updated your own file. Now you can add the Original SCL column.


Figure 6: The original SCL for messages in the quarantine mailbox can be displayed using the updated Outlook form.

The previous version of QTNE.cfg is still available.

Exchange Server 2007: Protect users from spam from your own domain

One of the common complaints from users and many messaging folks is spam received from senders that appear to be from your own domain. SMTP mail is exchanged with anonymous internet hosts without any authentication. Headers are effortlessly spoofed. Rather than use an unregistered or invalid domain in the From: field, many spammers find it attractive to use an address from your domain - even an invalid one, or worse - a perfectly valid one. (It is advisable to not have resolution of anonymous senders enabled. Read previous post "A Late New Year's Resolution: Do Not Resolve Anonymous Senders")

Messages that appear to be sent from your domain probably have a better shot at making it through some spam filters, and certainly have a higher chance of being opened by recipients in your domain, once they do make it to the Inbox.

One way of tackling this menace is by using SenderID filter to drop messages that return a FAIL result (from a SPF record lookup). It's advisable to publish SPF records for your domain(s), if you haven't already done so. Explicitly designate SMTP hosts authorized to send for your domain(s). However, the impact of SenderID and whether you should consider dropping email based on it is an open debate, and the topic of another post.

In addition to the above, Exchange Server 2007 allows you to configure ReceiveConnectors to drop messages where the FROM address is from your own domain. It's a good idea to do so if you don't expect any legitimate messages sent without authentication (with your domain in the From: field). To configure a ReceiveConnector to reject such messages:

Remove-ADPermsission "ReceiveConnector-Name" –user “NT AUTHORITY\Anonymous Logon" –extendedrights “ms-Exch-SMTP-Accept-Authoritative-Domain-Sender”

Consider doing this only on the ReceiveConnector that receives inbound internet mail. There may be a need for certain - generally internal - senders like copiers/scanners, and other non-Exchange mail hosts to be able to send with an address from your domain in the FROM field, without authentication. As discussed in previous post titled "Exchange Server 2007: How To Allow Relaying", it is advisable to create an additional ReceiveConnector for such trusted/internal hosts.

RFC 2821, HELO again: Validating the HELO/EHLO domain

RFCs 2821 and 1869 specify the format of HELO/EHLO commands issued by a SMTP client to initiate a session.

RFC 2821 on HELO/EHLO command:

4.1.1.1 Extended HELLO (EHLO) or HELLO (HELO)
These commands are used to identify the SMTP client to the SMTP server. The argument field contains the fully-qualified domain name of the SMTP client if one is available. In situations in which the SMTP client system does not have a meaningful domain name (e.g., when its address is dynamically allocated and no reverse mapping record is available), the client SHOULD send an address literal (see section 4.1.3), optionally followed by information that will help to identify the client system. y The SMTP server identifies itself to the SMTP client in the connection greeting reply and in the response to this command.

(paragraph snipped here)

In any event, a client MUST issue HELO or EHLO before starting a mail transaction.

These commands, and a "250 OK" reply to one of them, confirm that both the SMTP client and the SMTP server are in the initial state, that is, there is no transaction in progress and all state tables and buffers are cleared.

Syntax:

ehlo = "EHLO" SP Domain CRLF
helo = "HELO" SP Domain CRLF

The syntax, as the last 2 lines show - depending on whether you're using EHLO or HELO, should be:
HELO foo.somedomain.com
or
EHLO foo.somedomain.com
(followed by a CR LF)

The IIS/Exchange SMTP stack validates this syntax by default. SMTP clients that do not adhere to this syntax - by appending a period or a space at the end of the domain or address literal (IP address) get a 501 5.5.4 Invalid Address error. Exchange Server 2007, which has a brand new SMTP stack of its own, also continues this domain validation, with a slight modification in the verbiage: 501.5.5.4 Invalid domain name.

However, some SMTP clients do format the EHLO/HELO commands with a trailing period/dot or a space. Though not RFC-compliant, many mail systems allow it, including webmail services like Google's Gmail and Microsoft's Hotmail/Live Mail (I wasn't able to test this with Yahoo Mail today...- Bharat).

You can stop IIS/Exchange SMTP servers from checking if the domain in EHLO/HELO is well formatted by changing the value of SmtpDomainValidationFlags metabase property in IIS to a non-zero value. The property does not exist by default, but you can create it using a tool like the Metabase Explorer (in IIS 6 Resource Kit Tools) - it's a DWORD, with an ID of 36992. You can get step-by-step instructions on how to do this in KBA 291828: "501 5.5.4 Invalid Address" error message from a sending UNIX server.

A post in Microsoft's Exchange newsgroups enquired if this was going to open up a security hole in SMTP. Other than allowing domains or address literals not formatted correctly according to SMTP standards, which many SMTP clients and webmail systems allow and probably use - as stated above, I can't imagine this change creating any serious security issues. Having said that, the particular post was in regard to not being able to receive email from Google Alerts. I continue to receive email from Google Alerts without making any changes to the default behavior of SMTP in IIS/Exchange.

However, the IIS/Exchange SMTP stack does domain validation for a reason. One possibility to consider when evaluating whether to turn it off - it's likely that it could allow sessions from spammers that indulge in such behavior.

Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs

The anti-spam agents in Exchange Server 2007 log their actions in agent logs. By default, agent logs reside in \Exchange Server\TransportRoles\Logs\AgentLog. Each agent log is 10 Mb in size. The size of the directory is capped at 250 Mb, the age of a log to 30 days. (Logs are flushed when they "age" up to 30 days or if the directory size reaches 250 Mb - whichever happens first).

Additional agent log configuration parameters in SP1

Exchange Server 2007 RTM: In the RTM version, there's only one configuration option for the agent log - that to enable or disable it.

Exchange Server 2007 SP1: Additional parameters to control the max directory size, file size and age of agent logs added.

The config options available:
1) Enable/Disable agent log: Boolean value - TRUE/FALSE
2) Max directory size: In bytes. If not specified, the default is 250 Mb or 262144000 bytes.
3) Max file size: If not specified, the default is 10 Mb or 10485760 bytes.
4) Max age: If not specified, the default is 30 days.

The agent log configuration parameters can be controlled by editing the EdgeTransport.exe.config file, located in \Exchange Server\Bin folder on Edge and Hub Transport servers.

To disable agent logging, insert the following key under <appsettings></appsettings> in the config file:

<add key="AgentLogEnabled" value="FALSE" />

In the following example, we modify max directory size to 500 Mb, file size to 20 Mb, and age to 60 days, by creating new keys in EdgeTransport.exe.config:

<add key="AgentLogMaxDirectorySize" value="524288000" />
<add key="AgentLogMaxFileSize" value="20971520" />
<add key="AgentLogMaxAge" value="60.00:00:00" />

Parsing the agent log: You can parse the agent log using the Get-AgentLog command from the shell. By default, this parses the agent log residing in the default location. If you've copied a bunch of agent logs at an alternate location, you can specify the alternate location using the following syntax - in this example the agent logs have been copied to Z:\Antispam Agent Logs directory:

Get-AgentLog -location "Z:\AntiSpam Agent Logs"

Here's what an entry in the agent log looks like - note the different fields and their values:

Timestamp : 4/16/2007 12:39:49 AM
SessionId : 08C948C83FB951AC
IPAddress : 72.46.133.113
MessageId :
P1FromAddress : ret@noncornelan.com
P2FromAddresses : {}
Recipients : {foo@yourdomain.com}
Agent : Connection Filtering Agent
Event : OnRcptCommand
Action : RejectCommand
SmtpResponse : 550 5.7.1 Recipient not authorized, your IP has been found on a block list
Reason : BlockListProvider
ReasonData : Spamhaus SBL-XBL
Diagnostics :

As seen in the above output, the logs provide adequate information for reporting on anti-spam activity, as well as for troubleshooting anti-spam issues like messages not being received/wrongly filtered out.

Getting to know the agent logs will make troubleshooting such issues much easier.

By default, the Get-AgentLog command returns all the entries in the agent logs. It can be constrained to a particular date and time - the recommended way to perform most agent log searches, unless you want to immerse yourself in 30 days (or 250 Mb) of anti-spam goodness! This is done using the -StartDate and -EndDate parameters:

Get-AgentLog -StartDate "4/16/2007" -EndDate "4/17/2007"

You can also constrain it further by adding time of the day:

Get-AgentLog -StartDate "4/17/2007 8:00 AM" -EndDate "4/17/2007 2:00 PM"

Though the Get-AgentLog command only takes these 3 parameters - location, StartDate, and EndDate, you can further filter the logs using most of its logged fields.

To filter the log to show messages to a particular recipient:

Get-AgentLog -StartDate "4/16/2007" -EndDate "4/17/2007" | where {$_.recipients -like "foo@yourdomain.com"}

To search for messages from a particular sender:

Get-AgentLog -StartDate "4/16/2007" -EndDate "4/17/2007" | where {$_.P1FromAddress -like "aqe@easymoney2u.com" -or $_.P2FromAddresses -like "aqe@easymoney2u.com"}

To search for messages from a particular domain:

Get-AgentLog -StartDate "4/16/2007" -EndDate "4/17/2007" | where {$_.P1FromAddress -like "*somedomain.com" -or $_.P2FromAddress -like "*somedomain.com"}

To filter by the anti-spam agent that acted on a message, e.g. Connection Filtering Agent:

Get-AgentLog -StartDate "4/15/2007" -EndDate "4/17/2007" | where {$_.Agent -eq "Connection Filtering Agent"}

Similarly, you can filter by other agents that write to the agent logs: 1) Content Filter Agent 2) SenderID agent 3) Sender Filter agent 4) Recipient Filter agent and 5) Edge Rules agent.

To filter agent logs by the sending host's IP address, use the following command:

Get-AgentLog -StartDate "4/15/2007" -EndDate "4/17/2007" | where {$_.IPAddress -eq "72.46.133.113"}

The reason field in each log entry specifies the reason supplied by the anti-spam agent that takes the action. For instance, as seen in the agent log entry shown earlier in this article, the agent that acted on the message is the Connection Filtering Agent, the reason is BlockListProvider (i.e. "RBL" or "Real-time Block List", known as IP Block Lists in Exchange Server 2007). The ReasonData field gives you the name of the IP Block List Provider, as configured in Exchange. In the above agent log entry, it is "Spamhaus SBL-XBL". To constrain the search for messages blocked by IP Block List Providers:

Get-AgentLog -StartDate "4/15/2007" -EndDate "4/17/2007" | where {$_.Reason -eq "BlockListProvider"}

To get a list of all IP addresses blocked by IP Block List Providers:

Get-AgentLog -StartDate "12/21/2007" | where {$_.Reason -eq "BlockListProvider"} | ft Timestamp,IPAddress,ReasonData

You can also look for messages blocked by a particular IP Block List Povider:

Get-AgentLog -StartDate "4/15/2007" -EndDate "4/17/2007" | where {$_.ReasonData -eq "Spamhaus SBL-XBL"}

For messages scanned by the Content Filter Agent, the Reason field contains details like SCLAtOrAboveDeleteThreshold, SCLAtOrAboveRejectThreshold, etc. The ReasonData field contains the SCL value assigned to the message. To get a list of messages above the SCLDeleteThreshold, use the following command:

Get-AgentLog -StartDate "4/15/2007" -EndDate "4/17/2007" | where {$_.reason -eq "SCLAtOrAboveDeleteThreshold"}

As shown in the above examples, you can use the Get-AgentLog command and pipe the data to filter it based on the fields logged. You can get more details about agent logs - including the fields logged, from the Managing Agent Logging section in Exchange Server 2007 documentation.


2/9/2008: Added sidebar about additional configuration parameters available in SP1.

HOW TO: Remove values from multi-valued properties using Exchange shell

Related to the previous posts "HOW TO: Add additional email addresses to a recipient" and "HOW TO Update multi-valued attributes in PowerShell" - removing one or more values from a multi-valued property is equally easy (though not a one-liner... ). The following example shows you how to remove a domain from the list of BypassedSenderDomains - a multi-valued property of ContentFilterConfig:

$foo = Get-ContentFilterConfig
$foo.BypassedSenderDomains -="somedomain.com"
$foo | Set-ContentFilterConfig

Similarly, removing a proxy email address:

$mailbox = Get-Mailbox User1
$mailbox.EmailAddresses -="somexistingaddress@yourdomain.com"
$mailbox | set-mailbox

Symantec's State of Spam report: Spam rates declining

Here's something that should make messaging folks happier (as happy as you can be at this point given the DST 2007 huddle a lot of us may find ourselves in... :) Symantec's monthly State of Spam report shows some interesting trends over the past month.

In particular:
1. Overall percentage of spam "at the SMTP layer" declined to 69% in January 2007
2. Image spam reached an overall high of 45% but ended the month of January at 30%
3. Interestingly, "adult spam" is declining!

Interesting finds, though they may not echo the spam statistics from mail servers in your organization. Also note, the above does not represent anything blocked at the network layer - for instance, using techniques like Connection Filtering. It's unclear whether Symantec considers connections or messages blocked by the use of Real-Time Block Lists (RBLs) as being network layer or SMTP layer - a good portion of spam is blocked by using RBLs, imo.

Further, according to the report, North America still leads the world as the region where the highest amount of spam originates from.

Download Symantec's The State of Spam report (PDF, no registration required, short - 7 pages) from symantec.com.

HOW TO Update multi-valued attributes in PowerShell

Multivalued attributes are simply attributes that can have more than one value. Examples include members attribute of groups, or proxyAddresses attribute of recipients.

In GUI interfaces, it is simple to add/modify multivalued attributes. VBScript has control codes that allow you to control the interaction with multi-valued attributes - 1 = clear all entries, 2 = update all entries, 3 = append, and 4 = delete.

Exchange shell uses Add commands to add new entries to some multivalued attributes - e.g. Add-IPBlockListEntry to add another IP address or range to the IP Block List.

However, if you want to add more values to stuff like BypassedSenderDomains in ContentFilterConfig, there's no equivalent of an Add command. If you try to add a value to the attribute using the following syntax, the existing values will be replaced by the newer ones:

Set-ContentFilterConfig -BypassedSenderDomains "somedomain.com"

You could copy the existing values (e.g. microsoft.com,zenprise.com), and paste these along with the new value:

Set-ContentFilterConfig -BypassedSenderDomains "microsoft.com","zenprise.com","somedomain.com"

However, this copying and pasting can be tricky for attributes that have a lot of values. Another alternative (..thanks to Vivek for the suggestion):

$foo=Get-ContentFilterConfig
$foo.BypassedSenderDomains +="somedomain.com"
$foo | Set-ContentFilterConfig

This is a great workaround... and it works!

However, it'd be much easier to have a switch that lets you append to such multi-valued attributes, in keeping with the shell's promise of allowing most operations using one-liners.. something like Set-ContentFilterConfig -BypassedSenderDomains +domainthree.com,domainfour.com to add values to the attribute, and some other combination to remove a particular value.

Update:
To remove a value from a multivalued attribute, for example from the BypassedSenderDomains property of the ContentFilterConfig used in the preceding example:

$foo=Get-ContentFilterConfig
$foo.BypassedSenderDomains -="somedomain.com"
$foo | Set-ContentFilterConfig

Empower Your PowerShell with PowerGadgets

Exchange MVP Andy Webb showed off this great tool called PowerGadgets recently that can help you build cool gadgets - with or without Windows Vista. The gadgets take piped output from PowerShell commandlets and scripts (as well as SQL databases and web services).

Why I think this is so cool:
- it allows you to visualize data output by PowerShell commands and scripts. The samples shown here are simply displays, but you can create pretty fancy stuff like gauges and charts
- comparatively, it makes PerfMon - which is a useful tool, but so NT 4.0-ish - look like.. well, NT 4.0-ish :)
- critical or important data can be displayed in real-time or close-to-real-time on your desktop or Windows Vista sidebar

Andy created a gadget that takes the output from the get-antispamtoprblproviders.ps1 script [read previous post about this script - "Exchange Server 2007: How are RBLs performing?"]. The gadget runs the script and shows you number of messages blocked by the RBL provider Spamhaus.org (if you're using it with Exchange Server 2007's IP Blocklist Providers feature).

Yet another one of Andy's gadgets shows the Top 20 mailboxes by size. Very cool!

Thanks to input from Andy, and Vivek Sharma, I was able to effortlessly create a gadget that shows messages scanned by Exchange Server 2007's Content Filter agent.

I created another one that shows you number of messages dropped by Recipient Filter using Recipient Validation (drop messages for recipients that do not exist in AD).

Both the above gadgets use real-time output from perfmon counters captured by shell commands and refresh every minute.

You can create gadgets natively from within PowerShell, or use the PowerGadgets Creator GUI that walks you through creating one. Download PowerGadgets from www.powergadgets.com.

Here are the gadgets:
1. Andy's Spamhaus gadget
2. Andy's Top 20 Mailboxes gadget
3. Content Filter gadget - shows total messages scanned by Content Filter agent
4. Recipient Filter gadget - shows messages dropped because of invalid recipients

Update:
Right-click on the above links and save as/download.

If you've created a gadget using PowerShell, spread the word by adding a comment to this post.

Exchange 2007 Content Filter: How to move messages to Junk Mail folder

You've setup Exchange Server 2007, and configured the shiny new Content Filter agent (CFA), which is more than just a rewrite of the equally loved and hated Intelligent Message Filter (IMF) from Exchange Server 2003. The CFA has three thresholds - equivalent of Gateway thresholds and (gateway) actions in IMF. These are: 1) SCLDeleteThreshold 2) SCLRejectThreshold and 3) SCLQuarantineThreshold.

1. SCLDeleteThreshold: Messages with SCL equal to or higher than the SCLDeleteThreshold are deleted silently. To enable the SCLDeleteThreshold:

set-ContentFilterConfig -SCLDeleteThreshold 8 -SCLDeleteEnabled:$true

2. SCLRejectThreshold: Messages with SCL equal to or higher than the SCLRejectThreshold are rejected during the SMTP session, after the data is received. In this case, senders get a NDR. To enable the SCLRejectThreshold:

set-ContentFilterConfig -SCLRejectThreshold 7 -SCLRejectEnabled:$true

In the above case, Exchange doesn't accept the message. After the data is received, it responds with a 500 5.7.1 error and a rejection response (by default this response is: Message rejected due to content restrictions. This rejection message can be configured using the following command (response message used here is for illustration, not a real suggestion - Bharat) :

set-ContentFilterConfig -RejectionResponse "Stop spamming you *****!"

The actual NDR is generated and sent to the sender by the sending host. What the sending host will see after the message content is sent (and if you actually modified the rejection response based on my example :)
500 5.7.1 Stop spamming you *****!


Figure 1: Exchange Server 2007's Content Filtering agent can be configured with the equivalent of all 3 Gateway actions available in Exchange Server 2003's Intelligent Message Filter (IMF): 1) Delete messages 2) Reject messages and 4) Quarantine messages

3. SCLQuarantineThreshold: Messages with SCL equal to or higher than the SCLQuarantineThreshold are delivered to the quarantine mailbox, provided you have one configured. To enable the SCLQuarantineThreshold and configure a quarantine mailbox:

set-ContentFilterConfig -SCLQuarantineThreshold 6 -SCLQuarantineEnabled:$true -QuarantineMailbox:MyQuarantineMailbox@mydomain.com

This is an improvement over IMF, which had only one gateway action (and one corresponding gateway threshold). The Content Filter agent allows the flexibility of enabling all three actions on the gateway - the rule is: SCLDeleteThreshold > SCLRejectThreshold > SCLQuarantineThreshold.

To get a list of all three SCL values and whether each action is enabled or not, use the following command:

get-ContentFilterConfig | Select SCL*

So where's the equivalent of IMF's Store threshold? (to move messages to users' Junk Mail folders)?

It's called SCLJunkThreshold, and it resides in a different location - in the Organization configuration. It can be set using the set-OrganizationConfig command:

set-OrganizationConfig -SCLJunkThreshold 5

Setting SCLJunkThreshold not intuitive?

Before you jump to conclusions about this being counter-intuitive, or confusing - which it may be, consider this - it is in response to the different server roles in Exchange Server 2007.

The gateway actions - delete, reject and quarantine - can be thought of as message transport actions, and thus applicable to transport server roles (the Edge server, or the Hub if you have antispam agents enabled on it). Moving messages to users' Junk Mail folders can be thought of as something that happens at the Store, performed by the Mailbox Server role.

Another aspect to consider when setting the SCLJunkThreshold - if you're in a topology with an Edge server, the SCLJunkThreshold on the Edge doesn't impact anything. This needs to be set on your Exchange Org - the Edge server is not a part of it.

The Defaults: By default, the SCLJunkThreshold is set to 4. If you have an existing Exchange Server 2003 SP2 server installed, and you haven't tweaked the Store threshold, IMF v2's default Store threshold of 8 is used - this is what you'll see in the Org's SCLJunkThreshold. Given that the SCLRejectThreshold is set to 7 by default, messages will not move to users' Junk Mail folder unless the SCLJunkThreshold is lower than the transport thresholds (i.e. the Delete, Reject and Quarantine thresholds).

How the Junk Mail threshold is calculated: Unlike the transport actions of deleting, rejecting, and quarantining messages - which check for SCL equal to or higher than their respective thresholds, for moving messages to Junk Mail folder the Store checks for SCLs higher than the SCLJunkThreshold. This is consistent with the behavior of IMF in Exchange Server 2003 (read previous post: IMF Confusion - Store threshold rating text in UI).

If you want to disable rejection of messages with SCL of 7 or above, use the following command:

set-ContentFilterConfig -SCLRejectEnabled:$false

Related Posts:
- Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs
- Moving items to Junk Mail folder doesn't do much

Exchange Server 2007: Making SenderID work with non-Exchange smtp hosts

To make the SenderID filter work with non-Exchange smtp hosts acting as your mail gateways for inbound mail, you need to tell Exchange about them. If Exchange does not know about these, SenderID will not be able to determine the correct sending host, and you may see SenderID checks fail on all inbound mail.

This may have been a source of confusion for users in the past - which hosts do you consider as "internal" - the ones on your internal network? or only those located in perimeter networks (aka "DMZ")? how about those located at your ISP?

Exchange needs to know about all smtp hosts that do not belong to senders - the ones that will handle your inbound mail, whether at your ISP or in your perimeter or internal networks, before messages are delivered to the Exchange Org. (Start by taking a look at MX records in external DNS zones - do these point to non-Exchange servers? If yes, do these hosts route mail to Exchange servers?).

In Exchange Server 2003 this is done by entering internal smtp server addresses in Global Settings | Message Delivery properties | General tab | Perimeter IP List and internal IP Range Configuration.


Figure 1: Setting up internal or perimeter SMTP hosts in Exchange Server 2003's PerimeterIPList

On Exchange Server 2007, you can configure this from the shell:

set-transportconfig -InternalSMTPServers 1.2.3.4,1.2.3.5,1.2.3.6

Updates:
8/16/2007: SP1 Beta2 allows setting InternalSMTPServers parameter using the Exchange console, as shown below:


Figure 2: Setting up InternalSMTPServers using the Exchange console in Exchange Server 2007's new Global Settings tab | Transport Settings | properties | Message Delivery tab

Exchange 2007 Content FIlter: The Whitelist Is Here!

Messaging Hygiene features in Exchange Server 2003, including the Intelligent Message Filter (IMF), did not have a way to whitelist sending domains or SMTP addresses.

This is a follow up to a previous post, and one of the more popular ones on this blog— "IMF: Where's the whitelist?". ("IMF and whitelist" has for long been one of the most common search terms on this blog - Bharat).

Whitelists are common in most 3rd-party anti-spam tools. Adding domains or SMTP addresses of important senders like customers, vendors, or your CEO's home email address (almost always an AOL address... :) for instance, ensures messages from these domains or addresses do not get filtered by the anti-spam filter.

Bypassed Senders and Sender Domains: The Whitelist
The good news is— Exchange Server 2007's shiny new Content Filter Agent (or IMF v3 if you will) has whitelists! You can add SMTP addresses and domains to the Content Filter configuration, and have messages from these senders and domains bypass the Content Filter Agent. However, you need to resort to the Exchange shell (EMS) to manage it.

Use the following shell command to add sender SMTP addresses to the BypassedSenders list:

Set-ContentFilterConfig -BypassedSenders foo@somedomain.com,foo2@somedomain.com

Use the following command to whitelist the sending domain:

Set-ContentFilterConfig -BypassedSenderDomains somedomain.com,someotherdomain.com

Some whitelisting considerations
Before you start using whitelists, here are a few things you should consider:

• SMTP headers can be spoofed easily. If spammers spoof any of the addresses or domains you whitelist, your recipients may end up getting more spam as all of it will bypass the Content Filter.
  
• Use SenderID Filtering to detect and protect your mail system from header spoofing.
  
• Maintaining whitelists, just as maintaining blacklists, is a manual process that imposes its own management costs.
  
• Checking every inbound message against a list of whitelisted recipients imposes a performance penalty - miniscule as it may be. Use the whitelists sparingly.

Nevertheless, many IMF users have repeatedly demanded this functionality and it's great to finally have it in what some folks call IMF v3.0.

Bypassed Recipients: The Exception List
The Content Filter can also be configured with an exception list - to not apply the filter to inbound messages for particular recipients. This can be done from the console by going to Hub Transport | Anti-spam tab | Content Filtering -> properties | Exceptions. This list is limited to a 100 recipients - you can add generic recipients that you want to exempt from the Content Filter, such as sales@yourdomain.com, info@yourdomain.com, etc.

To add recipients to the exception list using the Exchange shell:

Set-ContentFilterConfig -BypassedRecipients sales@yourdomain.com,info@yourdomain.com

Related Posts:

• BypassedSenders, BypassedSenderDomains, and BypassedRecipients are multivalued attributes. The following post shows how to modify multivalued attributes:
  HOW TO Update multi-valued attributes in PowerShell
  
• Exchange 2007 Content Filter: How to move messages to Junk Mail folder
  
• HOW TO: Install anti-spam agents on Hub Transport server
  
• Quick antispam report or status check?
  
• HOW TO: Expose original senders and recipients of quarantined messages

IMF: Updates not showing up in Add/Remove Programs?

Some users report that recent updates to Microsoft's Intelligent Message Filter (for Exchange Server 2003) do not show up in Add/Remove Programs in Windows. Double-checking on some servers, I've noticed this - the last update that shows up on those in Add/Remove Programs is dated 8/10/2006.

If you look at the registry key for this update (all IMF Updates show up as update for KBA 907747) under HKLM\Software\Microsoft\Windows\CurrentVersion\Uninstall\KB907747, the DisplayVersion value - which is my best guess at what should show up in Add/Remove Programs or an indicator of which update was installed last - shows 2006.11.30. This was the update prior to the current one that was released in January.

One way to determine the version of IMF update currently installed on your system is to go to the \exchsrvr\bin\MSCFV2 directory (\exchsrvr is the install path for Exchange server), and look at the sub-folder that starts with the IMF update version - e.g. 6.5.xxx.x - with the latest date. The following screenshot shows the latest update that was downloaded on 1/18.

IMF: Updates live again

This just came in from Scott Roberts - IMF updates are live again. The latest update is dated Jan 11, 2007, version 6.5.7891.0.



Given the questions in different forums since the updates were paused/stopped, and particularly between yesterday (when the updates were supposed to kick in) and today - are some users really hooked to IMF? Or is it the warm & fuzzy feeling of seeing IMF being updated that makes you feel more secure or protected, just like updated anti-virus signatures? :)

If IMF doesn't update automatically on your servers, you can always check whether new updates are available by going to Microsoft Update (as shown in above screenshot). Alternatively, you can use the script provided by Scott in the post titled Demystifying Exchange Server 2003 SP2 IMF Updates on the Exchange team blog. The script also comes in handy if you do not want to use the auto-update settings to download and install *all* high-priority updates from Microsoft Update - you can schedule the script to run every other week, and have it only download and install IMF updates.

IMF: Updates stopped in December, to resume on Jan. 17th

A lot of users wondered why there were no updates to Microsoft's Intelligent Messages Filter (IMF) in recent weeks. Many Exchange administrators wondered if it was a problem with their servers when they saw IMF hadn't been updating automatically - some even posted on different Exchange forums wanting to compare the versions on their servers. (IMF updates are normally available twice a month).

Scott Roberts throws some light on this. According to his post on WSUS blog, IMF updates were blocked in December 2006. These will restart on Jan. 17th. The last published update is the one from Nov. 30th last year.

Perhaps this has something to do with Exchange Server 2007 being RTMed in early December.

I am surprised why there wasn't more information about this, and not as many users complained. Perhaps not as many users noticed, because there's no mechanism to nag you for older definitions like most anti-virus/anti-spyware agents do.

Either way, it would be great to hear about such issues from the Exchange team blog (rather than first being reported on the WSUS blog) a little earlier in the lifecycle of such incidents.

While we're still on the subject of IMF updates, it would be great to see an update to IMF that either allows you to keep the Custom Weighting file - MSExchange.UceContentFilter.xml - in the \exchsrvr\bin\MSCFV2 folder (rather than having to move/copy it to the new sub-folder created every time IMF updates are applied), or allows location of the file to be specified in a registry value.

Note: The Custom Weighting file is used to control SCL values assigned to messages using keywords found in message subject or body. Not all users use the Custom Weighting feature, introduced in Exchange Server 2003 Service Pack 2. More details in SP2 Release Notes.

Enabled by default: SMTP Tarpit in Exchange Server 2007

From a recent discussion, and something I've been wanting to post about for a while: SMTP tarpitting is enabled by default on ReceiveConnectors in Exchange Server 2007.

What is tarpitting? It's the process of introducing a delay in SMTP connections from hosts that are suspected of inappropriate SMTP behavior - for instance, by sending messages to non-existent addresses in your domain. (Tarpit is a noun, I use tarpitting as a verb to describe the process. The word probably can't be found in a dictionary, but perhaps appropriate usage to describe the process, just like telnetting, and emailing - Bharat)

If you've used Recipient Filtering on Exchange Server 2003 and selected the option to drop messages for recipients that do not exist in AD, it's a best practice to use SMTP tarpitting to get some level of protection from directory harvesting attacks. Directory harvesting is typically used by spammers to send email to addresses in your domain - which may or may not exist in your directory - to figure out which addresses are valid and which ones are not.

With the option in Recipient Filtering enabled, the SMTP virtual server will respond with a 550 error (550 5.1.1 User unknown) when it comes across an email address in the message's RCPT TO command - before the message body is transmitted. With tarpitting enabled, this response is delayed a few seconds, configurable using a registry setting (in Exchange Server 2003 on Windows Server 2003 SP1 - it is a Windows Server 2003 feature), as described in Microsoft KBA 842851. Most spammers will drop the connection if there's such a delay - it is more expensive for spammers to continue spamming/harvesting with such delays in place.

Does this sound too good to be true? What's the down side? Or are there any? On servers with high volume of SMTP traffic, you may notice more open connections, and open connections consume resources. The trick is to make sure this delay is not too high, resulting in more open connections for much longer, but high enough to make the sending hosts displaying suspicious behavior to drop connections.

Having said that, I've not come across many cases of performance degradation that could specifically be attributed to tarpitting delays, but you'll need to test this in your environment to figure out what works best.

Also note, authenticated connections are not subjected to tarpitting delays. Additionally, tarpitting only makes sense on ReceiveConnectors exposed to internet hosts.

Exchange Server 2007's ReceiveConnectors are configured with a tarpit interval of 5 seconds by default. A good way to observe this behavior is by telnetting to the SMTP port of an Exchange Server 2007 server and first sending a message to a valid recipient, and then trying to send a message to a recipient that does not exist.

To check the SMTP tarpit interval on your ReceiveConnectors, use the following shell command:

get-ReceiveConnector | select name,tarpitinterval

You can set it to a higher value - I have mine set to 10 seconds - using set-ReceiveConnector:

set-ReceiveConnector "Receive Connector Name" -tarpitinterval 00:00:10

The value is in hours:minutes:seconds.

At the time of writing, the documentation for set-ReceiveConnector command says this can be set in days as well (number of days and number of hours separated by a dot), but further it also states the maximum value for tarpitinterval is 10 minutes (00:10:00) - which is confirmed by the shell when you try to set it to a value higher than 10 minutes. (Nevertheless, technically speaking the documentation isn't wrong - you can in fact set it in days - e.g. 00.00:09:00 - as long as the value of days is zero! :) I'm told the doc will soon be changed/corrected).

To disable the tarpit behavior, set the value to 00:00:00.

Open Relay Database ORDB.org shuts down

After five and a half years, the non-profit organization ORDB is shutting down its DNS service (a Realtime Block List of open relays) and mailing list today, because the general consensus within ORDB is that open relay RBLs are "no longer the most effective way of preventing spam from entering your network".

The ordb.org web site will vanish come December 31st.

If you're using ORDB, the organization recommends removing ORDB checks from your mail servers.

I used ORDB in its early days (post-Exchange 2000) - they have been providing a valuable service, but as noted on their web site, spammers continue to evolve and are using more sophisticated methods (other than open relays) to send spam, and as such checking for open relays is far less effective. However, this is by no means a verdict against RBLs in general, some of which continue to list known sources of spam or spam operations, compromised hosts, et al (e.g. Spamhaus).

RBLs continue to drop a large amount of spam headed to our mail servers, in addition to other methods.

Exchange Server 2007: How are RBLs performing?

Exchange Server 2007 includes a script that reports on how Realtime Block Lists (RBLs) perform - it provides the number of messages blocked.

The script - Get-AntispamTopRBLProviders.ps1 can be found in the \Exchange Server\Scripts folder.

To run it, fire up Exchange Shell:

.\get-antispamtoprblproviders.ps1

Name Value
---- -----
Spamhaus SBL-XBL 6626
SORBS 33
NJABL 2

The script can be constrained to a particular time period by using the optional parameters -startdate and -enddate. If not constrained by time, it looks at all the logs. By default, a list of the top 10 RBL providers is provided. Hopefully you aren't using any more than that, but in case you do, you can get a list of top X number of RBL providers by adding the following: -top:X (where X is the number of RBL providers you want returned). This can also be used to get a list of less than 10 RBL providers.

If you feel like parsing through the log files, the agent logs are located in \Exchange Server\TransportRoles\Logs\AgentLog folder. To find out how to easily manage and filter agent logs, read related post "Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs".

Other Antispam-related scripts for reporting:
Get-AntispamFilteringReport.ps1
Get-AntispamSCLHistogram.ps1
Get-AntispamTopBlockedSenderDomains.ps1
Get-AntispamTopBlockedSenderIPs.ps1
Get-AntispamTopBlockedSenders.ps1
Get-AntispamTopRecipients.ps1

Anti-spam agents on Hub Transport servers

By default, anti-spam agents are not installed on Exchange Server 2007 servers with the Hub Transport server role - these logically belong on the Edge Transport server. However, if you do not intend to deploy an Edge server, you can install the agents on a Hub Transport server. [Read previous post: "How to install anti-spam agents on Hub Transport server"]

Update 6/15/2007
From answers to the comments below:

Perfmon counters v/s reporting from log files

Exchange Server 2003 and 2007 expose RBL statistics through performance counters. However, 1) Performance counters are flushed when services are restarted (in this case SMTP service / MSExchangeTransport service). As such, they neither provide historical information nor the rich details that Exchange Server 2007's agent logs provide (for more details on the agent log, read the related post "Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs") and 2) performance counters provide aggregate information, these are not instantiated per RBL/IP Block List provider, so you can't really determine number of messages blocked by each RBL, amongst other details.

Exchange Server 2003 Performance Counters


Figure 1: RBL-related Performance Monitor counters in Exchange Server 2003

As shown in the above screenshot of perfmon in report mode, the performance object is MSExchangeTransport Filter Sink. The following RBL-related counters are available for the object:
- Block List DNS Queries Issued
- Block List DNS Queries Issues/Sec
- Connections Rejected by Block List Providers
- Connections Rejected by Block List Providers/Sec
- Failed Block List DNS Queries
- Failed Block List DNS Queries/Sec

Exchange Server 2007 Performance Counters


Figure 2: RBL-related Performance Monitor counters in Exchange Server 2007

The corresponding perfmon object in Exchange Server 2007 is MSExchange Connection Filtering Agent, as shown in the above screenshot. The counters available are limited:
- Connections on IP Block List Providers
- Connections on IP Block List Providers/Sec
- Messages with Originating IP on IP Block List Providers
- Messages with Originating IP on IP Block List Providers/Sec

IMF: How to customize NDR message

Do you have IMF setup to reject messages at the gateway? If yes, you can customize the NDR message senders get. Exchange MVP and author of Exchange Server 2003 24Seven (renamed in its second edition to Microsoft Exchange Server 2003 Advanced Administration) Jim McBee shows you how in his post titled "Exchange 2003 SP2 IMF tuning".

IMF: Where's the whitelist?

Another frequently asked question - asked frequently enough to make its way here.

Does the Intelligent Message Filter (IMF) v1/v2 in Exchange 2003 provide any way of "whitelisting" sending SMTP addresses or domains? (I will continue to use whitelisting as one word. Although it cannot be found in the dictionary, it is a common term if you're reading this blog and work with messaging/anti-spam - Bharat)

The Sender Filtering feature provides a way to blacklist SMTP addresses and domains, so it's logical to expect a way to whitelist addresses and domains as well. However, there's no whitelisting feature in Exchange 2003 - or at least not out-of-the-box.

From the explanations I've heard, the one that comes across as the most reasonable: SMTP headers (and therefore the FROM: field in messages) can be easily spoofed. To a certain extent, it's hard to argue with that - it is true, and that is perhaps one of the biggest reasons we have to deal with spam and phishing today.

Some workarounds for whitelisting
In absence of a whitelist feature built into IMF, here's what can be done to ensure messages from trusted sources are not blocked:

 Global Accept List: Adding the IP address(es) of sending hosts to Connection Filtering's Global Accept List allows messages from the sending host to bypass Connection Filtering — inbound messages will be accepted even if the sending host is listed on a RBL, and IMF as well. To add IP addresses of sending hosts to the Global Accept list using the ESM:
1. Expand Global Settings | select Message Delivery -> Properties | Select the Connection Filtering tab


Figure 1: Connection Filtering's Global Accept List allows you to whitelist IP addresses of sending hosts

2. Click Accept | Click Add
3. To add a single IP address, type the address in the IP address field. To add a range of IP addresses, select Group of IP Addresses and type the subnet address and mask.
4. Click OK (thrice) to close the dialog boxes.

Next, we need to enable Connection Filtering in SMTP virtual server properties.
5. Select the SMTP Virtual Server that receives inbound internet mail | select Properties | General tab | Advanced button next to IP address
6. Select an IP address (you can enable/disable filters by IP address)
7. Click the Edit button
8. Check Apply Connection Filter


Figure 2: Antispam filters can be selectively enabled or disabled in SMTP Virtual Server properties

By whitelisting sending hosts' IP address(es), you're telling Exchange the sending host is trusted. Arguably, this is more effective and secure than whitelisting SMTP addresses and domains. (Also read Alexander Zammit's post regarding issues with having the same IP address on Global Accept list and on the local IP list).

Nevertheless, some folks would simply like to have the convenience of whitelisting addresses and domains. Almost every other anti-spam tool has it. (Update: Exchange 2007 has whitelisting. See Exchange 2007 Content FIlter: The Whitelist Is Here!)

 Custom Weighting: If the messages you want to whitelist contain particular keywords in the subject or message body, you can use the Custom Weighting feature as described in Exchange Server 2003 SP2 Release Notes. This allows you to lower or raise the SCL values assigned by IMF.

This entails creating an XML file called MSExchange.UceContentFilter.xml- the syntax is described in the Release Notes. You can simply cut and paste the sample from the Release Notes and modify it - it is fairly easy to do.

This approach could have been made a little more effective by simply allowing to look for the keywords in message headers as well. Not sure why headers were not included in the scanning mechanism for Custom Weighting - hopefully I will come across a reasonable explanation for this as well. :)

 IMF Tune: If the above methods don't work for you, and you absolutely need the convenience of being able to whitelist addresses and domains, you can look at third party utilities like IMF Tune from WinDeveloper. IMF Tune is an inexpensive tool that adds important functionality to IMF, making it closer to a full-blown anti-spam tool that many want IMF to be.

Besides whitelisting, IMF Tune allows you to configure a quarantine mailbox (instead of IMF's option to archive mail in the UceArchive folder), setup autoreplies for filtered mail, strip attachments, amongst other things.

Another likable IMF Tune feature is the ability to insert SCL values in message headers - which IMF will do for archived messages, but not for messages delivered to mailboxes. The SCL value is a MAPI property of a message that can be made to show up in Outlook [read 'Exposing SCL (Spam Confidence Level) in Outlook' on Exchange team blog], but the procedure isn't something most users would want to go through.

Having said this, I have always liked IMF and think Microsoft did a great job by including this - for free - as a web download with IMF v1, and as a part of Exchange Server 2003 SP2 with IMF v2. It serves a useful purpose in many small/SMB deployments where it meets the requirements to a good extent or where resource constraints rule out buying a third-party anti-spam product.

Besides, much as product managers want, not all features can be implemented in any given version, and given historical data, by version 3 these features/products become much more mature. (Think of the anti-spam features in Exchange Server 2007 as version 3 of Exchange's messaging hygiene features :).

Update: Manually removing IMF v1

Scott Schnoll has an update regarding the recent post Intelligent Message Filter (IMF) v1 download removed from microsoft.com, and the issue of not being able to remove IMF v1 from Add/Remove Programs (if one doesn't remember which account was used to install it or the account is no longer available) without reinstalling it. Here's the process to manually remove IMF v1 as documented in Microsoft Exchange Server Intelligent Message Filter v2 Operations Guide [available here].

To manually remove Intelligent Message Filter v1:
1. Stop all Exchange services (Information Store, System Attendant, SMTP, and Exchange-aware antivirus services).
2. Rename the MSCFV1 folder in X:\Program Files\Exchsrvr\bin, where X is the drive letter where Exchange Server is installed.
3. Rename the ContentFilter.dll file in \Exchsrvr\bin\
4. Open Regedit and export the following key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange
5. Delete the ControlFilterVersion subkey from the registry.
6. Restart the server. You should now be able to install Exchange Server 2003 SP2 without errors.

Intelligent Message Filter (IMF) v1 download removed from microsoft.com

A newsgroup poster recently informed that Microsoft has removed Intelligent Message Filter (IMF) v1 download from the IMF downloads page.

IMF v1 was a separate download. IMF (v2) is a part of Exchange Server 2003 SP2.

Excerpt from IMF v1 Release Notes [still available here] regarding uninstalling IMF:
To uninstall Intelligent Message Filter, use Add or Remove Programs in Control Panel. The account that was used to install Intelligent Message Filter must also be used to uninstall the filter. No other accounts will see Intelligent Message Filter in Add or Remove Programs.

It was (and continues to be to a certain extent) a common issue with IMF v2 installation: user is somehow able to install Exchange Server 2003 SP2 without uninstalling - or being able to uninstall - IMF v1. At times this happens because the user does not know which account was used to install IMF v1, or because the account no longer exists. As a result, IMF v1 does not show up in Add/Remove Programs.

In such cases, re-installing IMF v1 and then uninstalling it works. However, with IMF v1 now removed from the Microsoft web site, this will become a problem.

Perhaps it is justified to remove the IMF v1 download from the web site - to avoid scenarios where users inadvertently download and install it. However, with the download gone from the web site, you will now need to call Microsoft PSS to get it.

I haven't been able to verify with anyone at Microsoft if PSS will in fact provide a copy of IMF v1 when users are faced with this scenario, or if a workaround exists.

Microsoft opens up SenderID

Microsoft announced today it's putting its SenderID technology under its Open Specification Promise program. This allows anyone to create products or services using SenderID technology. SenderID lets (receiving) mail servers determine whether a sending mail host is authorized to send email for a particular domain, based on SPF records published in the sending domain's DNS.

SenderID has been around for 2 years, and is used by about 5 million domains according to Microsoft. It is a good anti-phishing defense, and also another component in a multi-layered approach to fighting spam.

CNET has more.

HOW TO: Install anti-spam agents on Hub Transport server

If you're using Exchange Server 2007 in a topology with an Edge Transport server, the anti-spam agents - Connection Filtering, Content Filter, Sender ID, Sender Filter, Recipient Filter, and Protocol Analysis - are installed on the Edge server. You do not need these agents installed on the Hub server(s).

In environments without an Edge server role, these need to be installed manually on the Hub Transport server(s).

To install anti-spam agents on a Hub Transport server, start Exchange Management Shell (EMS), navigate to the "\Exchange Server\Scripts" folder (by default C:\Program Files\Microsoft\Exchange Server\Scripts), and run:

.\install-antispamagents.ps1

Once installed successfully, the command will show each agent installed, whether it is enabled or not, and its priority [screenshot].

After the anti-spam agents are installed, you can see the Anti-spam tab in Organization Configuration | Hub Transport node in Exchange Management Console. (If your console was open during installation, you will need to restart it).

In such deployments, you can either route inbound mail through your service provider/ISP, or configure Hub Transport servers to receive inbound email directly. Read previous post "Exchange Server 2007: Internet email without Edge servers".

Related posts:
- Exchange Server 2007: Internet email without Edge servers
- Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs
- Exchange 2007 Content FIlter: The Whitelist Is Here!
- Exchange 2007 Content Filter: How to move messages to Junk Mail folder
- Exchange Server 2007: How are RBLs performing?

Spamhaus, the E360Insight lawsuit and the future of RBLs

If you've been keeping track of spam issues and RBLs, you probably took notice of the recent order by an Illinois court against leading RBL provider Spamhaus for listing Wheeling, IL-based E360Insight on its ROKSO (Register of Known Spam Operations) list. In a no-defence, default ruling, U.S. District Judge Charles Kokoras ordered [PDF] Spamhaus to pay $12 million to E360Insight.

Spamhaus claims 80% of spam received by users in U.S. and Europe is sent by a hard-core group of 200 known spam operations listed in ROKSO, and "can be traced via aliases and addresses, redirects, hosting locations of sites and domains". Steve Linford, head of U.K.-based Spamhaus, refuses to take E360Insight off the list. Here's the legal answer and update page on Spamhaus stating their side of the story.

E360Insight insists it's a legit operation that only sends marketing emails to folks who opt in.

(As a sidenote, Rule#1 for most users who want to fight spam is - do not click on the unsubscribe links in spam messages - it confirms to spammers that your address is live/valid. No wonder not many people really opt out.)

Spamhaus refused to respond to the lawsuit or to defend itself. Linford insists according to U.K. laws the messages E360Insight sends are spam, and therefore it will continue to list it. Besides, an Illinois court does not have jurisdiction on a company that's located on the other side of the pond.

Email users and sysadmins don't like spam, and can't be faulted to side with Spamhaus on this matter. I use Spamhaus and haven't yet come across cases where use of their list drops any legitimate mails for me. Besides, I use Spamhaus voluntarily, without any pressure or compulsion from anyone.

I've probably received some marketing messages from E360Insight in the past, and I doubt I would've ever opted in to any of their lists. I'm not an attorney and cannot comment about how this judgement was arrived at - E360Insight does have their side of the story including court filings available on their web site.

Here's an excerpt:

"Spamhaus didn't seem to care that we are an opt-in email marketing company. They didn't seem to care that the only way to get onto our mailing list was to signup for it. They didn't seem to care about the thousands of customers would not receive order confirmation messages or other email messages they requested. They didn't care about the lost dollars in legitimate commerce or about the employees that lost their jobs as a result".

(I can see you're all tears about now... :)

Disappointed as I am with the court's ruling against Spamhaus, I would've been more disappointed had Linford caved in and removed E360Insight from their ROKSO list. I'm glad that did not happen!

As a Spamhaus user I'm happy with their service.

RBLs provide a great mechanism to block plenty of spam from entering the system. Spam that won't enter your network, won't clog your email system, won't be processed by your anti-spam filters or stored in your mail queues or in your mail stores.

If mail senders get inappropriately listed on RBLs, they should be able to contact RBL providers and have their networks/IP addresses removed provided they have a valid case. And no, removal should not be as simple as going to a web site, entering an IP address and hitting the remove button.

Does this judgment cast a shadow on the future of RBLs/DNSBLs? I wouldn't be too quick to jump to that conclusion just yet. However, it will be interesting to watch the activity on this front.

IMF: Gateway thresholds per server

So far we've known that Exchange Server 2003's content filter - the Intelligent Message Filter (IMF), uses Gateway and Store thresholds and (Gateway) action that are configured once in Global Settings | Message Delivery | properties | Intelligent Message Filtering. These global settings apply to all SMTP Virtual Servers on all Exchange Servers in the Org that have IMF enabled.


Figure 1: Configuring Intelligent Message Filter's Gateway and Store thresholds for the entire Organization

Evan Dodds reveals the ability to configure per-server Gateway threshold and action for IMF in "Configure IMF gateway settings per server".

To enable the per-server Gateway thresholds, two registry values need to be created as follows:
Registry Key: HKLM\Software\Microsoft\Exchange\ContentFilter



Value: GatewayThreshold
Value type: REG_DWORD
Value data: 0-9

Value: GatewayAction
Value type: REG_DWORD
Value Data:
0x0 = No Action
0x1 = Delete
0x2 = Reject
0x5 = Archive

The above method allows you to control the Gateway threshold and action per server. You cannot control the Store threshold per server.

So the answer is, yes, it's possible using this undocumented registry hack, and Evan also suggests when one may need to use it.

Telling Exchange about (non-Exchange) SMTP servers

For anti-spam agents like IP Block List providers (aka DNSBLs/RBLs), and Sender ID, it is important to know about which non-Exchange SMTP servers receive inbound internet mail. This allows it to ignore the SMTP servers that belong to your organization when parsing headers (in topologies where the Edge server - or the Hub Transport server if you don't have an Edge server - sit behind other SMTP relay hosts).

In Exchange Server 2003, this can be done using Exchange System Manager by going to Global Settings | Message Delivery | Properties | General tab | Perimeter IP List and Internal IP Range Configuration | Add.

In Exchange Server 2007, this is done using the Set-TransportConfig commandlet:

Set-TransportConfig -InternalSMTPServers x.x.x.x,y.y.y.y,z.z.z.z

Related Posts:

- Exchange Server 2007: Making SenderID work with non-Exchange smtp hosts

Exchange Server 2007 exposes SCL in message headers

Exchange Server 2003's Intelligent Message Filter (IMF) assigns a SCL— a value that indicates the probability of a message being spam - to every message it scans. SCL is assigned based on factors such as message content, headers, and IP addresses.

By default, Exchange Server and Microsoft Outlook do not expose this SCL value assigned to a message - it's neither visible in any of Outlook's message views, nor inserted in message headers. It's a MAPI property of a message.

If IMF's Gateway action is set to Archive, Exchange allows you to save the SCL in message headers of archived messages. This is done using a registry hack mentioned in a previous post "IMF: Archiving spam".

You can also expose the SCL property as an additional column in Microsoft Outlook— after going through a series of steps documented in Exposing SCL (Spam Confidence Level) in Outlook on the Exchange team blog. You can then use that view in OWA as well.

Exchange Server 2007 inserts detailed anti-spam stamp in message headers. This includes both SCL and PCL values, SenderID lookup details such as the Purported Responsible Domain (PRD), SenderID status.

Here are antispam headers inserted by Exchange 2007:

X-MS-Exchange-Organization-PRD: alertsif.com
X-MS-Exchange-Organization-SenderIdResult: Pass
Received-SPF: Pass (MyServer.MyDomain.com: domain of
=?iso-8859-1?B?QWNhaVB1cmU=?=@alertsif.com designates 72.46.28.29 as
permitted sender) receiver=MyServer.MyDomain.com; client-ip=72.46.28.29;
helo=mx10.alertsif.com;
X-MS-Exchange-Organization-SCL: 4
X-MS-Exchange-Organization-PCL: 2
X-MS-Exchange-Organization-Antispam-Report:
DV:3.3.5705.600;SV:3.3.4604.600;SID:SenderIDStatus Pass;OrigIP:72.46.28.29

If you are a heavy OWA user, there's good news! Message headers can now be checked in OWA as well— although you need to open a message to check headers. There's no option to view full message headers in the preview pane by default, or a button to toggle between simple and full message headers that's common in some web-based email services such as Yahoo!'s.


Figure 1: Exchange 2007 inserts antispam stamp in message headers. You can also view message headers in OWA 2007.

Turning on Recipient Validation in Exchange Server 2007

The Edge Transport server role in Exchange Server 2007 performs messaging hygiene functions such as Connection Filtering, Sender & Recipient Filtering, Content Filtering, and SenderID Filtering. (You can also install antispam agents on the Hub Transport server role— for topologies without an Edge server. See 'HOW TO: Install anti-spam agents on Hub Transport server'.)

In Exchange Server 2003, Recipient Filtering has an option to "Filter recipients who are not in the directory". In Exchange Server 2007, this is called Recipient Validation. In a topology with Edge Transport server(s) deployed, Hub Transport servers transmit a list of valid SMTP addresses to the Edge. This is not required in environments without Edge servers - HT servers can look-up AD directly and drop messages for non-existent recipients.

The Exchange Management Console (in Exchange 2007 RTM) does not have the option to turn on Recipient Validation. Use the following command to turn it on:

Set-RecipientFilterConfig -RecipientValidationEnabled:$true

To check your Recipient Filtering configuration:

Get-RecipientFilterConfig | fl

In the above example, results from the Get-RecipientFilteringConfig cmdlet are piped to the Format-List cmdlet. fl is an alias for Format-List. See Using the Exchange Management Shell for some helpful information on Exchange shell.

Texas spammer turns "antispam-activist" after million dollar settlement

In an interesting twist, Texas spammer Rylak Pitylak - listed as the world's 4th "most prolific spammer" by Spamhaus - has turned over a new leaf and now refers to himself as an "antispam activist". This epiphany happened after coughing up a million dollars in a settlement with Microsoft and the state of Texas, according to a report on news.com.

Says Pitylak on his blog: "Over time I have come to see how I was wrong to think of spam as just a game of cat and mouse with corporate email administrators. I now understand why so much effort is put into stopping it."

It's great Mr. Pitylak now sees the light. If more spammers follow suit and turn into good guys, even if they don't go to the extent of becoming "antispam activists", it'll help us get rid of the scrouge of Unsolicited Commercial Email aka spam.

Thanks Microsoft, and the state of Texas, for pursuing this case and winning!

IMF: Archiving spam

As explained in previous posts, the Intelligent Message Filter (IMF) offers a few configuration options. The primary interface for configuring IMF is under Global Settings | Message Delivery properties | Intelligent Message Filtering tab.


Figure 1: IMF configuration options in Global Settings | Message Delivery | Properties | Intelligent Message Filtering tab

Here you specify a Gateway Blocking Configuration by telling Exchange what the Gateway SCL threshold is, and selecting an action to be taken when the threshold is met or exceeded. IMF assigns each message scanned a rating called "Spam Confidence Level" or SCL. Messages with SCL values that meet or exceed the Gateway threshold can be blocked from users' mailboxes.

One of the following actions can be selected to treat such messages:

•  Do Nothing: This simply forwards the message to the Store, and depending on the Store configuration, typically the messages should end up in a user's Junk Mail folder. Important consideration: the Store threshold should be *lower* than the gateway threshold.
  
•  Reject: Exchange rejects the message during the SMTP session. The sending server is responsible for generating any Non-Delivery Report (NDR) to the sender.
  
•  Delete: Exchange accepts the message and deletes it, without notifying the sending host and without generating a NDR. This is also referred to as silent delete.
  
•  Archive: When the Archive action is selected, Exchange delivers the message in the UCEArchive folder.

Some considerations and FAQs about archiving messages - most of these are documented in the IMF v2 Deployment Guide:

1. Where are the messages archived?
Messages are archived in the UCEArchive folder. By default this folder is created in \exchsrvr\mailroot\vsi 1 folder - where 1 is the instance number of SMTP virtual server.

2. Is there a way to change the location of the UCEArchive folder?
Yes, this is done by inserting a registry value of type String (REG_SZ) in the following location:
HKLM\Software\Microsoft\Exchange\ContentFilter
Value: ArchiveDir
Type: REG_SZ (string)
Value data: path to archive folder

3. I set the action to Archive. Why don't I see the UCEArchive folder in the default location?
The UCEArchive folder is created when the first message meeting or exceeding the gateway threshold is received.

Things to check when you don't see the UCEArchive folder:
- Is the action set to Archive?
- Is IMF enabled on SMTP virtual server?
Read the earlier post on changes to IMF v2 introduced in Exchange Server 2003 Service Pack 2 "Exchange Server 2003 SP2 and IMF"
- Is IMF working? (Check performance counters)
Read the earlier post "Troubleshooting IMF: Important Performance Counter" for more info.
- Are you receiving any messages meeting or exceeding the gateway threshold?
Check performance counters (again!) - MSExchange Intelligent Message Filter\Total Messages Assigned an SCL Rating of (number) - where number is the SCL rating. Counters are available for SCLs 0-9.
- Is the archive folder location modified using the above registry value?

4. I see a bunch of files with .eml extension in the UCEArchive folders. How do I view them?
You can open these .eml files with Outlook Express or using Notepad.

5. Managing archive files using the above method (opening each .eml file in Outlook Express or using Notepad) isn't very efficient. Is there a better way?
There's a popular third-party tool called IMF Archive Manager that makes managing the archived messages easier. You can download it for free from gotdotnet.com.

6. I don't have the time or inclination to sort through so many archived messages. Should I just set it to delete or reject?
If you're just starting out with IMF, I would recommend archiving initially. This will allow you to review the archive folder to see if there are any false positives - messages that are not spam, but got a higher SCL rating for some reason. Archiving will also allow you to finetune your IMF configuration based on the number of false positives you get using the gateway SCL threshold you use. Once you are comfortable with what you see in the archived messages, you can choose to reject or delete messages if you want.

7. I don't see the SCL rating anywhere in the archived messages. Is there a way to save the SCL with the messages?
Yes, by adding the following registry value, IMF can be configured to save the SCL assigned to archived messages. Exchange inserts the X-SCL x-header.
Location: HKLM\Software\Microsoft\Exchange\ContentFilter
Value: ArchiveSCL
Type: DWORD
Value data: 1

Exchange Server 2003 SP2, using SenderID? You need this hotfix!

If you've just upgraded to Exchange Server 2003 Service Pack 2 and have enabled Sender ID filtering, you need to apply a hotfix to prevent the server from stopping to respond.

According to the KBA, the problem occurs because of a race condition caused by the SMTP protocol. A race condition occurs when a device or a system tries to perform two or more operations at the same time. Because of the nature of the device or system, the operations must be performed in the correct sequence in order to be completed correctly.

Windows Server 2003: http://support.microsoft.com/default.aspx?scid=kb;en-us;905214
Windows 2000 Server: http://support.microsoft.com/default.aspx?scid=kb;en-us;909426
[No hotfix available yet for Windows 2000]

Interestingly, the Windows 2000 version of the KBA (909426) was released on Oct. 17 with revision 1.0, 2 days before the public release of Service Pack 2 on Oct. 19.
The Windows Server 2003 version (905214) was last revised on Oct. 21 and is currently at revision 3.0.

Exporting Sender Filter List

Messages from specific senders can be blocked by implementing sender filtering. Blocked senders are specified in Global Settings Message Delivery properties Sender Filtering tab (see screenshot).

The Senders list is saved in the msExchTurfListNames attribute of the Default Message Filter object. It is a multi-valued attribute. It can be exported using ldifde/csvde if you take the trouble to find out distinguishedName or objectClass of the attribute.

Here's a little script (... quite complicated actually, considering it can be done using either of the above methods using a single command... ) that will export the list to a text file -
getSenderFilterList.vbs (zipped).

Usage:
getSenderFilterList.vbs /f:filename.txt - exports to file
getSenderFilterList.vbs /f:filename.txt /s:y - exports to file and suppresses console output

Exchange Server 2003 SP2 and IMF

Before you install Exchange Server 2003 SP2 you need to uninstall Intelligent Message Filter (IMF). When you do this, the settings from the SMTP virtual server where you allowed filtering are also removed.

After SP installation if you go to Intelligent Message Filtering settings (Global Settings -> Message Delivery -> Properties -> Intelligent Message Filtering tab) and setup IMF with same settings as earlier, that alone does not reactivate IMF.

You still need to enable IMF from the SMTP virtual server properties -> General tab -> Advanced Edit (see screenshot, Flash demo of IMFv1 and IMFv2 settings - not entirely intuitive location for these settings, imo) and check "Apply Intelligent Message Filter" - just as you need to check the relevant options for recipient filter, connection filter, sender filter, and the new option of applying SenderID filter.

Exchange SP2 - Connection Filtering that actually works

I was excited to find out about Connection Filtering (screenshot) in Exchange 2003 - finally I could use RBLs (real-time block lists) without having to dabble with event sinks!! (This is from back in the days when RBLs were still sexy and could keep a good chunk of spam away from your users... )

Connection Filtering works with or without RBLs - you can also specifiy IP addresses manually in a Global Accept and Deny list.

However, in most enterprise environments Exchange does not perform the role of a smtp mail gateway to the Internet. That's usually assigned to the "more capable and secure" Linux/Unix servers running MTAs like Sendmail or Postfix. So all inbound email delivered to your Exchange servers is from the IP addresses of your own trusted mail gateways. Unfortunately, this renders Exchange's Connection Filtering useless!

Exchange Server 2003 SP2 changes that - it parses the headers of inbound email for the originating server's IP address, enabling connection filtering on any inside Exchange server. Bottomline, no matter where you put your Exchange Server 2003 SP2 box, connection filtering will just work!