Export and Import Content Filter Words or Phrases

In Exchange 2010 and Exchange 2007, you can add custom words or phrases as good or bad words to modify the Spam Confidence Level (SCL) assigned to messages. Messages with a good word or phrase are assigned an SCL of 0 and bypass other antispam agents that fire after the Content Filtering agent. Messages with a bad word are assigned an SCL of 9, and any configured action (delete/reject/quarantine) is taken based on the Content Filtering configuration.


Figure 1: Adding a custom word or phrase to Content Filtering configuration

To add a good or bad phrase to the custom words list using the EMC:

1. Go to Organization Configuration | Hub Transport | Anti-spam tab
   
2. Select Content Filtering and click Properties in the action pane
   
3. In Content Filtering Properties, select the Custom Words tab
   
4. Add a word or phrase in the following fields as required:
   • Messages containing these words or phrases will not be blocked:To add a good word or phrase, type it in this field
     
   • Messages containing these words or phrases will be blocked, unless the message contains a word or phrase from the list above: To add a bad word or phrase, type it in this field.
   
   

To add a word or phrase using the shell, besides the actual word or phrase, you must also specify the influence:

Add-ContentFilterPhrase "bad word" -Influence Badword

You can get a list of words or phrases added to Exchange by using the Get-ContentFilterPhrase cmdlet:

Get-ContentFilterPhrase | Select phrase,influence

Exporting and Importing Custom Words and Phrases
On the Edge Transport server, configuration information is stored in the local instance of Active Directory Application Mode (ADAM) on Windows Server 2003. In Windows Server 2008, ADAM is renamed to Active Directory Lightweight Directory Service (ADLDS). Unlike Exchange Server configuration information stored in Active Directory, which is replicated to all domain controllers in the AD forest, Edge Transport configuration information stored in ADAM/ADLDS is not replicated to other Edge Transport servers.

You can configure an Edge Transport server using a cloned configuration. See Using Edge Transport Server Cloned Configuration.

You can also export only the content filter phrases from one Edge Transport and import it to another Edge Transport server. To export the phrases, use the Get-ContentFilterPhrase cmdlet:

Get-ContentFilterPhrase | Select Phrase,Influence | Export-CSV "C:\MyFolder\CFPhrases.txt"

To import the phrases on another Edge Transport server, use the Add-ContentFilterPhrase cmdlet:

Import-Csv "C:\MyFolder\CFPhrases.txt" | foreach {Add-ContentFilterPhrase -Phrase $_.phrase -Influence $_.influence}

Troubleshooting Exchange Server Antispam: Disable Outlook Junk Mail

When troubleshooting antispam issues, particularly false positives (legitimate email incorrectly tagged as spam), frequently you run into scenarios where Exchange Server antispam features seem to be working correctly but you still see messages being delivered to the Junk Mail folder instead of the Inbox.

For instance, you whitelist a sender or sender domain, or add the sending IP address to the IP Allow List. You find message(s) from the whitelisted sender, domain or IP address still being delivered to the Junk Mail folder. You open the message and check the antispam headers - as expected, Exchange has stamped the message with a SCL of -1.

When troubleshooting Exchange antispam issues, it's best to turn off Microsoft Outlook's own antispam filtering. This is something Outlook does in addition to Exchange's server-side antispam agents. By default, Outlook's Junk E-mail Filter is turned on and set to Low.

To disable Junk Mail filtering in Outlook 2007:
1. Go to Tools menu | Options | Preferences tab | under E-mail section -> click the Junk Mail button.
2. In Junk E-mail Options, on the Options tab, select No Automatic Filtering...


Figure 1: Disable Outlook's own Junk E-mail Filtering when troubleshooting Exchange server antispam features

Related posts and links:

• About the Junk E-mail Filter
  

Disable Antispam agents on a Receive Connector

Exchange 2007's antispam agents are enabled for all Receive Connectors on a transport server. Is there a way to disable the agents on a particular Receive Connector?

Although not as simple as turning an agent off for each IP address or Receive Connector, Exchange 2007's new transport permissions model allows you to do this just as easily.

The ms-Exch-Bypass-Anti-Spam permission is what allows a sender to bypass antispam agents. By default, mail from authenticated senders is not filtered.

To allow unauthenticated/anonymous senders to bypass antispam filters on a particular Receive Connector, use the following command:

Get-ReceiveConnector "My Receive Connector" | Add-ADPermission -User "NT Authority\Anonymous Logon" -AccessRights ExtendedRight -ExtendedRights ms-exch-bypass-anti-spam

Related Posts and Links:

• Removing internal host names and IP addresses from message headers
  
• Why Get-TransportAgent doesn't agree with the Exchange console
  
• Exchange 2007 Content Filter: How to move messages to Junk Mail folder
  
• Exchange 2007 Content FIlter: The Whitelist Is Here!
  
• Exchange 2007 Transport Permissions Model
  

Quarantine Mailbox: Displaying original SCL, sender and recipients

In Exchange Server 2007, messages delivered to the quarantine mailbox show up as DSNs sent by the postmaster address of the default domain. In HOW TO: Expose original senders and recipients of quarantined messages, we modified the QTNE.cfg form for Microsoft Outlook to reveal original senders and recipients.

Although the original sender and recipient fields were added, the original SCL stamped on the quarantined message wasn't visible. The OriginalScl property was exposed in Exchange 2007 SP1, and is now included in the updated form in that post. Installing the updated form exposes the original SCL for messages in the quarantine mailbox, as seen in Figure 1.


Figure 1: The original SCL for messages in the quarantine mailbox can be displayed using the updated Outlook form

Exchange Server Antispam Updates: Do I have the latest update?

Not sure if the Exchange 2007 or 2003 (IMF) updates on your system are the latest and greatest? Doubt if the automatic update process is working?

You can use the Microsoft Update Catalog web site to search for these. You can also subscribe to the RSS feeds for each update (search result on the web site).

• Exchange Server 2007 Antispam updates  
  
• Exchange 2003 Intelligent Message Filter updates  
  

You can also use the site to search for other Microsoft products and subscribe to the feeds.

PowerShell: Listing multi-valued attributes

In previous posts, we've taken a look at how to update multi-valued attributes and remove values from multi-valued attributes using PowerShell/Exchange Shell (EMS).

Multi-valued attributes have a special significance in AD, and interfaces/APIs used to access AD. Whereas single-valued attributes can be retrieved and updated quite easily, multi-valued attributes come with a twist. Values from a multi-valued attribute are returned as an array (of values). To evaluate values in a multi-valued attribute, you need to iterate through each one (using a foreach loop in most cases). Similarly, when updating a multi-valued attribute, we need to remember we're adding/updating one value of what could possibly be multiple items in an array.

With that out of the way, a real-word experience relates to how these values are listed in Exchange shell cmdlet output. For instance, the BypassedSenders property of ContentFilterConfig may have a few dozen safe senders that you do not want to subject to the Content Filter. If you list these bypassed senders using Get-ContentFilterConfig, the output will list a few bypassed senders. Note the trailing dots to indicate there are more values?

Using a format-list or fl (Get-ContentFilterConfig |select BypassedSenders | fl) doesn't help.


Figure 1: Output from Exchange shell cmdlets does not list all values in multi-valued attributes

BypassedSenders and Safelist Aggregation

The Content Filter Agent does not filter messages from addresses on its BypassedSenders property, regardless of the recipient. This should not be confused with a recipient's Safe Senders list (used by the Safelist Aggregation feature) to bypass mail for a recipient from the senders he/she adds to Safe Senders list in Microsoft Outlook. CFA's BypassedSenders is global in scope.

To get a list of all values in a multi-valued attribute such as BypassedSenders:

$senders = (Get-ContentFilterConfig).BypassedSenders; $senders

Alternatively, you can list them without adding them to a hash table ($senders in above example):

(Get-ContentFilterConfig).BypassedSenders

Figure 2: Listing all values in BypassedSenders multi-valued attribute

Similarly, multiple IP addresses or address ranges in a Receive Connector's RemoteIPRanges property:

(Get-ReceiveConnector "MyConnector").RemoteIPRanges

or formatted as a table with the required info:

(Get-ReceiveConnector "MyConnector").RemoteIPRanges | ft Lowerbound,Upperbound,RangeFormat -AutoSize

Figure 3: Listing all values in RemoteIPRanges multi-valued attribute of a Receive Connector

Related posts:

• HOW TO Update multi-valued attributes in PowerShell
  
• HOW TO: Remove values from multi-valued properties using Exchange shell
  

Quick antispam report or status check?

Having received an annoyingly higher proportion of spam in my Inbox this morning, I wanted to quickly check what the antispam agents are doing. Here's a quick cmdlet (besides the ones to check whether the antispam agents are enabled, checking the Content Filter SCL thresholds, etc.).

Get-AgentLog -StartDate "6/16/2008" | group action | ft name,count -Autosize

What you get back:

Name Count
---- -----
RejectCommand 520
AcceptMessage 39
RejectMessage 163
QuarantineMessage 11
DeleteMessage 21

The filters are still working. Perhaps it's one of those days when you wake up to high volume of spam.

Note to self: Create a quick monitoring script that provides more information from agent logs, antispam configs, and perfmon counters.

Related posts:
- Keeping tabs on Antispam filters: A few handy scripts in Exchange Server 2007
- Exchange Server 2007: How are RBLs performing?
- Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs

Assigning SCL to messages scanned by 3rd-party antispam filters

Here's a problem I had a hard time resolving on Exchange Server 2003. Exchange Server 2007's Transport Rules resolve this within minutes.

Pretend you're taking a Microsoft exam:
Scenario: "You are the Exchange administrator for your organization... ".

Exchange has the Content Filter Agent (CFA), and the Edge Transport Server role designed to be a non-domain-joined mail gateway, located in perimeter networks. However, the Unix/Linux/Security folks in your organization don't trust Exchange to do the filtering (or act as the mail gateway). They insist on using open source anti-spam software, such as SpamAssasin (yes Eric, I remember CRM114 - the Controllable Regex Mutilator... and all the cool stuff you can command it to do. Just to set the record straight, I was suitably impressed back then, and continue to be so till this day.. :) on the non-Exchange SMTP gateways. After tweaking it for a number of weeks, they are able to make it work the way they want it to, or are close to it. It's implementation time!

Their solution is to insert an X-header in messages that looks like this:

X-Spam-Status:yes

That's it. Their job ends there.

As the Exchange team/administrator, your job is to ensure messages with that header end up in users' Junk Mail folder.

Exchange Server 2003 did not provide any way, out-of-the-box, to be able to inspect message headers and stamp SCL. To add some contextual fun - back then, our "solution" was a doc that showed the end-users how to create an Outlook rule to move such messages to Junk Mail. Only if Exchange/Windows admins could code... I can hear you think.

 Creating a Transport Rule: Exchange Server 2007's Transport Rules functionality allows you to accomplish this easily. Here's how:
1. Fire up EMC | Organization Config | Hub Transport | Transport Rules tab
2. Click on New Transport Rule in the Action pane
3. Give the new rule a name, add a comment if you wish
4. In the Conditions page, select the condition when a message header contains specific words
5. In the Step 2 edit box, click on the message header link

6. Type X-Spam-Status | click OK
7. In the edit box, click on the specific words link
8. Type yes | click OK | click Next
9. In the Actions page, select the action set the spam confidence level to value
10. In the rule description, click on the 0 link and add a value that's above your SCLJunkThreshold | click Next
11. On the Exceptions page, click Next if you do not want any exceptions to this rule
12. Click New | click Finish to close the wizard

Or you can use the following commands:

$condition = Get-TransportRulePredicate HeaderContains
$condition.MessageHeader = "X-Spam-Status"
$condition.words = @("yes")
$action = Get-TransportRuleAction SetSCL
$action.SCLValue = 5
new-TransportRule "Stamp SCL" -condition @($condition) -action @($action)

 Disabling the Content Filter agent: Since you have a 3rd-party filtering solution running on your non-Exchange SMTP host(s), you can disable the Content Filter Agent. Messages exceeding SCLJunkThreshold will still be moved to Junk Mail folder.

Disable-TransportAgent "Content Filter Agent"

Alternatively, you can leave the CFA enabled, but disable the Delete, Reject and Quarantine actions.

Set-ContentFilterConfig -SCLDeleteEnabled $false -SCLRejectEnabled $false -SCLQuarantineEnabled $false

Send a test message with the X-header X-Spam-Status:yes. The message has the SCL value set by the Transport Rule. If it is above the SCLJunkThreshold, it should be delivered to the Junk Mail folder.

Why this doesn't work with Delete, Reject or Quarantine thresholds enabled?

The Content Filter Agent fires on the EndOfData SMTP event. On Hub Transport servers, the Transport Rules Agent fires on the OnRoutedMessage event. If the CFA is left enabled, along with any of the above actions, messages may get deleted, rejected or quarantined. The Transport Rules Agent never gets to see deleted and rejected messages at all.

What about quarantined messages?

It's understandable if the message is already deleted or rejected - there's nothing left for the Transport Rule agent to act on! However, why doesn't the agent act on quarantined messages? With Pipeline Tracing turned on, I spent some precious cycles trying to unravel the mystery, with little success. As it turned out (thanks to helpful Exchange team folks for pointing out... ), I was overlooking the obvious - Quarantined messages are wrapped in an NDR "envelope". If you've ever logged into the quarantine mailbox and looked at quarantined messages, you know what these look like. (Related post: "HOW TO: Expose original senders and recipients of quarantined messages") The Transport Rules Agent does not see the original message headers, including our X-Spam-Status header. As a result, the rule never fires.

Life on the Edge

If using an Edge Transport server in the mix, things change. The Edge Rule agent fires on the EndOfData event, unlike Hub Transport servers. When two transport agents fire on the same SMTP event, you can set the priorities of each accordingly so one fires before the other. This is described in "How to Make the SCL Value Available to Edge Transport Rules" in Exchange Server 2007 documentation.

However, it's a lot simpler to just disable the Content Filter Agent. This meets the requirements of this scenario, where anti-spam filtering is done by 3rd-party anti-spam filters and Exchange isn't required to do any filtering at all.

Keeping tabs on Antispam filters: A few handy scripts in Exchange Server 2007

One of the more useful improvements in Exchange Server 2007 is the abundance of logging for different features and components (read previous post "Exchange Server 2007: How many logs hath thee?"). In particular, the antispam agent logs fill an important gap in monitoring, reporting and troubleshooting message flow as it relates to antispam agents (read previous post "Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs"). As a messaging/Exchange administrator, you want to be able to pin point what the antispam agents have been up to, and determine if particular messsages, or sending hosts, domains or email addresses have been blocked by any of the antispam agents. The antispam agent logs and the get-agentlog command allow you to do this quickly and efficiently.

In addition to the the get-agentlog command, Exchange Server 2007 also ships with a number of canned scripts that help you keep tabs on what the agents are doing. These scripts are found in the \Exchange Server\Scripts directory, where \Exchange Server is the path of the Exchange Server 2007 installation. Note, no documentation or support is available for these scripts - they are meant to be examples you can use to write your own scripts. Let's take a peek in the directory and see what we find.

Commandline parameters used by antispam scripts

Most of the following scripts take the same (optional) parameters.
-top n: Where n is the number of results to display. If not specified, the script defaults to (top) 10.
-StartDate: Start date/time
-EndDate: End date/time
-location: path of agent log files. If no path is specified, the agent works against the default agent log file location.

1. Get-AntispamFilteringReport.ps1: Takes one of the following values as a mandatory parameter: 1) connections 2) commands 3) messagesrejected 4) messagesdeleted 5) messagesquarantined, and displays statistics for each agent.
2. Get-AntispamSCLHistogram.ps1: Provides a breakdown of number of messages stamped with each SCL value.
3. Get-AntispamTopBlockedSenderDomains.ps1: Lists top ten sender domains from which mail was blocked.
4. Get-AntispamTopBlockedSenderIPs.ps1: Lists the top ten IP addresses blocked by antispam agents, and number of messages blocked from each.
5. Get-AntispamTopBlockedSenders.ps1: Lists the top ten blocked senders (SMTP email addresses) and number of messages blocked from each. The script can report on P1 (i.e. address in message envelope in the MAIL header) or P2 addresses (from headers in message body like FROM). Specify the option as a commandline argument (optional parameters -top n, -StartDate, and EndDate used in this example):

.\Get-AntispamTopBlockedSenders.ps1 P1 -top 20 -StartDate "12/1/2007" -EndDate "12/10/2007"

Replace the P1 in the above command with P2 to report on P2 senders.
6. Get-AntispamTopRBLProviders.ps1: Lists the top ten RBLs (aka "IP Block List Providers") and messages blocked by each (read previous post: "Exchange Server 2007: How are RBLs performing?").
7. Get-AntispamTopRecipients.ps1: Lists the top ten recipient addresses that receive spam. The addresses may or may not exist in your Organization - the reporting is based on actions taken by antispam agents on incoming messages. Recipient Filtering with Recipient Validation (dropping messages for recipients that do not exist in AD/GAL) is a great way to drop a large number of messages.

The following scripts are not used for reporting:
8. Reset-AntispamUpdates.ps1: Uninstalls any antispam updates and reinstalls the original "out-of-box" antispam data.
9. Install-AntispamAgents.ps1: By default, antispam agents are not installed on the Hub Transport server role (read previous post: "HOW TO: Install anti-spam agents on Hub Transport server"). This script installs them if you need to.
10. Uninstall-AntispamAgents.ps1: Uninstalls antispam agents.

Connection Filtering, RBLs and SMTP logs

Exchange Server 2003's Connection Filtering feature allows you to block connections from IP addresses explicitly added to the Global Deny List, or drop messages from IP addresses listed on a RBL (Real-Time Blackhole List / Real-Time Block List).

Note: The term "RBL" is commonly used to describe DNS Black Lists (DNSBLs), but it's a trademark of Mail Abuse Prevention System (MAPS), one of the first RBLs. It was acquired by Trend Micro in 2005. Exchange Server 2007 uses the term IP Block List Provider.

When connections from IP addresses listed on the Global Deny List are dropped, you don't find any traces of those dropped connections in SMTP logs [for more details about SMTP logging in Exchange Server 2003, read previous post "Logging SMTP protocol activity"].This is understandable - the IP addresses are explicitly added to the deny list by an administrator, and a SMTP session is never initiated. It's not as important to log these (though many would prefer to capture information about connections so blocked). If in doubt about which hosts are getting blocked, all one needs to do is look at this list.

When it comes to DNSBLs, the behavior is different. Connections are not dropped. Messages are dropped on RCPT TO: command. This allows you to receive mail for addresses on the Exception List. RFC 2821 requires senders to be able to contact the postmaster of a domain. You can also add some of the generic addresses from your domain to the Exception List if you don't want email to these addresses blocked.

Here's a demo of the SMTP session. As seen in the session, the sending host gets a 550 5.7.1 error code because the IP address is listed on a DNSBL, and the recipient (in the RCPT TO: command) is not listed on the Exception List.

Here's what the SMTP log shows (some irrelevant fields removed):

2007-07-25 19:04:54 172.31.1.10 0 HELO - - 250 0 49 4 0 SMTP - - - -
2007-07-25 19:04:59 172.31.1.10 0 MAIL - +from:foo@mydomain.com 250 0 41 26 50 SMTP - - - -
2007-07-25 19:05:11 172.31.1.10 0 RCPT - +to:jadams@exchangelabs.net 550 0 0 31 5017 SMTP - - - -
2007-07-25 19:05:15 172.31.1.10 0 QUIT - - 240 22943 72 4 0 SMTP - - - -

SMTP logs the response code 550, but not the enhanced status code 5.7.1. Having said that, this enhanced status code doesn't by itself allow you to narrow it down to DNSBLs.

Exchange Server 2007's Connection Filtering agent behaves differently - connections from IP addresses explicitly listed in the IP Block List (equivalent of the Global Deny List in Exchange Server 2003) are not blocked. This change in behavior allows such connections to be logged. Detailed information is available from the agent logs [read previous post "Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs"] about why a message was accepted/rejected by anti-spam agents.

One example of the kind of reporting/troubleshooting capabilities this provides: you can search the agent logs for messages blocked by Connection Filtering agent based on DNSBL lookups:

Get-AgentLog -StartDate "7/27/2007" | where {$_.Reason -eq "BlockListProvider"}

The Get-AgentLog output:

Timestamp : 7/27/2007 6:42:08 AM
SessionId : 08C99DA727C8EF31
IPAddress : 68.45.233.59
MessageId :
P1FromAddress : dbuuxwxjlm@bobach.com
P2FromAddresses : {}
Recipients : {bf5b7b59@mydomain.com}
Agent : Connection Filtering Agent
Event : OnRcptCommand
Action : RejectCommand
SmtpResponse : 550 5.7.1 Recipient not authorized, your IP has been found on
a block list
Reason : BlockListProvider
ReasonData : Spamhaus SBL-XBL
Diagnostics :

SCRIPT: Turning on Filter Junk Email

OWA users who never logon using Outlook do not have their Junk Mail filtering option turned on by default. Exchange MVP Glen Scales has a script here that allows you to turn this on programmatically for Exchange Server 2007 users.

Why Get-TransportAgent doesn't agree with the Exchange console

You disable a particular anti-spam agent - let's say the Content Filtering Agent - from the Exchange console.



Next, you use the Get-TransportAgent command to get the status of transport agents - and surprisingly the Content Filter Agent shows up as Enabled!



Why doesn't Get-TransportAgent agree with the Exchange console about the status of an agent?

Fellow MCT and Exchange geek David Elfassy blogged about it recently.

The Anti-Spam tab is located under the Organization Configuration | Hub Transport node in the console. Disabling an agent from the console prevents the agent on all transport servers in the Organization from taking any action. It's an Organization-wide or global setting.

The Get-TransportAgent command is transport server-specific. It shows the status of agents on a particular server. With the Org-wide configuration set to disabled, an agent enabled on a particular transport server behaves like a soldier instructed to fire but not kill. (May not be the best analogy - it doesn't exactly scare spam, and it doesn't do much else).

To draw a parallel to Exchange Server 2003, you can enable settings on a particular filter - e.g. Recipient Filtering, in Global Settings. However, you still have the flexibility of being able to enable/disable it on a per-SMTP virtual server basis. This is somewhat similar.

To get the global/Org-wide status of a particular filter, you need to use the corresponding command for that filter.
- For Content Filter Agent: Get-ContentFilterConfig
- For Recipient Filter Agent: Get-RecipientFilterConfig
- For Sender Filter Agent: Get-SenderFilterConfig
- For SenderID Agent: Get-SenderIDConfig
- For Protocol Analysis Agent: Get-SenderReputationConfig

If you've carefully looked at the above list, you're probably wondering why there's no mention of Connection Filtering Agent in it. The Connection Filtering Agent handles multiple tasks which are listed individually in the console.

To check the global setting for configuration of each, use the corresponding config command:
- For IP Block List: Get-IPBlockListConfig
- For IP Block List Providers: Get-IPBlockListProvidersConfig
- For IP Allow List: Get-IPAllowListConfig
- For IP Allow List Providers: Get-IPAllowListProvidersConfig

To "disable" an agent in Organization Configuration (equivalent of what you do in the console) using the shell, use the corresponding Set command, e.g.:

Set-ContentFilterConfig -Enabled $false

Next, let's take a look at how you can disable an agent on a transport server. Generally, if the Get-Blah command returns a property like Enabled: True, you can use Set-Blah -Enabled $false to disable it (You probably know where we're going with this by now... :). Not so for Get-TransportAgent. Its "Set" counter-part - Set-TransportAgent allows you to modify only one property -Priority. You can change the order in which these agents fire by changing their Priority.

You use the Disable-TransportAgent command to disable an agent on a transport server:

Disable-TransportAgent "Connection Filtering Agent"

No points for guessing the command used to enable a transport agent - Enable-TransportAgent

Header Firewall: Why spammers can't insert fake SCL (and other Exchange Organization) X headers

After almost a year and a half of working with Exchange Server 2007, I still stumble into features I haven't tested before. When it happens, it blows holes into my theories about how well I know E12. :)

It started when investigating why a certain message - likely not from an authenticated source, would have a SCL of -1 (seen in message headers as X-MS-Exchange-Organization-SCL: -1).

One suggestion that got me thinking: it may have been a fake header, perhaps inserted by a spammer. I tested this a few times in different combinations (by inserting SCL and PCL headers before, after and in between other X-headers, like the ones below:


Note: Exchange uses the PCL and SCL headers seen above, but the QCL header in the above screenshot is fabricated/fake.

When the above messages came in, all the headers with X-MS-Exchange-Organization* were removed by Exchange. Nice work, Microsoft, I said to myself, and asked for an explanation.



Someone from the team immediately pointed me to Header Firewall - a supercool feature (not so much in what it does but the fact that it exists). It removes protected Exchange Organization and Forest headers from messages that enter or exit the Exchange Organization. This is the reason you're not likely to see fake SCL, PCL, SenderID, and other such headers in messages received by Exchange Server 2007 Edge or Hub Transport servers.

Preserving Organization and Forest X headers: If you want these to be preserved for inbound mail from particular sources, it is recommended to do so using a separate Receive Connector, just as you would create one if you needed to allow unauthenticated relaying [read previous post "Exchange Server 2007: How To Allow Relaying"], and assign MS-Exch-Accept-Headers-Organization and MS-Exch-Accept-Headers-Forest permissions. Similarly, to send Organization and Forest headers to particular external mail systems, it is recommended to create a separate Send Connector for those address spaces, and assign the MS-Exch-Send-Headers-Organization and MS-Exch-Send-Headers-Forest permissions.

Getting back to the original question - when do you see a SCL of -1? Either when the sender is authenticated, or the sending host is on the IP Allow List.

Read more about the Header Firewall feature in the Exchange Server 2007 documentation. A list of Exchange Organization and Forest x-headers can also be found there.

Moving items to Junk Mail folder doesn't do much

Scenario: Junk mail makes it to your Inbox, without getting filtered by any of Exchange's anti-spam filters.
Action: User moves item to Junk Mail folder.
Result: This accomplishes little more than having the piece of UCE moved out of the Inbox. Contrary to what many may think and like to believe, moving a message manually to the Junk Mail folder doesn't really have Outlook or Exchange do anything at all with that item. No, the sender or sending domain or host do not get blacklisted or added to Microsoft Outlook's Blocked Senders List. Yes, you're not likely to see a similar message from same sender, domain or host get treated any differently based on that action.

To make Microsoft Outlook filter messages from the same sender, the sender needs to be added to the Blocked Senders list manually (by right-clicking message | Junk E-mail | Add Sender to Blocked Senders List).



Does the Blocked Senders List make it to the Exchange Hub or Edge server using the Safelist Aggregation feature? No. Safelist Aggregation is only used for aggregating the safe list, as the name of this feature correctly suggests. The Blocked Senders List is only used by Microsoft Outlook, without any server-side action being taken on messages from senders on this list.

Related Posts:
- Exchange 2007 Content Filter: How to move messages to users' Junk Mail folder
- TechEd 2007: Notes from the BOF Session on Fighting Spam With Exchange Server 2007
- Exchange 2007 Content Filter: The Whitelist Is Here!
- HOW TO: Expose original senders and recipients of quarantined messages
- Connection Filtering, RBLs and SMTP logs

Exchange Server 2007: Can the quarantine mailbox be an external smtp address?

In Tuesday's Birds of a Feather (BOF) session on Fighting Spam with Exchange Server 2007, someone from the audience wanted to know if an external (smtp) email address can be used as a quarantine mailbox. Exchange does no validation of the email address used for a quarantine mailbox - as long as it's a valid smtp address, it can be used.

However, the results may not be what one's looking for. The quarantined message appears as a NDR. Outlook knows how to treat it correctly using the Send Again functionality, which won't happen if you're forwarding quarantined messages to an external address. The From headers will not be rewritten to identify the original sender of the message.

So yes, it can be done - but what's the point?

TechEd 2007: Notes from the BOF Session on Fighting Spam With Exchange Server 2007

Tuesday night was a Birds of a Feather (BOF) night, with plenty of BOF sessions and chalk talks scheduled. My session on Fighting Spam With Exchange Server 2007 (not a very creative title, I know... but it got the message across :) was scheduled for 7:45 PM. I was positive I'll probably be the only person in the room - 7:45 PM isn't exactly the time I'd want to attend a TechEd session on anything, with the exception of a few good speakers.

On a side note, keeping my expectations low also meant I could claim the attendance was more than I expected even if only one person showed up. Nevertheless, the room filled up as we got closer to the start time.

It was an interesting discussion on anti-spam in Exchange Server 2007, as well as Exchange 2003.

Some takeaways:

• Exchange's anti-spam features consist of more than just Intelligent Message Filter (IMF) - there's 1) Connection Filter: allows using IP Allow Lists, IP Block Lists, IP Block List Providers (RBLs) and IP Allow List Providers 2) Sender Filter: block sending domain or SMTP address 3) Recipient Filter: block internet messages to list of valid recipients, perform Recipient Validation to block messages for non-existent recipients 4) Sender ID Filter: looks up SPF records in purported sending domain's DNS zone, protects from address spoofing and phishing 5) Content Filter (aka "IMF"): Blocks messages based on content
• 3 Gateway Thresholds and Actions: Exchange Server 2007's Content Filter Agent supports 3 gateway thresholds and actions - 1) Delete 2) Reject and 3) Quarantine
• Quarantine functionality: Though the quarantine functionality in Exchange Server 2007 is a step forward compared to Exchange Server 2003's archiving feature, it's not as full-featured as that offered by many third-party products. End-users have no control over the quarantine - an administrator has to inspect quarantined messages and make the decision to deliver or delete. However, it's important to understand that a) it's not compulsory to use the quarantine and b) The end-user mechanism for accessing suspicious messages is the Junk Mail folder c) Quarantining is one more layer available to admins - between the gateway actions of Rejecting or Deleting mail and the Store action of delivering to Junk Mail
• Exchange Server 2007's Connection Filter agent does not immediately drop connections from IP addresses explicitly listed on the IP Block List. Exchange Server 2003 does this for IPs listed in its Global Deny List. This change in behaviour allows such "filtered" connections to be logged in the agent log. Some customers like to see these logged, whereas many want connections to be dropped immediately.
• Many customers prefer anti-spam appliances like Barracuda and IronPort in the perimeter, and consider these very effective, and generally lower cost compared to an Exchange 2007 Edge Transport server.
• No studies or test results are available yet to demonstrate how the Edge Transport server performs compared to such appliances. It would be great to have a feature and performance comparison of these.
• Many customers are interested in Greylisting - a technique that registers each connection's "triplet" of a) Sending host's IP address b) Sender's address c) Recipient's address and looks up a database. If not found, the connection is immediately dropped and the triplet added to the database. The technique is based on the idea that valid RFC-compliant SMTP hosts will attempt to resend the message after a set interval. The (receiving) SMTP server will find the triplet in the database upon the second attempt, and consequently accept the connection.
• Interested in learning more about Greylisting? Evan Harris has a whitepaper titled "The Next Step in Spam Control: Greylisting"
• Exchange Server 2007 SDK has the code for a Greylisting agent. This needs to be compiled into a DLL using Visual Studio 2005, and installed using the accompanying installation script. Since this is just a programming sample provided in the SDK, it makes sense to test this adequately in a test environment and not expect any production support
• Freeware Greylisting transport sinks/add-ins for Exchange Server 2003 are available on the web, like this one from Chris on Grynx.com
• Over a period of time, the effectiveness of techniques like Greylisting may decrease, but as of now it still eliminates 80-90% of spam.
• SMTP Tarpitting is a technique that inserts a delay in SMTP responses to SMTP clients that display suspicious protocol behavior. More details about tarpitting in a previous post titled "Enabled by default: SMTP Tarpit in Exchange Server 2007"
• Safelist Aggregation: This allows Exchange to aggregate senders marked as safe by (Outlook/OWA) users. If an EdgeSync subscription exists (in topologies with an Edge server), the safe list is replicated to the Edge as part of EdgeSync. Exchange's use of safe lists can reduce false positives significantly. It's not required to have an Edge server - Hub Transport servers with anti-spam agents installed can also use this feature.
  

TechEd 2007: Let's talk about fighting spam with Exchange Server 2007

If you're going to be at Microsoft TechEd 2007 in Orlando next week, and are interested in sharing your experiences in fighting spam using Exchange Server's anti-spam features, here's a Birds-of-a-feather (BOF) session just for you.

What: BOF session "Fighting Spam With Exchange Server 2007"
When: Tuesday, June 5th @ 7:45 PM
Where: Microsoft TechEd, Orlando, FL
What's a BOF session? : Birds-of-a-feather sessions are not presentations or lectures. These are meant to be interactive discussions, sans the PowerPoint slides and demos you typically see in breakout sessions.
Why so late?: Tuesday night is an all-BOF session night, with no other conference activities. Microsoft will be serving food and drinks.

Abstract:
“Exchange Server 2007 anti-spam features are a significant improvement over previous versions, and come with interesting deployment and topology questions. Are these effective in the fight against spam? This BOF session will be a round table for Exchange administrators to talk about their experiences with Exchange Server 2007 anti-spam features, including issues such as Edge servers, use of IP Block List Providers, and new features and techniques such as Sender Reputation, and Safelist Aggregation.”
Audience: IT Professional

Grimes Gripes About RBLs

InfoWorld columnist Roger Grimes gripes about Real-Time Block Lists (aka Real-Time Blackhole Lists, DNSBLs or DNS-based Block Lists. The Exchange Server 2007 term for RBLs is "IP Block List Providers").

The feelings towards RBLs are understandable - you won't be a fan of RBLs if your IP addresses are the ones getting listed, and your users breathing down your neck about bounced messages. It's important to monitor RBLs to determine if your IP addresses are listed in any. (Incidentally, Zenprise for Exchange does this automatically several times a day - configurable by the administrator - and alerts you if any of your IP addresses are listed on RBLs. Your external IP addresses are detected automatically based on messages sent to Zenprise's Zmail hosted service).

Once listed, some RBL providers make the process of getting IP addresses delisted a long and painful one. At times, and in spite of your best efforts and assurances by the provider that the IP address(es) will be delisted, they continue to live on in such lists. On the other end of the spectrum are RBL providers that make the delisting process as simple as entering the listed IP address in a form on their web site, without any checks in place. I would shy away from using the latter.

Regardless, if you are on the other side, as a victim of spam rather than a victim of RBLs, good/reputed RBL providers like Spamhaus (which has its detractors as well - read previous post: "Spamhaus, the E360Insight lawsuit and the future of RBLs"), can help in dramatically reducing spam at the mail gateway - by as much as 90+ percent. This further reduces the stress on mechanisms like content filters (like Intelligent Message Filter or IMF in Exchange Server 2003, and the Content Filter agent in Exchange Server 2007).

Whether you use RBLs to block inbound internet mail from black-listed addresses or not, it makes sense to monitor whether your IP addresses are listed or not on a regular basis.

Read Grimes' column titled "Why I hate RBLs" on InfoWorld.com.

HOW TO: Expose original senders and recipients of quarantined messages

The Content Filter Agent in Exchange Server 2007 allows you to quarantine messages above the SCLQuarantineThreshold. Messages so quarantined are delivered to a quarantine mailbox, specified in the Content Filter configuration. [Read previous post for more info, "Exchange 2007 Content Filter: How to move messages to users' Junk Mail folder"]

Messages arriving in the quarantine mailbox look like NDRs. They appear to be sent from the postmaster address of the default domain. As such, you can't easily see the original sender and recipient's addresses, as shown in the screenshot below:


Figure 1: By default, messages in the quarantine mailbox show the envelope sender— the postmaster address

"How to Recover Quarantined Messages from the Spam Quarantine Mailbox" in the Exchange Server 2007 documentation shows how to use a custom Outlook form to expose the original sender - the doc has code that can be copied to a text editor like Notepad and saved as a custom form (QTNE.CFG), which can be used in Outlook to expose the relevant MAPI property as a field that can be added to the view.

This exposes the original sender, but doesn't expose the original recipient of the message. To expose the recipient, you need to modify the form from the above link and add the relevant MAPI properties.

 Download the modified QTNE.CFG file
Download the modified form file and save it in the correct Forms folder for your locale.

For US English, the path is: \Program Files\Microsoft Office\Office12\Forms\1033. You can find out the locale ID (LCID) for your locale from KB 221435: List of supported locale identifiers in Word..

What was added in my Modified Quarantine Extension Form:
Under [Properties]

Property02=OriginalRecipient

After [Property.OriginalSender] section:

[Property.OriginalRecipient]
Type=30
NmidInteger=0x0E04
DisplayName=Original Recipient

 Install the form in Microsoft Outlook

1. Tools | Options | Other tab | Advanced Options | Custom Forms | Manage Forms | Install.
2. Navigate to the path where you saved the QTNE.CFG file, and select the file.


Figure 2: Installing the Modified Quarantine Extension form (QTNE.cfg) in Microsoft Outlook

3. Click OK a few times till all the dialog boxes are closed. The form is now installed.


 Add the fields
With the form installed in Outlook, it's time to make the fields show up!

1. In the Outlook messages pane, right-click the column heading on the top | select Custom.
2. In the Custom View dialog box, click Fields | from the "Select available fields from:" drop-down (which says "Frequently-used fields" by default), select Forms [screenshot].
3. Select the Modified Quarantine Extension Form | click Add
4. In the Show Fields dialog box, select Original Sender and Original Recipient | click Add.


Figure 3: Adding the Original Sender and Original Recipient fields to the view

5. Additionally, you can remove unnecessary fields and order the fields the way you want [screenshot of fields I have selected].

Once you're done, the Original Sender and Original Recipient fields are displayed as shown in the screenshot below:


Figure 4: The updated view after adding the Original Sender and Original Recipient columns

When the message list pane is expanded in Outlook, this is what it looks like.


Figure 5: Expanding the columns in the updated view

Exposing these fields makes managing the quarantine mailbox easier.

Update: Adding the Original SCL column
08/19/2008 7:39 PM

Exchange 2007 SP1 also exposes the original SCL. The Outlook form file (QTNE.cfg) can be updated by adding the following.

- In the [Properties] section:

Property03=OriginalSCL

- After the [Property.OriginalRecipient] section:

[Property.OriginalSCL]
Type=3
NmidPropset={41F28F13-83F4-4114-A584-EEDB5A6B0BFF}
NmidString=OriginalScl
DisplayName=Original SCL

You can download the updated QTNE.cfg form or manually updated your own file. Now you can add the Original SCL column.


Figure 6: The original SCL for messages in the quarantine mailbox can be displayed using the updated Outlook form.

The previous version of QTNE.cfg is still available.

Exchange Server 2007: Protect users from spam from your own domain

One of the common complaints from users and many messaging folks is spam received from senders that appear to be from your own domain. SMTP mail is exchanged with anonymous internet hosts without any authentication. Headers are effortlessly spoofed. Rather than use an unregistered or invalid domain in the From: field, many spammers find it attractive to use an address from your domain - even an invalid one, or worse - a perfectly valid one. (It is advisable to not have resolution of anonymous senders enabled. Read previous post "A Late New Year's Resolution: Do Not Resolve Anonymous Senders")

Messages that appear to be sent from your domain probably have a better shot at making it through some spam filters, and certainly have a higher chance of being opened by recipients in your domain, once they do make it to the Inbox.

One way of tackling this menace is by using SenderID filter to drop messages that return a FAIL result (from a SPF record lookup). It's advisable to publish SPF records for your domain(s), if you haven't already done so. Explicitly designate SMTP hosts authorized to send for your domain(s). However, the impact of SenderID and whether you should consider dropping email based on it is an open debate, and the topic of another post.

In addition to the above, Exchange Server 2007 allows you to configure ReceiveConnectors to drop messages where the FROM address is from your own domain. It's a good idea to do so if you don't expect any legitimate messages sent without authentication (with your domain in the From: field). To configure a ReceiveConnector to reject such messages:

Remove-ADPermsission "ReceiveConnector-Name" –user “NT AUTHORITY\Anonymous Logon" –extendedrights “ms-Exch-SMTP-Accept-Authoritative-Domain-Sender”

Consider doing this only on the ReceiveConnector that receives inbound internet mail. There may be a need for certain - generally internal - senders like copiers/scanners, and other non-Exchange mail hosts to be able to send with an address from your domain in the FROM field, without authentication. As discussed in previous post titled "Exchange Server 2007: How To Allow Relaying", it is advisable to create an additional ReceiveConnector for such trusted/internal hosts.

Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs

The anti-spam agents in Exchange Server 2007 log their actions in agent logs. By default, agent logs reside in \Exchange Server\TransportRoles\Logs\AgentLog. Each agent log is 10 Mb in size. The size of the directory is capped at 250 Mb, the age of a log to 30 days. (Logs are flushed when they "age" up to 30 days or if the directory size reaches 250 Mb - whichever happens first).

Additional agent log configuration parameters in SP1

Exchange Server 2007 RTM: In the RTM version, there's only one configuration option for the agent log - that to enable or disable it.

Exchange Server 2007 SP1: Additional parameters to control the max directory size, file size and age of agent logs added.

The config options available:
1) Enable/Disable agent log: Boolean value - TRUE/FALSE
2) Max directory size: In bytes. If not specified, the default is 250 Mb or 262144000 bytes.
3) Max file size: If not specified, the default is 10 Mb or 10485760 bytes.
4) Max age: If not specified, the default is 30 days.

The agent log configuration parameters can be controlled by editing the EdgeTransport.exe.config file, located in \Exchange Server\Bin folder on Edge and Hub Transport servers.

To disable agent logging, insert the following key under <appsettings></appsettings> in the config file:

<add key="AgentLogEnabled" value="FALSE" />

In the following example, we modify max directory size to 500 Mb, file size to 20 Mb, and age to 60 days, by creating new keys in EdgeTransport.exe.config:

<add key="AgentLogMaxDirectorySize" value="524288000" />
<add key="AgentLogMaxFileSize" value="20971520" />
<add key="AgentLogMaxAge" value="60.00:00:00" />

Parsing the agent log: You can parse the agent log using the Get-AgentLog command from the shell. By default, this parses the agent log residing in the default location. If you've copied a bunch of agent logs at an alternate location, you can specify the alternate location using the following syntax - in this example the agent logs have been copied to Z:\Antispam Agent Logs directory:

Get-AgentLog -location "Z:\AntiSpam Agent Logs"

Here's what an entry in the agent log looks like - note the different fields and their values:

Timestamp : 4/16/2007 12:39:49 AM
SessionId : 08C948C83FB951AC
IPAddress : 72.46.133.113
MessageId :
P1FromAddress : ret@noncornelan.com
P2FromAddresses : {}
Recipients : {foo@yourdomain.com}
Agent : Connection Filtering Agent
Event : OnRcptCommand
Action : RejectCommand
SmtpResponse : 550 5.7.1 Recipient not authorized, your IP has been found on a block list
Reason : BlockListProvider
ReasonData : Spamhaus SBL-XBL
Diagnostics :

As seen in the above output, the logs provide adequate information for reporting on anti-spam activity, as well as for troubleshooting anti-spam issues like messages not being received/wrongly filtered out.

Getting to know the agent logs will make troubleshooting such issues much easier.

By default, the Get-AgentLog command returns all the entries in the agent logs. It can be constrained to a particular date and time - the recommended way to perform most agent log searches, unless you want to immerse yourself in 30 days (or 250 Mb) of anti-spam goodness! This is done using the -StartDate and -EndDate parameters:

Get-AgentLog -StartDate "4/16/2007" -EndDate "4/17/2007"

You can also constrain it further by adding time of the day:

Get-AgentLog -StartDate "4/17/2007 8:00 AM" -EndDate "4/17/2007 2:00 PM"

Though the Get-AgentLog command only takes these 3 parameters - location, StartDate, and EndDate, you can further filter the logs using most of its logged fields.

To filter the log to show messages to a particular recipient:

Get-AgentLog -StartDate "4/16/2007" -EndDate "4/17/2007" | where {$_.recipients -like "foo@yourdomain.com"}

To search for messages from a particular sender:

Get-AgentLog -StartDate "4/16/2007" -EndDate "4/17/2007" | where {$_.P1FromAddress -like "aqe@easymoney2u.com" -or $_.P2FromAddresses -like "aqe@easymoney2u.com"}

To search for messages from a particular domain:

Get-AgentLog -StartDate "4/16/2007" -EndDate "4/17/2007" | where {$_.P1FromAddress -like "*somedomain.com" -or $_.P2FromAddress -like "*somedomain.com"}

To filter by the anti-spam agent that acted on a message, e.g. Connection Filtering Agent:

Get-AgentLog -StartDate "4/15/2007" -EndDate "4/17/2007" | where {$_.Agent -eq "Connection Filtering Agent"}

Similarly, you can filter by other agents that write to the agent logs: 1) Content Filter Agent 2) SenderID agent 3) Sender Filter agent 4) Recipient Filter agent and 5) Edge Rules agent.

To filter agent logs by the sending host's IP address, use the following command:

Get-AgentLog -StartDate "4/15/2007" -EndDate "4/17/2007" | where {$_.IPAddress -eq "72.46.133.113"}

The reason field in each log entry specifies the reason supplied by the anti-spam agent that takes the action. For instance, as seen in the agent log entry shown earlier in this article, the agent that acted on the message is the Connection Filtering Agent, the reason is BlockListProvider (i.e. "RBL" or "Real-time Block List", known as IP Block Lists in Exchange Server 2007). The ReasonData field gives you the name of the IP Block List Provider, as configured in Exchange. In the above agent log entry, it is "Spamhaus SBL-XBL". To constrain the search for messages blocked by IP Block List Providers:

Get-AgentLog -StartDate "4/15/2007" -EndDate "4/17/2007" | where {$_.Reason -eq "BlockListProvider"}

To get a list of all IP addresses blocked by IP Block List Providers:

Get-AgentLog -StartDate "12/21/2007" | where {$_.Reason -eq "BlockListProvider"} | ft Timestamp,IPAddress,ReasonData

You can also look for messages blocked by a particular IP Block List Povider:

Get-AgentLog -StartDate "4/15/2007" -EndDate "4/17/2007" | where {$_.ReasonData -eq "Spamhaus SBL-XBL"}

For messages scanned by the Content Filter Agent, the Reason field contains details like SCLAtOrAboveDeleteThreshold, SCLAtOrAboveRejectThreshold, etc. The ReasonData field contains the SCL value assigned to the message. To get a list of messages above the SCLDeleteThreshold, use the following command:

Get-AgentLog -StartDate "4/15/2007" -EndDate "4/17/2007" | where {$_.reason -eq "SCLAtOrAboveDeleteThreshold"}

As shown in the above examples, you can use the Get-AgentLog command and pipe the data to filter it based on the fields logged. You can get more details about agent logs - including the fields logged, from the Managing Agent Logging section in Exchange Server 2007 documentation.


2/9/2008: Added sidebar about additional configuration parameters available in SP1.

HOW TO Update multi-valued attributes in PowerShell

Multivalued attributes are simply attributes that can have more than one value. Examples include members attribute of groups, or proxyAddresses attribute of recipients.

In GUI interfaces, it is simple to add/modify multivalued attributes. VBScript has control codes that allow you to control the interaction with multi-valued attributes - 1 = clear all entries, 2 = update all entries, 3 = append, and 4 = delete.

Exchange shell uses Add commands to add new entries to some multivalued attributes - e.g. Add-IPBlockListEntry to add another IP address or range to the IP Block List.

However, if you want to add more values to stuff like BypassedSenderDomains in ContentFilterConfig, there's no equivalent of an Add command. If you try to add a value to the attribute using the following syntax, the existing values will be replaced by the newer ones:

Set-ContentFilterConfig -BypassedSenderDomains "somedomain.com"

You could copy the existing values (e.g. microsoft.com,zenprise.com), and paste these along with the new value:

Set-ContentFilterConfig -BypassedSenderDomains "microsoft.com","zenprise.com","somedomain.com"

However, this copying and pasting can be tricky for attributes that have a lot of values. Another alternative (..thanks to Vivek for the suggestion):

$foo=Get-ContentFilterConfig
$foo.BypassedSenderDomains +="somedomain.com"
$foo | Set-ContentFilterConfig

This is a great workaround... and it works!

However, it'd be much easier to have a switch that lets you append to such multi-valued attributes, in keeping with the shell's promise of allowing most operations using one-liners.. something like Set-ContentFilterConfig -BypassedSenderDomains +domainthree.com,domainfour.com to add values to the attribute, and some other combination to remove a particular value.

Update:
To remove a value from a multivalued attribute, for example from the BypassedSenderDomains property of the ContentFilterConfig used in the preceding example:

$foo=Get-ContentFilterConfig
$foo.BypassedSenderDomains -="somedomain.com"
$foo | Set-ContentFilterConfig

Empower Your PowerShell with PowerGadgets

Exchange MVP Andy Webb showed off this great tool called PowerGadgets recently that can help you build cool gadgets - with or without Windows Vista. The gadgets take piped output from PowerShell commandlets and scripts (as well as SQL databases and web services).

Why I think this is so cool:
- it allows you to visualize data output by PowerShell commands and scripts. The samples shown here are simply displays, but you can create pretty fancy stuff like gauges and charts
- comparatively, it makes PerfMon - which is a useful tool, but so NT 4.0-ish - look like.. well, NT 4.0-ish :)
- critical or important data can be displayed in real-time or close-to-real-time on your desktop or Windows Vista sidebar

Andy created a gadget that takes the output from the get-antispamtoprblproviders.ps1 script [read previous post about this script - "Exchange Server 2007: How are RBLs performing?"]. The gadget runs the script and shows you number of messages blocked by the RBL provider Spamhaus.org (if you're using it with Exchange Server 2007's IP Blocklist Providers feature).

Yet another one of Andy's gadgets shows the Top 20 mailboxes by size. Very cool!

Thanks to input from Andy, and Vivek Sharma, I was able to effortlessly create a gadget that shows messages scanned by Exchange Server 2007's Content Filter agent.

I created another one that shows you number of messages dropped by Recipient Filter using Recipient Validation (drop messages for recipients that do not exist in AD).

Both the above gadgets use real-time output from perfmon counters captured by shell commands and refresh every minute.

You can create gadgets natively from within PowerShell, or use the PowerGadgets Creator GUI that walks you through creating one. Download PowerGadgets from www.powergadgets.com.

Here are the gadgets:
1. Andy's Spamhaus gadget
2. Andy's Top 20 Mailboxes gadget
3. Content Filter gadget - shows total messages scanned by Content Filter agent
4. Recipient Filter gadget - shows messages dropped because of invalid recipients

Update:
Right-click on the above links and save as/download.

If you've created a gadget using PowerShell, spread the word by adding a comment to this post.

Exchange 2007 Content Filter: How to move messages to Junk Mail folder

You've setup Exchange Server 2007, and configured the shiny new Content Filter agent (CFA), which is more than just a rewrite of the equally loved and hated Intelligent Message Filter (IMF) from Exchange Server 2003. The CFA has three thresholds - equivalent of Gateway thresholds and (gateway) actions in IMF. These are: 1) SCLDeleteThreshold 2) SCLRejectThreshold and 3) SCLQuarantineThreshold.

1. SCLDeleteThreshold: Messages with SCL equal to or higher than the SCLDeleteThreshold are deleted silently. To enable the SCLDeleteThreshold:

set-ContentFilterConfig -SCLDeleteThreshold 8 -SCLDeleteEnabled:$true

2. SCLRejectThreshold: Messages with SCL equal to or higher than the SCLRejectThreshold are rejected during the SMTP session, after the data is received. In this case, senders get a NDR. To enable the SCLRejectThreshold:

set-ContentFilterConfig -SCLRejectThreshold 7 -SCLRejectEnabled:$true

In the above case, Exchange doesn't accept the message. After the data is received, it responds with a 500 5.7.1 error and a rejection response (by default this response is: Message rejected due to content restrictions. This rejection message can be configured using the following command (response message used here is for illustration, not a real suggestion - Bharat) :

set-ContentFilterConfig -RejectionResponse "Stop spamming you *****!"

The actual NDR is generated and sent to the sender by the sending host. What the sending host will see after the message content is sent (and if you actually modified the rejection response based on my example :)
500 5.7.1 Stop spamming you *****!


Figure 1: Exchange Server 2007's Content Filtering agent can be configured with the equivalent of all 3 Gateway actions available in Exchange Server 2003's Intelligent Message Filter (IMF): 1) Delete messages 2) Reject messages and 4) Quarantine messages

3. SCLQuarantineThreshold: Messages with SCL equal to or higher than the SCLQuarantineThreshold are delivered to the quarantine mailbox, provided you have one configured. To enable the SCLQuarantineThreshold and configure a quarantine mailbox:

set-ContentFilterConfig -SCLQuarantineThreshold 6 -SCLQuarantineEnabled:$true -QuarantineMailbox:MyQuarantineMailbox@mydomain.com

This is an improvement over IMF, which had only one gateway action (and one corresponding gateway threshold). The Content Filter agent allows the flexibility of enabling all three actions on the gateway - the rule is: SCLDeleteThreshold > SCLRejectThreshold > SCLQuarantineThreshold.

To get a list of all three SCL values and whether each action is enabled or not, use the following command:

get-ContentFilterConfig | Select SCL*

So where's the equivalent of IMF's Store threshold? (to move messages to users' Junk Mail folders)?

It's called SCLJunkThreshold, and it resides in a different location - in the Organization configuration. It can be set using the set-OrganizationConfig command:

set-OrganizationConfig -SCLJunkThreshold 5

Setting SCLJunkThreshold not intuitive?

Before you jump to conclusions about this being counter-intuitive, or confusing - which it may be, consider this - it is in response to the different server roles in Exchange Server 2007.

The gateway actions - delete, reject and quarantine - can be thought of as message transport actions, and thus applicable to transport server roles (the Edge server, or the Hub if you have antispam agents enabled on it). Moving messages to users' Junk Mail folders can be thought of as something that happens at the Store, performed by the Mailbox Server role.

Another aspect to consider when setting the SCLJunkThreshold - if you're in a topology with an Edge server, the SCLJunkThreshold on the Edge doesn't impact anything. This needs to be set on your Exchange Org - the Edge server is not a part of it.

The Defaults: By default, the SCLJunkThreshold is set to 4. If you have an existing Exchange Server 2003 SP2 server installed, and you haven't tweaked the Store threshold, IMF v2's default Store threshold of 8 is used - this is what you'll see in the Org's SCLJunkThreshold. Given that the SCLRejectThreshold is set to 7 by default, messages will not move to users' Junk Mail folder unless the SCLJunkThreshold is lower than the transport thresholds (i.e. the Delete, Reject and Quarantine thresholds).

How the Junk Mail threshold is calculated: Unlike the transport actions of deleting, rejecting, and quarantining messages - which check for SCL equal to or higher than their respective thresholds, for moving messages to Junk Mail folder the Store checks for SCLs higher than the SCLJunkThreshold. This is consistent with the behavior of IMF in Exchange Server 2003 (read previous post: IMF Confusion - Store threshold rating text in UI).

If you want to disable rejection of messages with SCL of 7 or above, use the following command:

set-ContentFilterConfig -SCLRejectEnabled:$false

Related Posts:
- Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs
- Moving items to Junk Mail folder doesn't do much

Exchange Server 2007: Making SenderID work with non-Exchange smtp hosts

To make the SenderID filter work with non-Exchange smtp hosts acting as your mail gateways for inbound mail, you need to tell Exchange about them. If Exchange does not know about these, SenderID will not be able to determine the correct sending host, and you may see SenderID checks fail on all inbound mail.

This may have been a source of confusion for users in the past - which hosts do you consider as "internal" - the ones on your internal network? or only those located in perimeter networks (aka "DMZ")? how about those located at your ISP?

Exchange needs to know about all smtp hosts that do not belong to senders - the ones that will handle your inbound mail, whether at your ISP or in your perimeter or internal networks, before messages are delivered to the Exchange Org. (Start by taking a look at MX records in external DNS zones - do these point to non-Exchange servers? If yes, do these hosts route mail to Exchange servers?).

In Exchange Server 2003 this is done by entering internal smtp server addresses in Global Settings | Message Delivery properties | General tab | Perimeter IP List and internal IP Range Configuration.


Figure 1: Setting up internal or perimeter SMTP hosts in Exchange Server 2003's PerimeterIPList

On Exchange Server 2007, you can configure this from the shell:

set-transportconfig -InternalSMTPServers 1.2.3.4,1.2.3.5,1.2.3.6

Updates:
8/16/2007: SP1 Beta2 allows setting InternalSMTPServers parameter using the Exchange console, as shown below:


Figure 2: Setting up InternalSMTPServers using the Exchange console in Exchange Server 2007's new Global Settings tab | Transport Settings | properties | Message Delivery tab

Exchange 2007 Content FIlter: The Whitelist Is Here!

Messaging Hygiene features in Exchange Server 2003, including the Intelligent Message Filter (IMF), did not have a way to whitelist sending domains or SMTP addresses.

This is a follow up to a previous post, and one of the more popular ones on this blog— "IMF: Where's the whitelist?". ("IMF and whitelist" has for long been one of the most common search terms on this blog - Bharat).

Whitelists are common in most 3rd-party anti-spam tools. Adding domains or SMTP addresses of important senders like customers, vendors, or your CEO's home email address (almost always an AOL address... :) for instance, ensures messages from these domains or addresses do not get filtered by the anti-spam filter.

Bypassed Senders and Sender Domains: The Whitelist
The good news is— Exchange Server 2007's shiny new Content Filter Agent (or IMF v3 if you will) has whitelists! You can add SMTP addresses and domains to the Content Filter configuration, and have messages from these senders and domains bypass the Content Filter Agent. However, you need to resort to the Exchange shell (EMS) to manage it.

Use the following shell command to add sender SMTP addresses to the BypassedSenders list:

Set-ContentFilterConfig -BypassedSenders foo@somedomain.com,foo2@somedomain.com

Use the following command to whitelist the sending domain:

Set-ContentFilterConfig -BypassedSenderDomains somedomain.com,someotherdomain.com

Some whitelisting considerations
Before you start using whitelists, here are a few things you should consider:

• SMTP headers can be spoofed easily. If spammers spoof any of the addresses or domains you whitelist, your recipients may end up getting more spam as all of it will bypass the Content Filter.
  
• Use SenderID Filtering to detect and protect your mail system from header spoofing.
  
• Maintaining whitelists, just as maintaining blacklists, is a manual process that imposes its own management costs.
  
• Checking every inbound message against a list of whitelisted recipients imposes a performance penalty - miniscule as it may be. Use the whitelists sparingly.

Nevertheless, many IMF users have repeatedly demanded this functionality and it's great to finally have it in what some folks call IMF v3.0.

Bypassed Recipients: The Exception List
The Content Filter can also be configured with an exception list - to not apply the filter to inbound messages for particular recipients. This can be done from the console by going to Hub Transport | Anti-spam tab | Content Filtering -> properties | Exceptions. This list is limited to a 100 recipients - you can add generic recipients that you want to exempt from the Content Filter, such as sales@yourdomain.com, info@yourdomain.com, etc.

To add recipients to the exception list using the Exchange shell:

Set-ContentFilterConfig -BypassedRecipients sales@yourdomain.com,info@yourdomain.com

Related Posts:

• BypassedSenders, BypassedSenderDomains, and BypassedRecipients are multivalued attributes. The following post shows how to modify multivalued attributes:
  HOW TO Update multi-valued attributes in PowerShell
  
• Exchange 2007 Content Filter: How to move messages to Junk Mail folder
  
• HOW TO: Install anti-spam agents on Hub Transport server
  
• Quick antispam report or status check?
  
• HOW TO: Expose original senders and recipients of quarantined messages

IMF: Updates not showing up in Add/Remove Programs?

Some users report that recent updates to Microsoft's Intelligent Message Filter (for Exchange Server 2003) do not show up in Add/Remove Programs in Windows. Double-checking on some servers, I've noticed this - the last update that shows up on those in Add/Remove Programs is dated 8/10/2006.

If you look at the registry key for this update (all IMF Updates show up as update for KBA 907747) under HKLM\Software\Microsoft\Windows\CurrentVersion\Uninstall\KB907747, the DisplayVersion value - which is my best guess at what should show up in Add/Remove Programs or an indicator of which update was installed last - shows 2006.11.30. This was the update prior to the current one that was released in January.

One way to determine the version of IMF update currently installed on your system is to go to the \exchsrvr\bin\MSCFV2 directory (\exchsrvr is the install path for Exchange server), and look at the sub-folder that starts with the IMF update version - e.g. 6.5.xxx.x - with the latest date. The following screenshot shows the latest update that was downloaded on 1/18.

IMF: Updates live again

This just came in from Scott Roberts - IMF updates are live again. The latest update is dated Jan 11, 2007, version 6.5.7891.0.



Given the questions in different forums since the updates were paused/stopped, and particularly between yesterday (when the updates were supposed to kick in) and today - are some users really hooked to IMF? Or is it the warm & fuzzy feeling of seeing IMF being updated that makes you feel more secure or protected, just like updated anti-virus signatures? :)

If IMF doesn't update automatically on your servers, you can always check whether new updates are available by going to Microsoft Update (as shown in above screenshot). Alternatively, you can use the script provided by Scott in the post titled Demystifying Exchange Server 2003 SP2 IMF Updates on the Exchange team blog. The script also comes in handy if you do not want to use the auto-update settings to download and install *all* high-priority updates from Microsoft Update - you can schedule the script to run every other week, and have it only download and install IMF updates.

IMF: Updates stopped in December, to resume on Jan. 17th

A lot of users wondered why there were no updates to Microsoft's Intelligent Messages Filter (IMF) in recent weeks. Many Exchange administrators wondered if it was a problem with their servers when they saw IMF hadn't been updating automatically - some even posted on different Exchange forums wanting to compare the versions on their servers. (IMF updates are normally available twice a month).

Scott Roberts throws some light on this. According to his post on WSUS blog, IMF updates were blocked in December 2006. These will restart on Jan. 17th. The last published update is the one from Nov. 30th last year.

Perhaps this has something to do with Exchange Server 2007 being RTMed in early December.

I am surprised why there wasn't more information about this, and not as many users complained. Perhaps not as many users noticed, because there's no mechanism to nag you for older definitions like most anti-virus/anti-spyware agents do.

Either way, it would be great to hear about such issues from the Exchange team blog (rather than first being reported on the WSUS blog) a little earlier in the lifecycle of such incidents.

While we're still on the subject of IMF updates, it would be great to see an update to IMF that either allows you to keep the Custom Weighting file - MSExchange.UceContentFilter.xml - in the \exchsrvr\bin\MSCFV2 folder (rather than having to move/copy it to the new sub-folder created every time IMF updates are applied), or allows location of the file to be specified in a registry value.

Note: The Custom Weighting file is used to control SCL values assigned to messages using keywords found in message subject or body. Not all users use the Custom Weighting feature, introduced in Exchange Server 2003 Service Pack 2. More details in SP2 Release Notes.

IMF: How to customize NDR message

Do you have IMF setup to reject messages at the gateway? If yes, you can customize the NDR message senders get. Exchange MVP and author of Exchange Server 2003 24Seven (renamed in its second edition to Microsoft Exchange Server 2003 Advanced Administration) Jim McBee shows you how in his post titled "Exchange 2003 SP2 IMF tuning".

IMF: Where's the whitelist?

Another frequently asked question - asked frequently enough to make its way here.

Does the Intelligent Message Filter (IMF) v1/v2 in Exchange 2003 provide any way of "whitelisting" sending SMTP addresses or domains? (I will continue to use whitelisting as one word. Although it cannot be found in the dictionary, it is a common term if you're reading this blog and work with messaging/anti-spam - Bharat)

The Sender Filtering feature provides a way to blacklist SMTP addresses and domains, so it's logical to expect a way to whitelist addresses and domains as well. However, there's no whitelisting feature in Exchange 2003 - or at least not out-of-the-box.

From the explanations I've heard, the one that comes across as the most reasonable: SMTP headers (and therefore the FROM: field in messages) can be easily spoofed. To a certain extent, it's hard to argue with that - it is true, and that is perhaps one of the biggest reasons we have to deal with spam and phishing today.

Some workarounds for whitelisting
In absence of a whitelist feature built into IMF, here's what can be done to ensure messages from trusted sources are not blocked:

 Global Accept List: Adding the IP address(es) of sending hosts to Connection Filtering's Global Accept List allows messages from the sending host to bypass Connection Filtering — inbound messages will be accepted even if the sending host is listed on a RBL, and IMF as well. To add IP addresses of sending hosts to the Global Accept list using the ESM:
1. Expand Global Settings | select Message Delivery -> Properties | Select the Connection Filtering tab


Figure 1: Connection Filtering's Global Accept List allows you to whitelist IP addresses of sending hosts

2. Click Accept | Click Add
3. To add a single IP address, type the address in the IP address field. To add a range of IP addresses, select Group of IP Addresses and type the subnet address and mask.
4. Click OK (thrice) to close the dialog boxes.

Next, we need to enable Connection Filtering in SMTP virtual server properties.
5. Select the SMTP Virtual Server that receives inbound internet mail | select Properties | General tab | Advanced button next to IP address
6. Select an IP address (you can enable/disable filters by IP address)
7. Click the Edit button
8. Check Apply Connection Filter


Figure 2: Antispam filters can be selectively enabled or disabled in SMTP Virtual Server properties

By whitelisting sending hosts' IP address(es), you're telling Exchange the sending host is trusted. Arguably, this is more effective and secure than whitelisting SMTP addresses and domains. (Also read Alexander Zammit's post regarding issues with having the same IP address on Global Accept list and on the local IP list).

Nevertheless, some folks would simply like to have the convenience of whitelisting addresses and domains. Almost every other anti-spam tool has it. (Update: Exchange 2007 has whitelisting. See Exchange 2007 Content FIlter: The Whitelist Is Here!)

 Custom Weighting: If the messages you want to whitelist contain particular keywords in the subject or message body, you can use the Custom Weighting feature as described in Exchange Server 2003 SP2 Release Notes. This allows you to lower or raise the SCL values assigned by IMF.

This entails creating an XML file called MSExchange.UceContentFilter.xml- the syntax is described in the Release Notes. You can simply cut and paste the sample from the Release Notes and modify it - it is fairly easy to do.

This approach could have been made a little more effective by simply allowing to look for the keywords in message headers as well. Not sure why headers were not included in the scanning mechanism for Custom Weighting - hopefully I will come across a reasonable explanation for this as well. :)

 IMF Tune: If the above methods don't work for you, and you absolutely need the convenience of being able to whitelist addresses and domains, you can look at third party utilities like IMF Tune from WinDeveloper. IMF Tune is an inexpensive tool that adds important functionality to IMF, making it closer to a full-blown anti-spam tool that many want IMF to be.

Besides whitelisting, IMF Tune allows you to configure a quarantine mailbox (instead of IMF's option to archive mail in the UceArchive folder), setup autoreplies for filtered mail, strip attachments, amongst other things.

Another likable IMF Tune feature is the ability to insert SCL values in message headers - which IMF will do for archived messages, but not for messages delivered to mailboxes. The SCL value is a MAPI property of a message that can be made to show up in Outlook [read 'Exposing SCL (Spam Confidence Level) in Outlook' on Exchange team blog], but the procedure isn't something most users would want to go through.

Having said this, I have always liked IMF and think Microsoft did a great job by including this - for free - as a web download with IMF v1, and as a part of Exchange Server 2003 SP2 with IMF v2. It serves a useful purpose in many small/SMB deployments where it meets the requirements to a good extent or where resource constraints rule out buying a third-party anti-spam product.

Besides, much as product managers want, not all features can be implemented in any given version, and given historical data, by version 3 these features/products become much more mature. (Think of the anti-spam features in Exchange Server 2007 as version 3 of Exchange's messaging hygiene features :).

Update: Manually removing IMF v1

Scott Schnoll has an update regarding the recent post Intelligent Message Filter (IMF) v1 download removed from microsoft.com, and the issue of not being able to remove IMF v1 from Add/Remove Programs (if one doesn't remember which account was used to install it or the account is no longer available) without reinstalling it. Here's the process to manually remove IMF v1 as documented in Microsoft Exchange Server Intelligent Message Filter v2 Operations Guide [available here].

To manually remove Intelligent Message Filter v1:
1. Stop all Exchange services (Information Store, System Attendant, SMTP, and Exchange-aware antivirus services).
2. Rename the MSCFV1 folder in X:\Program Files\Exchsrvr\bin, where X is the drive letter where Exchange Server is installed.
3. Rename the ContentFilter.dll file in \Exchsrvr\bin\
4. Open Regedit and export the following key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange
5. Delete the ControlFilterVersion subkey from the registry.
6. Restart the server. You should now be able to install Exchange Server 2003 SP2 without errors.

Intelligent Message Filter (IMF) v1 download removed from microsoft.com

A newsgroup poster recently informed that Microsoft has removed Intelligent Message Filter (IMF) v1 download from the IMF downloads page.

IMF v1 was a separate download. IMF (v2) is a part of Exchange Server 2003 SP2.

Excerpt from IMF v1 Release Notes [still available here] regarding uninstalling IMF:
To uninstall Intelligent Message Filter, use Add or Remove Programs in Control Panel. The account that was used to install Intelligent Message Filter must also be used to uninstall the filter. No other accounts will see Intelligent Message Filter in Add or Remove Programs.

It was (and continues to be to a certain extent) a common issue with IMF v2 installation: user is somehow able to install Exchange Server 2003 SP2 without uninstalling - or being able to uninstall - IMF v1. At times this happens because the user does not know which account was used to install IMF v1, or because the account no longer exists. As a result, IMF v1 does not show up in Add/Remove Programs.

In such cases, re-installing IMF v1 and then uninstalling it works. However, with IMF v1 now removed from the Microsoft web site, this will become a problem.

Perhaps it is justified to remove the IMF v1 download from the web site - to avoid scenarios where users inadvertently download and install it. However, with the download gone from the web site, you will now need to call Microsoft PSS to get it.

I haven't been able to verify with anyone at Microsoft if PSS will in fact provide a copy of IMF v1 when users are faced with this scenario, or if a workaround exists.

HOW TO: Install anti-spam agents on Hub Transport server

If you're using Exchange Server 2007 in a topology with an Edge Transport server, the anti-spam agents - Connection Filtering, Content Filter, Sender ID, Sender Filter, Recipient Filter, and Protocol Analysis - are installed on the Edge server. You do not need these agents installed on the Hub server(s).

In environments without an Edge server role, these need to be installed manually on the Hub Transport server(s).

To install anti-spam agents on a Hub Transport server, start Exchange Management Shell (EMS), navigate to the "\Exchange Server\Scripts" folder (by default C:\Program Files\Microsoft\Exchange Server\Scripts), and run:

.\install-antispamagents.ps1

Once installed successfully, the command will show each agent installed, whether it is enabled or not, and its priority [screenshot].

After the anti-spam agents are installed, you can see the Anti-spam tab in Organization Configuration | Hub Transport node in Exchange Management Console. (If your console was open during installation, you will need to restart it).

In such deployments, you can either route inbound mail through your service provider/ISP, or configure Hub Transport servers to receive inbound email directly. Read previous post "Exchange Server 2007: Internet email without Edge servers".

Related posts:
- Exchange Server 2007: Internet email without Edge servers
- Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs
- Exchange 2007 Content FIlter: The Whitelist Is Here!
- Exchange 2007 Content Filter: How to move messages to Junk Mail folder
- Exchange Server 2007: How are RBLs performing?

IMF: Gateway thresholds per server

So far we've known that Exchange Server 2003's content filter - the Intelligent Message Filter (IMF), uses Gateway and Store thresholds and (Gateway) action that are configured once in Global Settings | Message Delivery | properties | Intelligent Message Filtering. These global settings apply to all SMTP Virtual Servers on all Exchange Servers in the Org that have IMF enabled.


Figure 1: Configuring Intelligent Message Filter's Gateway and Store thresholds for the entire Organization

Evan Dodds reveals the ability to configure per-server Gateway threshold and action for IMF in "Configure IMF gateway settings per server".

To enable the per-server Gateway thresholds, two registry values need to be created as follows:
Registry Key: HKLM\Software\Microsoft\Exchange\ContentFilter



Value: GatewayThreshold
Value type: REG_DWORD
Value data: 0-9

Value: GatewayAction
Value type: REG_DWORD
Value Data:
0x0 = No Action
0x1 = Delete
0x2 = Reject
0x5 = Archive

The above method allows you to control the Gateway threshold and action per server. You cannot control the Store threshold per server.

So the answer is, yes, it's possible using this undocumented registry hack, and Evan also suggests when one may need to use it.

Exchange Server 2007 exposes SCL in message headers

Exchange Server 2003's Intelligent Message Filter (IMF) assigns a SCL— a value that indicates the probability of a message being spam - to every message it scans. SCL is assigned based on factors such as message content, headers, and IP addresses.

By default, Exchange Server and Microsoft Outlook do not expose this SCL value assigned to a message - it's neither visible in any of Outlook's message views, nor inserted in message headers. It's a MAPI property of a message.

If IMF's Gateway action is set to Archive, Exchange allows you to save the SCL in message headers of archived messages. This is done using a registry hack mentioned in a previous post "IMF: Archiving spam".

You can also expose the SCL property as an additional column in Microsoft Outlook— after going through a series of steps documented in Exposing SCL (Spam Confidence Level) in Outlook on the Exchange team blog. You can then use that view in OWA as well.

Exchange Server 2007 inserts detailed anti-spam stamp in message headers. This includes both SCL and PCL values, SenderID lookup details such as the Purported Responsible Domain (PRD), SenderID status.

Here are antispam headers inserted by Exchange 2007:

X-MS-Exchange-Organization-PRD: alertsif.com
X-MS-Exchange-Organization-SenderIdResult: Pass
Received-SPF: Pass (MyServer.MyDomain.com: domain of
=?iso-8859-1?B?QWNhaVB1cmU=?=@alertsif.com designates 72.46.28.29 as
permitted sender) receiver=MyServer.MyDomain.com; client-ip=72.46.28.29;
helo=mx10.alertsif.com;
X-MS-Exchange-Organization-SCL: 4
X-MS-Exchange-Organization-PCL: 2
X-MS-Exchange-Organization-Antispam-Report:
DV:3.3.5705.600;SV:3.3.4604.600;SID:SenderIDStatus Pass;OrigIP:72.46.28.29

If you are a heavy OWA user, there's good news! Message headers can now be checked in OWA as well— although you need to open a message to check headers. There's no option to view full message headers in the preview pane by default, or a button to toggle between simple and full message headers that's common in some web-based email services such as Yahoo!'s.


Figure 1: Exchange 2007 inserts antispam stamp in message headers. You can also view message headers in OWA 2007.

IMF: Archiving spam

As explained in previous posts, the Intelligent Message Filter (IMF) offers a few configuration options. The primary interface for configuring IMF is under Global Settings | Message Delivery properties | Intelligent Message Filtering tab.


Figure 1: IMF configuration options in Global Settings | Message Delivery | Properties | Intelligent Message Filtering tab

Here you specify a Gateway Blocking Configuration by telling Exchange what the Gateway SCL threshold is, and selecting an action to be taken when the threshold is met or exceeded. IMF assigns each message scanned a rating called "Spam Confidence Level" or SCL. Messages with SCL values that meet or exceed the Gateway threshold can be blocked from users' mailboxes.

One of the following actions can be selected to treat such messages:

•  Do Nothing: This simply forwards the message to the Store, and depending on the Store configuration, typically the messages should end up in a user's Junk Mail folder. Important consideration: the Store threshold should be *lower* than the gateway threshold.
  
•  Reject: Exchange rejects the message during the SMTP session. The sending server is responsible for generating any Non-Delivery Report (NDR) to the sender.
  
•  Delete: Exchange accepts the message and deletes it, without notifying the sending host and without generating a NDR. This is also referred to as silent delete.
  
•  Archive: When the Archive action is selected, Exchange delivers the message in the UCEArchive folder.

Some considerations and FAQs about archiving messages - most of these are documented in the IMF v2 Deployment Guide:

1. Where are the messages archived?
Messages are archived in the UCEArchive folder. By default this folder is created in \exchsrvr\mailroot\vsi 1 folder - where 1 is the instance number of SMTP virtual server.

2. Is there a way to change the location of the UCEArchive folder?
Yes, this is done by inserting a registry value of type String (REG_SZ) in the following location:
HKLM\Software\Microsoft\Exchange\ContentFilter
Value: ArchiveDir
Type: REG_SZ (string)
Value data: path to archive folder

3. I set the action to Archive. Why don't I see the UCEArchive folder in the default location?
The UCEArchive folder is created when the first message meeting or exceeding the gateway threshold is received.

Things to check when you don't see the UCEArchive folder:
- Is the action set to Archive?
- Is IMF enabled on SMTP virtual server?
Read the earlier post on changes to IMF v2 introduced in Exchange Server 2003 Service Pack 2 "Exchange Server 2003 SP2 and IMF"
- Is IMF working? (Check performance counters)
Read the earlier post "Troubleshooting IMF: Important Performance Counter" for more info.
- Are you receiving any messages meeting or exceeding the gateway threshold?
Check performance counters (again!) - MSExchange Intelligent Message Filter\Total Messages Assigned an SCL Rating of (number) - where number is the SCL rating. Counters are available for SCLs 0-9.
- Is the archive folder location modified using the above registry value?

4. I see a bunch of files with .eml extension in the UCEArchive folders. How do I view them?
You can open these .eml files with Outlook Express or using Notepad.

5. Managing archive files using the above method (opening each .eml file in Outlook Express or using Notepad) isn't very efficient. Is there a better way?
There's a popular third-party tool called IMF Archive Manager that makes managing the archived messages easier. You can download it for free from gotdotnet.com.

6. I don't have the time or inclination to sort through so many archived messages. Should I just set it to delete or reject?
If you're just starting out with IMF, I would recommend archiving initially. This will allow you to review the archive folder to see if there are any false positives - messages that are not spam, but got a higher SCL rating for some reason. Archiving will also allow you to finetune your IMF configuration based on the number of false positives you get using the gateway SCL threshold you use. Once you are comfortable with what you see in the archived messages, you can choose to reject or delete messages if you want.

7. I don't see the SCL rating anywhere in the archived messages. Is there a way to save the SCL with the messages?
Yes, by adding the following registry value, IMF can be configured to save the SCL assigned to archived messages. Exchange inserts the X-SCL x-header.
Location: HKLM\Software\Microsoft\Exchange\ContentFilter
Value: ArchiveSCL
Type: DWORD
Value data: 1

IMF does not scan messages larger than 3 Mb

Microsoft has just released a knowledgebase article - KB 907691 - documenting the inability of its Intelligent Message Filter (IMF) - the anti-spam filter that was earlier offered as a free add-on for Exchange Server 2003 - to scan messages over 3 Mb. in size.

When IMF encounters a message that is larger than 3 Mb., it logs the following event in the Application Event Log:

Event ID: 7515
Category: SMTP Protocol
Source: MSExchangeTransport
Type: Error Message : An error occurred while Microsoft Exchange Intelligent Message Filter attempted to filter a message with ID <ID_Number>, Priority_Number From smtp:E-mail_ID and Subject Subject_Text. This message will not be filtered. The error code is 0x800710f0.

(In some of my tests IMF has behaved differently when scanning these large (over 3 Mb.) messages with and without attachments. I haven't been able to complete testing to come to any conclusion because of lack of time - will update this post if I find something - Bharat)

IMF became part of Exchange Server 2003 when Microsoft included it in Exchange Server 2003 Service Pack 2. The inability to scan messages over 3 Mb. in size is common to both IMF v1 and the new version 2 included with SP2.

The article states this is by design - typical spam messages are not as large, and it costs more to send such spam.

It'll be interesting to find out what Exchange administrators think about this!

IMF: How to exclude recipients from IMF filtering

Microsoft recently released a hotfix for Intelligent Message Filter that adds functionality to exclude certain recipients from IMF filtering using an exclusive list, or only apply IMF to certain users using an inclusive list. The latter should be great for limited testing in a production environment.

You do need to call Microsoft PSS to get the hotfix mentioned in KBA 912587. This updates MSGFILTER.DLL to version 6.5.7650.22.

With the hotfix applied, you need to fire up regedit and
- create a new key called ContentFilter in HKLM\Software\Microsoft\Exchange.
- create a new DWORD value called CheckRecipients
- if you want to use an exclusive list - which means messages IMF will bypass (not scan) the message if ALL recipients in the message are on the list, modify CheckRecipients DWORD value you just created and type 2 in Value data
- Create a new Multi-String value called RecipList - and add names of recipients you want to exclude from IMF filtering.

Again, the emphasis is on all recipients - if there's a single recipient of a message not on the exclusion list, it will get scanned by IMF.

To do the opposite - create an inclusive list - change the value of CheckRecipients to 1, and add names of recipients who would like to get their messages filtered by IMF to the RecipList value.

It is not required to restart the server after you apply this hotfix.

Note: Calling PSS for required hotfixes that resolve particular issues is free. Just have the KBA number handy!

IMF Confusion - Store threshold rating text in UI

A major source of confusion regarding the Store threshold value used by Intelligent Message Filter (IMF) is the IMF UI itself. This is documented in Intelligent Message Filter release notes.


Figure 1: The text for the Store SCL incorrectly states that messages with SCL greater than or equal to the Store threshold will be moved to Junk Mail

What it says: In ESM | Global Settings | Message Delivery | properties | Intelligent Message Filtering tab, the Store Junk E-mail Configuration section reads Move messages with an SCL rating greater than or equal to. If you never read the Release Notes - and I know many of us don't, particularly in smaller environments where IMF is more popular - you follow what the UI says and assume setting a store SCL fo 6 would makes messages with a SCL of 6 or higherto be moved to the Junk Mail folder.

What it does: The Release Notes clarify this - the UI should say "Move messages with an SCL rating greater than". Only messages with a higher SCL than that value are moved to Junk Mail.

Apparently, Exchange Server 2003 SP2 did not fix this either and SP2 Release Notes make no mention of this. For many Exchange environments that did not use IMF v1 - a separate download - SP2 is their first encounter with IMF.

Related Posts:
- TechEd 2007: Notes from the BOF Session on Fighting Spam With Exchange Server 2007
- Exchange Server 2003 SP2 and IMF
- Troubleshooting IMF: Important performance counter
- IMF: Archiving spam
- IMF: Where's the whitelist?
- IMF does not scan messages larger than 3 Mb
- IMF: Updates not showing up in Add/Remove Programs?
- Intelligent Message Filter (IMF) v1 download removed from microsoft.com
- Update: Manually removing IMF v1
- IMF: How to exclude recipients from IMF filtering
- IMF: Gateway thresholds per server

Troubleshooting IMF: Important performance counter

Earlier today I saw some strange behavior with Intelligent Message Filter (IMF) v2. Incoming internet mail wasn't getting scanned by IMF. Outlook did not show the SCL values for any message!

That in itself wasn't the strange part. I'm positive I have the right gateway and store thresholds, and I have IMF enabled on the SMTP Virtual Server - I double-checked that as soon as I saw the issue.

Next stop - check IMF performance counters. A word about monitoring SMTP (and IMF in particular) with Perfmon— the default view is a graph, and if you're dealing with a server that doesn't get much traffic (the server in question being my primary Exchange server on the home network - it doesn't get hundreds of messages on a Sunday night), switching to the Report view and looking at the numbers makes it much easier to detect any activity. This could potentially save a lot of time, as it often does for me.

I quickly added the following counters of MSExchange Intelligent Message Filter object:

• 1. Total Messages Scanned for UCE
  
• 2. Messages Scanned for UCE/sec
  
• 3. %UCE of Total Messages Scanned
  
• 4. %UCE of Messages Scanned in the previous 30 minutes

Also added the following from SMTP Server object:

• 1. Messages Received Total
  
• 2. Messages Received/sec

With the Report view fired up, I started sending messages - some with all the characteristics of UCE, and some simple test messages. Here's where the strange behavior comes in - the SMTP Messages Received counters were rising, and surprisingly so were Total Messages Scanned for UCE. However, Outlook still didn't show any SCL values and all the obvious spam was still hitting my Inbox!

Here's where another important IMF counter comes in - Total UCE Messages Acted Upon (and its sibling - UCE Messages Acted Upon/sec). Once these were added to the Perfmon console, I spent a few minutes sending some more messages. The UCE Messages Acted Upon counters did not move at all, whereas Total Messages Scanned was inching higher.

IMF was scanning, but not acting on those messages!

Next stop (and don't ask me why this wasn't the first thing I checked :) - Event Logs. The familiar Event ID 7514 that I was trying to reproduce for another test earlier in the day showed up in the Application Event Log! This is an error logged by MSExchangeTransport. Description: An error occurred while loading Microsoft Exchange Intelligent Message Filter. The error code is 0x80004005.

The error was caused by saving the MSExchange.UceContentFilter.xml file as text. This file is used by the new Custom Weighting feature introduced with IMF v2 in SP2. The file needs to be saved with encoding set to Unicode.


Figure 1: The Custom Weighting file MSExchangeUceContentFilter.xml should be saved in Unicode format

Opened file in notepad, changed to Unicode and saved. Restarted SMTPSvc. I sent some more test messages - now I could instantly see the messages acted upon counter rising. Spam started showing up in the Junk Mail folder and Outlook started showing the SCL values, as expected.

Exchange Server 2003 SP2 and IMF

Before you install Exchange Server 2003 SP2 you need to uninstall Intelligent Message Filter (IMF). When you do this, the settings from the SMTP virtual server where you allowed filtering are also removed.

After SP installation if you go to Intelligent Message Filtering settings (Global Settings -> Message Delivery -> Properties -> Intelligent Message Filtering tab) and setup IMF with same settings as earlier, that alone does not reactivate IMF.

You still need to enable IMF from the SMTP virtual server properties -> General tab -> Advanced Edit (see screenshot, Flash demo of IMFv1 and IMFv2 settings - not entirely intuitive location for these settings, imo) and check "Apply Intelligent Message Filter" - just as you need to check the relevant options for recipient filter, connection filter, sender filter, and the new option of applying SenderID filter.