Exchange Server 2007: Managing And Filtering Anti-Spam Agent Logs

by Bharat Suneja

Exchange 2007 includes a number of anti-spam agents to filter spam. The anti-spam agents log their actions in (anti-spam) agent logs. The default agent log locations:

  • Exchange 2010: \Exchange Server\V14\TransportRoles\Logs\AgentLog
  • Exchange 2007: \Exchange Server\TransportRoles\Logs\AgentLog

Agent Log Configuration

You can’t change the agent log location. Here are the available config options:

  1. Enable/Disable agent log: On transport servers with anti-spam agents installed, agent logging is enabled by default. You can disable it by adding the AgentLogEnabled key, and setting it to FALSE.
  2. Max file size: By default, the maximum file size of each agent log is 10 Mb (10485760 bytes). You can modify it by adding AgentLogMaxFileSize key and providing the desired value (in bytes). Parsing very large log files can be slower— the default max file size works for most deployments.
  3. Max directory size:: By default, the size of the directory is capped at 250 Mb (262144000 bytes). You can modify it by adding the AgentLogMaxDirectorySize key and providing the desired value (in bytes). In a high volume environment, you must consider the volume of log files generated in a day, and determine the space required based on how many days’ worth of log files you want to retain for troubleshooting or to meet your organization’s retention requirements.
  4. Agent log age: The age of a log is capped at 30 days. You can modify it by adding the AgentLogMaxAge key and providing the desired value. Logs are flushed when they’re older than the AgentLogMaxAge, or if the directory size reaches the AgentLogMaxDirectorySize (250 Mb by default), whichever happens first. If it’s important to maintain agent logs for a certain number of days, you must consider the volume of logs generated daily, and raise the AgentLogMaxDirectorySize if required.

Exchange 2007 RTM has only one configuration option for the agent log – that to enable or disable it. Exchange Server 2007 SP1 includes the additional configuration parameters listed above to control the max directory size, file size and age of agent logs added.

The agent log configuration parameters can be controlled by editing the EdgeTransport.exe.config file, located in \Exchange Server\Bin folder on Edge and Hub Transport servers.

Note, the key names in EdgeTransport.exe.config are case-sensitive.

Disable agent logging

To disable agent logging, insert the following key under <appsettings></appsettings> in the config file:

<add key=”AgentLogEnabled” value=”FALSE” />

Modify agent log configuration

In the following example, we modify max directory size to 500 Mb, file size to 20 Mb, and age to 60 days, by creating new keys in EdgeTransport.exe.config:

<add key=”AgentLogMaxDirectorySize” value=”524288000″ />
<add key=”AgentLogMaxFileSize” value=”20971520″ />
<add key=”AgentLogMaxAge” value=”60.00:00:00″ />

Parsing the agent log

You can parse the agent log using the Get-AgentLog command from the shell.

By default, the Get-AgentLog cmdlet parses agent logs in the default location (\Exchange Server\TransportRoles\Logs\AgentLog). Depending on your troubleshooting and log retention requirements, you can move older files to an another location. To parse agent logs in an alternate location, you must specify the path. In this example, the agent logs have been copied to Z:\Antispam Agent Logs directory:

Get-AgentLog -location “Z:\AntiSpam Agent Logs”

Agent log format and fields

Agent log file names are named AGENTLOGyyyymmdd-nnnn.log. It’s easy to decipher yyyy stands for the year, mm is the month, and dd is the date. The nnnn is a serial number, starting at 0001, rolled over when more than one log is generated in a day. The date is UTC.

Each agent log starts with the following header fields:

#Software: Microsoft Exchange Server
#Version: 8.0.0.0
#Log-type: Agent Log
#Date: 2010-08-07T00:00:01.500Z
#Fields: Timestamp,SessionId,LocalEndpoint,RemoteEndpoint,EnteredOrgFromIP,
MessageId,P1FromAddress,P2FromAddresses,Recipient,NumRecipients,
Agent,Event,Action,SmtpResponse,Reason,ReasonData,Diagnostics

Here’s what an entry in the agent log looks like, as output by the Get-AgentLog cmdlet. Note, not all fields are displayed by default when you use the cmdlet:

Timestamp : 4/16/2007 12:39:49 AM
SessionId : 08C948C83FB951AC
IPAddress : 72.46.133.113
MessageId :
P1FromAddress : [email protected]
P2FromAddresses : {}
Recipients : {[email protected]}
Agent : Connection Filtering Agent
Event : OnRcptCommand
Action : RejectCommand
SmtpResponse : 550 5.7.1 Recipient not authorized, your IP has been found on a block list
Reason : BlockListProvider
ReasonData : Spamhaus SBL-XBL
Diagnostics :

As seen in the above output, the logs provide adequate information for reporting on anti-spam activity, as well as for troubleshooting anti-spam issues like messages not being received/wrongly filtered out.

Getting to know the agent logs will make troubleshooting such issues much easier.

Searching agent logs

By default, the Get-AgentLog command returns all the entries in the agent logs. This can take a long time and results in all entries scrolling by quickly in your shell window. The only filtering options available with the Get-AgentLog cmdlet are start and end dates.

Filter by date and time

Agent logs can be constrained to a particular date and time – the recommended way to perform most agent log searches, unless you want to immerse yourself in 30 days (or 250 Mb) of anti-spam goodness! This is done using the -StartDate and -EndDate parameters, as shown in this example:

Get-AgentLog -StartDate “4/16/2007” -EndDate “4/17/2007”

You can also constrain it further by adding time of the day:

Get-AgentLog -StartDate “4/17/2007 8:00 AM” -EndDate “4/17/2007 2:00 PM”

Search messages by recipients and senders

Although the Get-AgentLog command only takes these 3 parameters – location, StartDate, and EndDate, you can further filter the logs using most of its logged fields by using pipelining.

To filter the log to show messages to a particular recipient:

Get-AgentLog -StartDate “4/16/2007” -EndDate “4/17/2007” | where {$_.recipients -like “[email protected]”}

To search for messages from a particular sender:

Get-AgentLog -StartDate “4/16/2007” -EndDate “4/17/2007” | where {$_.P1FromAddress -like “[email protected]” -or $_.P2FromAddresses -like “[email protected]”}

Search messages from a sender domain

To search for messages from a particular domain:

Get-AgentLog -StartDate “4/16/2007” -EndDate “4/17/2007” | where {$_.P1FromAddress -like “*somedomain.com” -or $_.P2FromAddress -like “*somedomain.com”}

Search messages filtered by a specified anti-spam agent

To filter by the anti-spam agent that acted on a message, e.g. Connection Filtering Agent:

Get-AgentLog -StartDate “4/15/2007” -EndDate “4/17/2007” | where {$_.Agent -eq “Connection Filtering Agent”}

Similarly, you can filter by other agents that write to the agent logs: 1) Content Filter Agent 2) SenderID agent 3) Sender Filter agent 4) Recipient Filter agent and 5) Edge Rules agent.

Searching by IP address

To filter agent logs by the sending host’s IP address, use the following command:

Get-AgentLog -StartDate “4/15/2007” -EndDate “4/17/2007” | where {$_.IPAddress -eq “72.46.133.113”}

Search messages blocked by RBLs

The reason field in each log entry specifies the reason supplied by the anti-spam agent that takes the action. For instance, as seen in the agent log entry shown earlier in this article, the agent that acted on the message is the Connection Filtering Agent, the reason is BlockListProvider (i.e. “RBL” or “Real-time Block List”, known as IP Block Lists in Exchange Server 2007). The ReasonData field gives you the name of the IP Block List Provider, as configured in Exchange. In the above agent log entry, it is “Spamhaus SBL-XBL”. To constrain the search for messages blocked by IP Block List Providers:

Get-AgentLog -StartDate “4/15/2007” -EndDate “4/17/2007” | where {$_.Reason -eq “BlockListProvider”}

To get a list of all IP addresses blocked by IP Block List Providers:

Get-AgentLog -StartDate “12/21/2007” | where {$_.Reason -eq “BlockListProvider”} | ft Timestamp,IPAddress,ReasonData

You can also look for messages blocked by a particular IP Block List Povider:

Get-AgentLog -StartDate “4/15/2007” -EndDate “4/17/2007” | where {$_.ReasonData -eq “Spamhaus SBL-XBL”}

Search by SCL thresholds

For messages scanned by the Content Filter Agent, the Reason field contains details like SCLAtOrAboveDeleteThreshold, SCLAtOrAboveRejectThreshold, etc. The ReasonData field contains the SCL value assigned to the message. To get a list of messages above the SCLDeleteThreshold, use the following command:

Get-AgentLog -StartDate “4/15/2007” -EndDate “4/17/2007” | where {$_.reason -eq “SCLAtOrAboveDeleteThreshold”}

As shown in the above examples, you can use the Get-AgentLog command and pipe the data to filter it based on the fields logged. You can get more details about agent logs – including the fields logged, from the Managing Agent Logging section in Exchange Server 2007 documentation.

Updates
2/9/2008: Added sidebar about additional configuration parameters available in SP1.

{ 18 comments… read them below or add one }

[email protected] April 17, 2007 at 10:14 pm

Hi,

please, do you know, what does it mean “LocalBlockList” as Reason in Get-AgentLog cmdlet result?

Reply

Bharat Suneja April 18, 2007 at 9:44 am

You’ll see the Reason field with value “LocalBlockList” if the Connection Filtering Agent blocks a message from a host that’s listed on the IP Block List. The ReasonData field tells you if the IP was listed manually by the admin, or if the Sender Reputation agent added it.

Bharat

Reply

Chris Mace February 8, 2008 at 12:24 pm

IS there a way to increase the size of the log folder for the Agent logs? 250mb only gives me about 2.5 days worth of historical data and thats just not enough.

Thanks
Chris Mace

Reply

Bharat Suneja February 8, 2008 at 1:08 pm

No. From the above post:
There’s only one configuration option for the agent log – that to enable or disable it.

Reply

Bharat Suneja February 8, 2008 at 2:57 pm

Chris,

Update to my previous response: it seems additional configuration parameters were indeed added in Exchange Server 2007 SP1. The post has been modified to reflect that.

Thanks for asking this – prompted me to look, and correct the post. :)

Thanks to the helpful folks in the Exchange team for pointing me in the right direction: How to Manage Agent Log Output.

Reply

yuvika September 30, 2008 at 3:26 am

HI ,
could you pls tel me where can I find a complete list of all the values that can populate the “reason” field in the agent logs? Am a log analyst and this info is realy crucial for my analysis.
Thanks
Yuvika

Reply

Anonymous February 19, 2009 at 1:46 am

Hi,

I have changed the AgentLogMaxDirectorySize and restarted the Transport service but this has no effect. Even after rebooting the server the log dir cannot grow larger than 250M.
I followed all steps in http://technet.microsoft.com/en-us/library/bb691337.aspx but no good.

Reply

Anonymous May 4, 2009 at 1:44 am

Hi,

why do i find entries in the log that look like this:

2009-04-02T00:00:54.046Z,,,,,,,,[email protected],,,,,,,,

Filtering gets quite difficult ….

Thanks

Reply

kaveh June 15, 2009 at 7:10 pm

Hi, great blog, I've learned a lot from it. I have a question if you have a moment to help me out.

When I run Get-AgentLog, most of the time the IPAddress I see is an internal IP, 192.168.2.2, which is the internal IP of my ISA server. Does this mean that the IP Block list isn't able to see the actual IP that the email is coming from? How do I check the real IP?

Here's a sample:

Timestamp : 6/15/2009 11:00:59 AM
SessionId : 08CBB9AFB27BF9C7
IPAddress : 192.168.2.2
MessageId : [email protected]
P1FromAddress : [email protected]
P2FromAddresses : {[email protected]}
Recipients : {[email protected]}
Agent : Content Filter Agent
Event : OnEndOfData
Action : RejectMessage
SmtpResponse : 550 5.7.1 Message rejected as spam by Content Filtering.
Reason : SclAtOrAboveRejectThreshold
ReasonData : 8
Diagnostics : DV:3.3.7813.600;SV:3.3.7815.344

Reply

kaveh June 22, 2009 at 8:49 am

I figured it out. My ISA servers are set to replace the SMTP IP in the headers to it's own IP so that the Hub Transport talks to the external server through ISA. Thus Exchange never sees the original IP and can't filter against it.

I'll probably have to put another box in front of the ISA servers to do IP filtering based on Blocklists or put something on the ISA servers to do the filtering there.

But I noticed that when I look through the logs some DO have the external IP. But that is only maybe 100 messages a day which is like like 0.01% of incoming mail… I wonder how these few messages manage to keep their external IP in their headers…

Reply

armin September 29, 2009 at 10:27 am

I have errors too mutch for normal perason :D

Reply

Exchange server October 5, 2009 at 8:04 am

Finaly my server is working for some days great
Thanks a lot..

Reply

Anormalen March 30, 2010 at 6:43 am

I have a very strange problem. The agent log just doesn't exist. When I use "get-agentlog" command I recive an error: "C:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\AgentLog" does not contain any logs for the specified timerange. The directory is empty, but AntiSpam filters are intalled. I'm using Exchange 2007 on a sigle server without Edge Transport.

Reply

Markus Becker December 13, 2010 at 11:39 am

Hello Anormalen
have you found a solution?
I have the same problem
I will beat you out of ideas

THX

Reply

Bharat Suneja March 30, 2010 at 6:50 am

@Anormalen: Ensure agents are enabled (config and perfmon counters) and logging is enabled.

Reply

Anormalen March 30, 2010 at 7:03 am

I have enabled logging in EdgeTransport.exe.config file. Agents/anti spam fiters are also enabled, they appear in the Hub Transport. Everything is made according to this article, and the server is working. Just the agent logs are missing. I've configured IP blocklist providers that are tested and work as they should.

Reply

Mike June 18, 2013 at 11:20 am

Have you verified the permissions on the default directory? It seems unlikely but it’s possible the permissions are not right. Make sure whatever account your Transport service is running in has permission to that directory.

Reply

Nilesh Kumar June 20, 2010 at 2:27 am

Anormalen – Did you ever find out why the AgentLog folder was missing, I have the same issue in Exchange 2010. SPAM filtering is enabled on the Hub but no AgentLog folder.

Reply

Leave a Comment

{ 6 trackbacks }

Previous post:

Next post: