Troubleshooting Intra-Org SMTP Traffic Issues (and disabling Cisco ASA ESMTP Inspection)

I had a particular issue that is not well documented on the Technet site so I decided I would blog about it and share my experience.

Today I was doing an Exchange 2003 to 2010 upgrade for a customer. Their AD setup was quite typical, two Active Directory sites under a single forest interconnected by a Cisco ASA in each site doing IPSEC. They wanted an Active / Active configuration (one site will be doing SMTP and CAS traffic, which is NY and the FL site will be doing just CAS traffic) which was fine. Here comes the problems.

We tested mail flow from the NY site both external (inbound/outbound) to a user in the NY site along with the Exchange 2003 setup. All worked without a problem. When we tried to send an internal or external message to a user within the FL site, it would get stuck in the messaging queue with the status of “Retry”.

The first thing when troubleshooting intra org SMTP traffic is to review the Receive connectors. By default, you would have two…

  • Client <servername>
  • Default <servername>

The default receive connector is used for SMTP traffic, and is listening over TCP 25. The client receive connector is used for ESMTP, as it is listening over TCP 587.

You should look at several things:

1)      What are the subnets allowed within the Network’s tab on my Default <servername> connector? By default, you should be allowing all subnets to utilize this connector to send message to this server.

Image

2)      What is my authentication set to for this receive connector? I have seen this one a lot, and where people think it’s an awesome idea to remove all types of authentication. Its not, don’t do it. By default is what you see below, and should be kept this way. If you need to change this, I would recommend making a new SMTP connector with the values desired

Image

3)      Who is actually allowed to use this connector? The values for the defaults are below (Exchange 2010 SP3) but should be kept alone. If you want to select “Anonymous” I would recommend setting up a new Receive connector and setting up the network section to allow only the servers that would need to relay off of this server. This prevents your Exchange 2010 servers from being a generic SMTP relay out in the interwebs for anyone to use and abuse.

Image

Since we are troubleshooting Exchange 2010 intra-org SMTP connectivity, we want to make sure that we keep the Exchange servers checked within the Authentication and Permissions group tabs. If this is unchecked, check it off and restart the “Microsoft Exchange Transport Service” (anything changed in here should have a restart of this service).

If this is fine, we should move onto actually checking if port 25 is available between the two sites. You can check this by running the following from the command line..

telnet <Name of the HUB server in the other site) 25

The response should look similar to below

Image

If we are able to telnet to this without a problem, then check out the other ports listed here. You should have these also open and available to for intra org SMTP communications and also for the HUB Transport services to work properly.

The best troubleshooting in my opinion would be to turn up the IntraOrgProtocolLoggingLevel to Verbose. This should be done on all HUB transport servers when troubleshooting, and can be done like this:

      Get-TransportServer | Set-TransportServer –IntraOrgProtocolLoggingLevel Verbose

Once this is done, attempt to send another test message. Give it about 5 minutes and then check the logs from the sending HUB transport server. These logs can be located (by default) at %PROGRAMFILES%\Microsoft\Exchange Server\V14\TransportRoles\Logs\ProtocolLogs\SmtpSend

From there review the errors. There are usually typically a handful that can be there, but the one distinct one I saw today was “cannot achieve Exchange server authentication – failed”.

When reviewing further I noticed the following error within the queue viewer:

“451 5.7.3 – Cannot achieve Exchange Server authentication”

The Cisco ASA does protocol level filtering, the 250 STARTTLS reply is not established and dropped. To get IntraOrg SMTP Traffic to work, we would need to turn off the ESMTP inspection which can be done as shown below (thank you Lazy Network Admin for these steps below)

      CiscoASA# config t 
      CiscoASA(config)# policy-map global_policy 
      CiscoASA(config-pmap)# class inspection_default 
      CiscoASA(config-pmap-c)# no inspect esmtp 
      CiscoASA(config-pmap-c)# exit 
      CiscoASA(config-pmap)# exit 
      CiscoASA(config)# exit 
      CiscoASA# wr me

Once this is done retest, and you should be receiving emails from the other site. If you did turn on the intra organization protocol logging, I would recommend turning it off.

       Get-TransportServer | Set-TransportServer –IntraOrgProtocolLoggingLevel None


Now, these are not the only troubleshooting steps in the world. If you continue to have issues with intra-organization mail delivery I would also check within the Exchange Toolbox the routing configuration, and run through the Mail Flow Troubleshooting wizard.

Hope this is helpful to all! Any questions please let me know below.

Adam F

Exchange Proxy and Redirection (Exchange 2007 and 2010) Explained

There has been a lot of discussion recently around proxy/redirection with Exchange 2007/2010 so I wanted to clear the air about what each one does, and how they work with CAS services.

What is the real definition of Proxying and Redirection?

To understand proxying and redirection, you should understand the meaning of both. Later in this article I will cover when/why proxy or redirection happens.

Proxying occurs when a CAS role sends traffic to another CAS role in particular situations (read below for more about that). For example here are two common situations:

  • CAS to CAS communication between two AD sites
  • CAS to CAS communication between Exchange 2010 and 2007 or 2003

Redirection occurs when there are two AD sites with Exchange 2010 / 2007 / 2003 and both are internet facing.

When do I proxy or redirect a CAS connection?

This is a little more complex. The following CAS protocols / services are proxy enabled:

  • Outlook Web App (OWA) and Exchange Control Panel (ECP)
  • Exchange ActiveSync (EAS)
  • Exchange Web Services (EWS) and the availability service (part of EWS)
  • POP3 / IMAP

And then the only three services that will actually redirect are OWA/ECP and EAS.

On to the fun part..

When do I actually proxy my connections from one AD site to another?

Simple, and we will use OWA as the example.

Adam’s mailbox is in the EU AD site, which has an ExternalURL set to $NULL (no value) for OWA but the InternalURL is set to euemail.exchangelaboratory.com. My internet facing site is the US AD Site, and we have an ExternalURL/InternalURL of naemail.exchangelaboratory.com.

Since OWA does have the capability of Proxy and Redirection, when the initial authentication request goes to the UAG or CAS (depending on your deployment) exchange is looking for three items from the GCS in a LDAP/RPC request, which is

  • My homeMDB (mailbox location)
  • My msExchangeVersion (the version of Exchange the user mailbox is on)
  • Authentication Token to fulfill my request

If it notices that my homeMDB is in another site, then it will query that CAS for the InternalURL and ExternalURL of the services in question (in this case, OWA). Since the ExternalURL is set to $NULL in the EU AD Site CAS, then the only option we have will be to proxy.

What occurs at this point is that the request gets proxied from the source server (cashub1.exchangelaboratory.com) to the target server (cashub2.exchangelaboratory.com), the CASHUB2 server will contact the Mailbox server for the appropriate data from the user’s mailbox and then proxy the connection back to the source server, which then delivers the content to the user. Beautiful thing.

Image

Now for redirection:

In this situation, we now have an “Active / Active” configuration as in both the US AD Site and the EU AD Site are internet enabled. My ExternalURL for OWA in the EU AD Site is euemail.exchangelaboratory.com, and my ExternalURL for OWA in the US AD Site is naemail.exchangelaboratory.com. We will use Adam again as our example mailbox.

When Adam attempts to authenticate to the URL naemail.exchangelaboratory.com the Exchange CAS (or UAG depending on your configuration) will perform the auth request along with the Mailbox Location and Version lookup. Exchange will realize that Adam’s mailbox is actually in the EU AD Site, and will send a HTTP 451 request back to the client to redirect to the proper internet facing URL (in this situation, euemail.exchangelaboratory.com). Once this is complete, it will either be a silent redirection or the user will have to authenticate again.

Image

ActiveSync and OWA Virtual Directory permissions to make redirection and proxying to work?

To make sure we can successfully redirect Exchange ActiveSync and OWA requests, we need to ensure that the proper permissions are setup on the IIS Virtual Directories.

Exchange 2003 (non-internet facing) and Exchange 2010:

In this situation, our endpoint for client access is Exchange 2010 as this is internet facing and the most up to date version of Exchange. The Exchange 2003 site is not internet facing, so the Exchange ActiveSync protocol would have to proxy from Exchange 2010 to 2003 for any users within the Exchange 2003 environment.

To allow this to work, you should enable IWA (Integrated Windows Auth) on the IIS Virtual Directory Microsoft-Server-ActiveSync on the Exchange 2003 server. This should not be done through the IIS Manager, as the System Attendant in Exchange 2003 will overwrite these changes and break proxying. You can do this through the steps available here.

Exchange 2007 (non-internet facing) to Exchange 2007

In this situation, we have two AD sites both running Exchange 2007. One site is internet facing and the other is not, thus if a user is within the non-internet facing site we have to proxy the connection from the internet facing CAS to the non-internet facing CAS.

To allow this to work we need to enable IWA on the non-internet facing Exchange 2007 CAS. To do this we will need to go through IIS Manager, and the steps can be found here.

Single Sign On for OWA – Exchange 2010 SP2?

A new feature the Exchange Team at Microsoft implemented in Exchange 2010 is called silent redirection. I would recommend setting this up, the reason for this is because you now eliminate the need for a double authentication when doing a redirection within OWA on Exchange 2010 SP2.

This does get a little lengthy, and there is an excellent blog from the Exchange Product Group located here.

 

Well, that’s all I have for now. Leave your questions below if you have any. Thanks for reading!

Adam F