Fixing Information Store Error -2147221213 (ecTimeSkew) – Exchange 2007 / 2010 / 2013

(Late) Sunday nights are not a fun time to get paged about a customer down…

Situation

Engineer reboots an Exchange 2010 server that is apart of a two node DAG, but when the server comes back online the MS Exchange Information Store service is stuck in the “Starting” phase within the Services MMC, and the Microsoft Exchange System Attendant will just not start. All other services are online. What gives?

Environment

Another pretty simple buildout, Exchange 2010 SP3UR6 running on top of Server 2008 R2 with all patches (security / hotfixes) that are available. Both servers are multiple-role (CAS/HUB/Mailbox) which also has a CAS array and a two node DAG. Two copies of each database in the environment.

Troubleshooting

The first thing I did was confirm that the only two services that were boinked at the time were the Information Store and System Attendant. Best way to actually check this is by running Test-ServiceHealth if you are running Exchange 2010 / 2013, which showed exactly that.

When I looked in the Services MMC I did see the Exchange Information Store stuck in the starting state. I looked in the Event Viewer: Application logs to find this error:

Log Name:      Application
Source:        MSExchangeIS
Date:          8/24/2014 11:51:29 PM
Event ID:      5003
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EXCH01.company.com
Description:
Unable to initialize the Information Store service because the clocks on the client and server are skewed. This may be caused by a time change either on the client or on the server, and may require a restart of that computer. Verify that your domain is correctly configured and  is currently online.

The first thing I did was make sure the Windows Time service was properly pulling the right time source..

w32tm /query /source

Since that looked proper (it was the PDC) I moved on and physically checked the time between the two Exchange servers, which looked fine. Confused, I decided to try a force sync of the Windows Time service..

w32tm /resync

Afterwards, performed a reboot and avail.. nothing. Back to square one with the databases showing a state of “Failed” and but this time the Information Store service did not get stuck in “starting”, it was stopped all together. I attempted to start the service with the following error message:

“Unable to start service. Error -2147221213. Please contact your support provider or the vendor”

This is a hex error. Luckily enough if you know the internals a bit to Exchange, sometimes you can look up these errors using the Err.exe tool MSFT provides (link).

That then shows us this..

# for decimal -2147221213 / hex 0x80040123 :
  ecTimeSkew                                                    ec.h
  MAPI_E_INVALID_ACCESS_TIME                                    mapicode.h

We now know its a windows time issue, so now what?

Resolution

There are two routes you can go with this:

– Manually set the Windows Time service to sync off the PDC (which is what I did)

– Completely reset the Windows Time service

Manually set Windows Time service

This is pretty easy, and all you have to do is run this command

net time \\PDCName /Set

It will ask you if you want to confirm, click Y and then confirm that the time is now synced. Afterwards, restart the MS Exchange Active Directory Topology service which will restart all the other Exchange services, and hopefully its fixed for you.

If that doesn’t work try resetting Windows Time all together, and then set it to point to the PDC (sometimes required)..

Reset the Windows Time service

Open up a command prompt and then each of the commands below (each line break is a new command to run)..

net stop w32time
w32tm /unregister
w32tm /register
net start w32time

Once this is done, you should check the actual provider to make sure it is either the DC that is hosting the PDC FSMO role (recommended) or another domain controller that is local to Exchange (e.g: within the same AD site)

w32tm /query /source

If it is NOT the DC holding the PDC FSMO role or even a DC (it might show as “CMOS”, which means your BIOS) then I would set it to either one of the options I said above..

net time \\PDCName /Set

At this point restart the MS Exchange Active Directory Topology service and make sure everything starts.

Once either one of these steps are done, you should be fully working again.

 

TL;DR: Windows Time sync is required for domain controller authentication on a domain joined machine, along with authentication requests. If windows time breaks (or Windows thinks its broken when the time is actually synced), now you know how to fix it!

Leave any questions and comments below.

 

Cheers!

 

– Adam F

 

 

 

 

 

 

 

Advertisements

Exchange 2010 / 2013: Address Book Service, EMC SourceOne and E_MAPI_FAILURES

So I was recently tasked with a pretty interesting case escalation this week, which was in regards to an address book sync failure from a third party application (EMC SourceOne).

Environment

For the sake of this blog, the environment was pretty simple. A single Exchange 2010 SP3UR5 CAS / HUB combination with an Exchange 2010 SP3UR5 Mailbox role. No HA on the DAG and no CAS Array. The EMC SourceOne server is in the same AD site. Multiple global catalog servers and domain controllers utilizing Kerberos.

The Issue

This was a fun one, mainly due to the lack of information provided. What I understood is that the EMC SourceOne application would utilize a local MAPI profile (Control Panel > Mail (32)) to connect to the Exchange role over MAPI / TCP, then start the Address Book Sync through the CAS to the Address Book Service. This is similar to the way RIM BlackBerry Enterprise Server does it (well.. I can speak for BES 4 / 5), and also how both applications do further lookups.

When the EMC SourceOne server would reach out to the CAS’s Address Book Service it would fail out with the error E_MAPI_FAILURE. Not very helpful, since that would be a few hundred things.. lets dive into troubleshooting.

Troubleshooting

The first thing that ran into my head was “is this being throttled”? If you don’t know much about client access throttling, I would recommend reading up on it. It does change a bit in Exchange 2013, but I can write a blog about this later if there is interest. The best way to check throttling is to go directly to the Exchange server, open the Event Viewer: Application logs and filter them for “ADACCESS Event ID 2915”, which is client throttling. You can look at the SID~ within the log to see which account is being throttled..

The Event ID looks something like this:

ID: 2915
Level: Error
Source: MSExchange ADAccess
Message: Process Microsoft.Exchange.RpcClientAccess.Service.exe (PID=5372). User 'Sid~DOMAIN\SERVICEACCOUNT~RCA~false' has gone over budget '263' times for component 'RCA' within a one minute period. Info: 'Policy:[Fallback], Parts:MaxConcurrency:262;'. Threshold value: '100'.

Ironically, NOBODY went other throttling as per the Event Viewer: Application logs this month. Cool, lets move on.

If it’s not client throttling, then what could it be.. it could be NTLM token bloat if NTLM was actually enabled, but they are using Kerberos so that’s not the case. The mailbox does exist for the service account, and the permissions are setup properly. The next logical step is to take a deeper look at the address book service logs (which are located by default at C:\Program Files\Microsoft\Exchange Server\V14\Logging\Address Book Service\)

Looking at the logs using this wonderful tool (TextAnalysisTool.net – really nifty tool everyone should use if you want an easy to use log searching tool) and filtering for the service account (SourceOne-Service) I found the following..

10666 2014-08-20T17:18:04.826Z,165558,1221,/o=COMPANY/ou=First Administrative Group/cn=Recipients/cn=SourceOne-Service,,10.12.4.178,COMPANYMBX1,ncacn_ip_tcp,DNToEph,80004005,0,,Throttled,,15000
10689 2014-08-20T17:18:20.201Z,165558,1222,/o=COMPANY/ou=First Administrative Group/cn=Recipients/cn=SourceOne-Service,,10.12.4.178,COMPANYMBX1,ncacn_ip_tcp,ResolveNames,80004005,0,,Throttled,,15000
10743 2014-08-20T17:18:35.108Z,165558,1223,/o=COMPANY/ou=First Administrative Group/cn=Recipients/cn=SourceOne-Service,,10.12.4.178,COMPANYMBX1,ncacn_ip_tcp,Unbind,,0,60395,,,14906

So what you see above is within the Address Book Service logs that during the address book sync the service account for SourceOne was throttled during an NSPI call to the GAL. Afterwards, that connection was unbound since it went over its budget and sub-sequentially dropped, which would have caused the E_MAPI_FAILURE error mid-sync.

Now that we actually have the error, and know the issue how do we fix this? (thank you for asking kind voice, let me explain that part)

The Fix

This one had me scratching my head at first until I looked at my notes. Originally in Exchange 2010 RTM the Address Book Service would throttle the amount of connections concurrent connections to 50 maximum, and any new connection (when it hit its maximum of 50) would be dropped. This setting use to exist within the exchange-addressbook.service.exe.config file, but was move in Exchange 2010 SP1+ to the registry along to the Client Access throttling policies (RPC Client Access, also known as RCA). What confused me originally is that the CAS does NOT show any ADACCESSS Event ID 2915 errors at all for this account, but I will digress.

To fix this, you should actually create a new throttling policy and either remove the RCA limits or up them to the limit you know you are hitting (I would speak to the vendor to figure out the desired limit, they are the best resource).

1. Create a new throttling policy

The creation of the throttling policy is pretty simple. All you have to do is open Exchange Management Shell (as an account with Org Admin or Org Management rights) and run the following:

New-ThrottlingPolicy "EMC SourceOne"

 2. Once the throttling policy is created, you would need to set the RCA* values within it to something higher than the defaults or simply remove them.

In my scenario, I know the EMC SourceOne application is going to only run the sync over a few minutes in the AM, so I chose to remove the RCA* values completely (e.g: setting them to $NULL)

Set-ThrottlingPolicy "EMC SourceOne" -RCAPercentTimeInAD $null -RCAMaxConcurrency $null -RCAPercentTimeInCAS $null -RCAPercentTimeInMailboxRPC $null

 You then can review the policy to make sure the RCA* values within the throttling policy are shown as blank (meaning there is nothing actually set)

Get-ThrottlingPolicy -Identity "EMC SourceOne" | Select RCA*

3. Apply the throttling policy to the service account

Set-Mailbox -Identity SourceOne-Service@company.com -ThrottlingPolicy "EMC SourceOne"

You can then view the mailbox settings itself to make sure the throttling policy change from the default (or whatever you had previously) to the EMC SourceOne policy..

Get-Mailbox -Identity SourceOne-Service@company.com | Select ThrottlingPolicy

4. Test and review the ABS (Address Book Service) logs. It should not work.

TL;DR: for some reason the address book service throttling does not show up within the actual Event Viewer: Application log under Event ID 2915 when the service account goes over budget. To fix this create a new throttling policy and then up (or remove) the limit for the RPC Client Access (RCA*) attributes. I do not recommend completely removing the limit unless you are sure it will NOT affect production in a negative way, so contact the vendor for there best practices (in this case, EMC Source One would be EMC).

Any other questions, comments or anything like that drop them below in the comments.

Cheers!

– Adam F

Exchange 2010, /PrepareSchema and a Cryptic Error

(note: TL;DR at the bottom in bold)

 

Its 11PM after a long day and I am here trying to rebuild my lab. I sit back, copy over the VHD templates for the VM I need (Exchange 2010 is what I was up too) and got all the prerequisites installed for Exchange 2010. I can get pretty lazy (remember this part, as this will bite me in the rear in a minute) and honestly I find PowerShell to be the lazy mans answer. I used the following to install the required prereqs for Exchange 2010 SP3 on Windows Server 2012 (works for 2012 R2)…

 

Add-WindowsFeature NET-Framework-Features,NET-HTTP-Activation,RPC-over-HTTP-proxy,RSAT-Clustering,Web-Mgmt-Console,WAS-Process-Model,Web-Asp-Net,Web-Basic-Auth,Web-Client-Auth,Web-Digest-Auth,Web-Dir-Browsing,Web-Dyn-Compression,Web-Http-Errors,Web-Http-Logging,Web-Http-Redirect,Web-Http-Tracing,Web-ISAPI-Ext,Web-ISAPI-Filter,Web-Lgcy-Mgmt-Console,Web-Metabase,Web-Net-Ext,Web-Request-Monitor,Web-Server,Web-Static-Content,Web-Windows-Auth,Web-WMI -Restart

Still awake? Good.. lets move on to the actual reason of me writing this..

So I am now at the point of preparing the schema and active directory. As a helpful reminder you will first prepare the schema and then active directory, which has not changed since Exchange 2007. You will also be required to use a user account that is in the following groups:

  • Schema Admins
  • Domain Admins
  • Enterprise Admins

A little piece of hidden advise for users who are doing this in a multi-site production / lab build, is that if you are preparing the schema or extending active directory the schema master FSMO role needs to be in the AD site you are actually preparing.

I download Exchange 2010 SP3 onto my brand new Server 2012 R2 machine which is domain joined, extract the package and run the “Setup.com /PrepareSchema” to get the following error thrown back in my face…

SchemaFailure

 

The snipplet is small due to the size of my monitor but you can click on it to expand it. The error code is also copied below..

 

The following error was generated when "$error.Clear(); install-ExchangeSchema -LdapFileName ($roleInstallPath + "Setup\Data\"+$RoleSchemaPrefix + "schema0.ldf")" was run: "The system could not find the file specific".

*sigh* .. Really? I went ahead and bing’ed and google’ed parts of these errors which came back with nothing. Great.. further examination of the logs (C:\ExchangeSetupLogs\) showed the following..

 

ExchangeSetup.log

[08/11/2014 02:59:42.0175] [1] Executing: 
    install-ExchangeSchema -LdapFileName ($roleInstallPath + "Setup\Data\"+$RoleSchemaPrefix + "schema0.ldf")


[08/11/2014 02:59:42.0175] [2] Active Directory session settings for 'Install-ExchangeSchema' are: View Entire Forest: 'True', Configuration Domain Controller: 'DC-01.extest.com', Preferred Global Catalog: 'DC-01.extest.com', Preferred Domain Controllers: '{ DC-01.extest.com }'
[08/11/2014 02:59:42.0175] [2] Beginning processing install-ExchangeSchema -LdapFileName:'Setup\Data\PostWindows2003_schema0.ldf'
[08/11/2014 02:59:42.0175] [2] Running <C:\Windows\system32\ldifde.exe> with arguments <-i -s "DC-01.extest.com" -f "C:\Windows\Temp\ExchangeSetup\Setup\Data\PostWindows2003_schema0.ldf" -j "C:\Users\administrator.EXTEST\AppData\Local\Temp\1" -c "<SchemaContainerDN>" "CN=Schema,CN=Configuration,DC=extest,DC=com">.
[08/11/2014 02:59:42.0175] [2] [WARNING] An unexpected error has occurred and a Watson dump is being generated: The system cannot find the file specified
[08/11/2014 02:59:42.0175] [2] [ERROR] The system cannot find the file specified

 

ExchangeSetupWatson.log

[08/11/2014 02:58:43.0366] [1] The following 1 error(s) occurred during task execution:
[08/11/2014 02:58:43.0366] [1] 0.  ErrorRecord: The system cannot find the file specified
[08/11/2014 02:58:43.0366] [1] 0.  ErrorRecord: System.ComponentModel.Win32Exception: The system cannot find the file specified
   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
   at Microsoft.Exchange.Management.Deployment.InstallExchangeSchema.RunProcess(String fileName, String arguments, WriteVerboseDelegate writeVerbose)
   at Microsoft.Exchange.Management.Deployment.InstallExchangeSchema.ImportSchemaFile(String schemaMasterServer, String schemaFilePath, String macroName, String macroValue, WriteVerboseDelegate writeVerbose)
   at Microsoft.Exchange.Management.Deployment.InstallExchangeSchema.InternalProcessRecord()
   at Microsoft.Exchange.Configuration.Tasks.Task.ProcessRecord()

 

Reading the error provided above was a bit much. I know its not permissions since I was using the “Administrator” account as its a brand new domain (as I just created it) and it could not be replication as its a single DC. I went back and then checked the actual prerequisite script I ran, and realized that it did not include the RSAT-ADDS toolset. Because these tools are not installed, and I am not preparing the schema on a domain controller (which has these tools installed by default) then the Exchange installation has no way to actually contact AD Schema to prepare it. Doh!

The simple solution here was to install the RSAT-ADDS module. The PowerShell command for each of these are below…

 

  • Server 2008 / 2008R2: ServerManagerCmd -i RSAT-ADDS
  • Server 2012 / 2012R2: Add-WindowsFeature RSAT-ADDS

 

Once this was done (no reboot is required) I reran the Setup.com /PrepareSchema and it worked without an issue:

 

PS working

 

I really wished there would be a normal error here, instead of outputting a bunch of nonsense that a normal admin, engineer or architect would not realize. Seeing something like that makes you think “oh man, there HAS to be permissions issues or something”, which there is not.. well at least not in this case.

 

Then again, I think MSFT read my mind as I purposely didn’t install the RSAT-ADDS module on my other Exchange server (which is now running Exchange 2013 CU5) and I got the following error:

 

Exchange 2013 error

 

At least they threw something non-cryptic in there :]

 

TL;DR: If you are preparing the schema or active directory on a server that is a non-domain controller, make sure you actually install the RSAT-ADDS module. Its required, and will throw a cool cryptic error at you if you don’t.

 

Hope this helps out more folks out there, and as always if you have any questions feel free to leave them in the comments below.

 

Cheers!

– Adam F