(Late) Sunday nights are not a fun time to get paged about a customer down…
Situation
Engineer reboots an Exchange 2010 server that is apart of a two node DAG, but when the server comes back online the MS Exchange Information Store service is stuck in the “Starting” phase within the Services MMC, and the Microsoft Exchange System Attendant will just not start. All other services are online. What gives?
Environment
Another pretty simple buildout, Exchange 2010 SP3UR6 running on top of Server 2008 R2 with all patches (security / hotfixes) that are available. Both servers are multiple-role (CAS/HUB/Mailbox) which also has a CAS array and a two node DAG. Two copies of each database in the environment.
Troubleshooting
The first thing I did was confirm that the only two services that were boinked at the time were the Information Store and System Attendant. Best way to actually check this is by running Test-ServiceHealth if you are running Exchange 2010 / 2013, which showed exactly that.
When I looked in the Services MMC I did see the Exchange Information Store stuck in the starting state. I looked in the Event Viewer: Application logs to find this error:
Log Name: Application
Source: MSExchangeIS
Date: 8/24/2014 11:51:29 PM
Event ID: 5003
Task Category: General
Level: Error
Keywords: Classic
User: N/A
Computer: EXCH01.company.com
Description:
Unable to initialize the Information Store service because the clocks on the client and server are skewed. This may be caused by a time change either on the client or on the server, and may require a restart of that computer. Verify that your domain is correctly configured and is currently online.
The first thing I did was make sure the Windows Time service was properly pulling the right time source..
w32tm /query /source
Since that looked proper (it was the PDC) I moved on and physically checked the time between the two Exchange servers, which looked fine. Confused, I decided to try a force sync of the Windows Time service..
w32tm /resync
Afterwards, performed a reboot and avail.. nothing. Back to square one with the databases showing a state of “Failed” and but this time the Information Store service did not get stuck in “starting”, it was stopped all together. I attempted to start the service with the following error message:
“Unable to start service. Error -2147221213. Please contact your support provider or the vendor”
This is a hex error. Luckily enough if you know the internals a bit to Exchange, sometimes you can look up these errors using the Err.exe tool MSFT provides (link).
That then shows us this..
# for decimal -2147221213 / hex 0x80040123 :
ecTimeSkew ec.h
MAPI_E_INVALID_ACCESS_TIME mapicode.h
We now know its a windows time issue, so now what?
Resolution
There are two routes you can go with this:
– Manually set the Windows Time service to sync off the PDC (which is what I did)
– Completely reset the Windows Time service
Manually set Windows Time service
This is pretty easy, and all you have to do is run this command
net time \\PDCName /Set
It will ask you if you want to confirm, click Y and then confirm that the time is now synced. Afterwards, restart the MS Exchange Active Directory Topology service which will restart all the other Exchange services, and hopefully its fixed for you.
If that doesn’t work try resetting Windows Time all together, and then set it to point to the PDC (sometimes required)..
Reset the Windows Time service
Open up a command prompt and then each of the commands below (each line break is a new command to run)..
net stop w32time
w32tm /unregister
w32tm /register
net start w32time
Once this is done, you should check the actual provider to make sure it is either the DC that is hosting the PDC FSMO role (recommended) or another domain controller that is local to Exchange (e.g: within the same AD site)
w32tm /query /source
If it is NOT the DC holding the PDC FSMO role or even a DC (it might show as “CMOS”, which means your BIOS) then I would set it to either one of the options I said above..
net time \\PDCName /Set
At this point restart the MS Exchange Active Directory Topology service and make sure everything starts.
Once either one of these steps are done, you should be fully working again.
TL;DR: Windows Time sync is required for domain controller authentication on a domain joined machine, along with authentication requests. If windows time breaks (or Windows thinks its broken when the time is actually synced), now you know how to fix it!
Leave any questions and comments below.
Cheers!
– Adam F
The Exchange Guy 