Tim Anderson's ITWriting

September 21, 2005

Fixing Windows Server 2003

Generally I find Windows reliable (client and server), but I do stress it by installing and removing all sorts of software for test purposes, some of it beta. Last night I noticed a problem with Sharepoint Services on my Windows 2003 server. I don't have a limitless budget for servers, so this machine is the sole Domain Controller and also handles Exchange Server and even Apache. I now know that having Sharepoint Services and Exchange on one server is asking for trouble, though I recollect that it used to work. Not any more; when I tried to log onto the admin site it asked me for username and password (bad sign, since it does integrated authentication) and then gave me Access Denied. I also noticed that Windows Rights Management administration had similar issues - yes, I have a lot of stuff installed here.

I tried to fix it. I read up on the problems with Sharepoint disabling Kerberos and breaking Outlook Web Access and took the recommended steps. I removed and reinstalled Sharepoint services. Still no go. I tweaked a few things and tried a server restart.

Now I was really in trouble. My event viewer filled with exotic messages - does anyone collect these? - including:

• 32777 The LSA was unable to register its RPC interface over the TCP/IP interface. Please make sure that the protocol is properly installed.
• 12261 SAM failed to start the TCP/IP or SPX/IPX listening thread.
• 5706 The Netlogon service could not create server share C:\WINDOWS\SYSVOL\sysvol\mydomain.co.uk\SCRIPTS. The following error occurred: The remote procedure call failed.
• 4294 The IPSec driver has entered Block mode. IPSec will discard all inbound and outbound TCP/IP network traffic that is not permitted by boot-time IPSec Policy exemptions.
• 1053 Windows cannot determine the user or computer name. (There are no more endpoints available from the endpoint mapper. ). Group Policy processing aborted.

Naturally I Googled on these, to no great advantage. Other stuff failed too. Basic TCP/IP seemed to be working - I could ping localhost - but the network was not visible and the network connections service would not start, breaking the Control Panel applet. Of course Exchange was well out of things.

What to do? Zap the box and rebuild - reluctant to do this as it's time-consuming. Restore a recent backup - reluctant here as well, since there was some configuration work that would be lost. I did have a backup of system state from before I fiddled with Sharepoint, so I booted into Directory Services Restore Mode and restored system state.

Nothing changed, all the same errors.

Normally I would fix problems by finding the first error after boot, fixing it, and going on from there. However this looked too severe. We've all been there - hours spent manually applying permissions, checking for the existence of files, tweaking registry entries, and often no result. Another little problem: the installer service was broken, so reinstalling anything was problematic.

There was one more easy thing to try. I downloaded Windows Server 2003 with SP1 from MSDN (what would I do without it?) and booted from the CD. Then I reinstalled Windows choosing the Repair option. Downloading the CD was necessary, since Repair only works with the same build of the OS.

It was fantastic. Almost everything worked. Networking was fixed, Exchange started, OWA worked, in fact the only service to fail, with a mysterious "file not found", was WINS. Removing and reinstalling fixed that. Even Sharepoint sort-of worked; you could log on to the admin site. It was not quite right though; the Sharepoint installer database was corrupt such that you could neither update it nor remove it. I used the invaluable msizap to delete the Installer entries and manually removed Sharepoint Services. I've not tried putting it back yet; that can wait for a quiet afternoon.

I have a couple of comments following this saga. First, Windows is infuriating when it is sufficiently broken that the GUI management tools won't run. At this point you have to dive into the registry with some deep technical skills, or give up. Text config files are easier in these cases; it is interesting that Microsoft is moving in this direction (.NET apps, for example). Second, kudos to Microsoft for the Repair install, which was highly effective in this case. Third, keep good backups folks, especially if you are a one-server shop.