Tag: Troubleshooting

When doing troubleshooting one of the best places where you will find clues about the problem is in the Event Viewer logs. And when doing research on what is going on in the background when performing certain actions in Windows, the Event Viewer will come in handy.

The challenge with Event Viewer is that there are hundreds of log files, and how do you know which one(s) to investigate?

In this blog post you will learn:

• How to use Process Monitor to figure out which the relevant event logs are, for instance when doing troubleshooting or figuring out what a specific action does in the background when researching solutions in Windows.

Method to find the relevant event logs

All event logs which are found in Event Viewer are basically files named <EventLog>.evtx and are located in C:\System32\Winevt\Logs. The key takeaway from this first part is the .evtx file ending.

The tool to use next is my favorite tool when it comes to troubleshooting and learning about stuff in Windows; and the tool is Process Monitor from Sysinternals (owned by Microsoft).

Process Monitor will gather thousands of rows of data within seconds, and this is where filtering comes in.

Crash course in Process Monitor

This is my standard procedure when using Process Monitor. Download Process Monitor if you do not already have it available.

1. Start Process Monitor.
2. Press Ctrl + E to pause recording events.
3. Press Ctrl + X to clear out all events.

Adding the filter for identifying the event Logs

Now, to be able to identify the Event logs files that are of relevance to troubleshooting or to a specific action that you perform, using filters is the key in Process Monitor:

1. In Process Monitor, press Ctrl + L to get to the filters.
2. In the first drop down menu choose Path followed by contains in the next drop down and last enter .evtx
3. Make sure Include is selected and click Add and OK.

Identify the relevant logs

Now you can prepare the troubleshooting step(s) or the action you want to perform and when you are ready for this:

1. In Process Monitor, press Ctrl + E to start recording.
2. As quickly as possible, now reproduce the problem or perform the action you want to know more about.
3. Go back to Process Monitor and hit Ctrl + E to stop recording. Voila, you now have the relevant event logs to further investigate.

After you have the list above, you know exactly what event logs are being written to and you can do further analysis of these. Fire up Event Viewer – and happy hunting! 😊

Examples

Some examples when I’m using this method performing troubleshooting or researching in Windows:

• Troubleshooting failed Store app installation.
• How can I find what event logs are used when plugging in a FIDO2 security key so that I can add a scheduled task to trigger on a certain event to lock the machine when unplugged?
• What log files are written to when doing a Sync with Intune?

Summary

Using Process Monitor to find the relevant event logs is a quick and efficient way to locate where to look next for more details. This applies to both troubleshooting and when researching the inner workings of Windows and figuring out what is going on in the background.

This blog post is an example of a problem I encountered the other day in a project I am in. An application that is used by a part of the business is installed properly but crashes. I thought I’d share some tips and tricks based on this troubleshooting, a troubleshooting which turned out to be a true sunshine story.

Problem

A ClickOnce application is installed in Windows 10 and 11 but when trying to start the application it never starts and instead silently crashes.

Investigation

As always when something crashes, more details can be found in the Event Viewer. The event ID 1000 lists some very general information:

Faulting application name: X.Y.Client.WpfClient.exe, version: 1.0.0.0, time stamp: 0x565f048b
Faulting module name: KERNELBASE.dll, version: 10.0.25357.1, time stamp: 0xc0dc8053
Exception code: 0xe0434352
Fault offset: 0x0014e0a4
Faulting process id: 0x0x473C
Faulting application start time: 0x0x1D9824B72D664C7
Faulting application path: C:\Users\andre\AppData\Local\Apps\2.0\KP6YOQBZ.QBT\10VX2RL2.D3R\X...tion_ae3633e36a16d69b_0004.0000_6d9d02277ead5c24\X.Y.Client.WpfClient.exe
Faulting module path: C:\WINDOWS\System32\KERNELBASE.dll

This in turn gives me nothing more to go on so next thing to do to get more information is to enable crash dump file generation for application crashes (or any other crashes apart from Windows crashes which already have dump files generated each time Windows crashes).

Enable crash dumps

Go to the following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting

First Create a key named LocalDumps so that you end up with this key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps

Then in the LocalDumps registry key create these three registry values:

Name: DumpFolder
Type: REG_SZ
Value: C:\CrashTemp

Name: DumpCount
Type: REG_DWORD (32-bit)
Value: 10

Name: DumpType
Type: REG_DWORD (32-bit)
Value: 2

Restart the service named “Windows Error Reporting Service” and then start the application and note the DMP file created in the location that you specified above.

Analyze crash with WinDbg

Now we can analyze the DMP file with the classic tool Windows Debugging Tools. This is available in Windows ADK and SDK but the easiest way is to install WinDbg (Preview) via Store (or publish to Company portal). You can also use the winget command to install WinDbg by using “winget install 9PGJGD53TN86“.

Start WinDbg and then open the DMP file and choose:

!analyze -v

We can then clearly see some interesting exceptions:

Key  : CLR.Exception.System.ArgumentException._message
Value: Source property was not set before writing to the event log. 
Key  : CLR.Exception.System.Security.SecurityException._message
Value: The source was not found, but some or all event logs could not be searched.  Inaccessible logs: Security.

This, plus the below entries also found as a result of analyzing the DMP file, clearly points toward event logs and specifically the security event log.

STACK_TEXT:
00f8eedc 64b54041 System_ni!System.Diagnostics.EventLogInternal.WriteEntry+0x16bc4d
00f8ef0c 649e53f9 System_ni!System.Diagnostics.EventLog.WriteEntry+0x19
00f8ef18 0314223b X_Y_Client_Business!X.Y.Client.Business.Logger.WriteToLog+0x103
00f8f034 0314211d X_Y_Client_Business!X.Y.Client.Business.Logger.WriteToLog+0x25

Process Monitor for the win!

To figure out what is going on I turn to my personal favorite tool named Process Monitor, a tool that has helped me troubleshoot and learn stuff about Windows for many years.

In Process Monitor, I did a simple recording and filtered on “Access denied”. The application process showed one access denied entry.

The application need read permissions on the registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EventLog\security 

Said and done, I set users to “Read” on the registry key and started the application again. It crashed still.

I did another trace with Process Monitor and this time it showed that read/write permissions was required on the registry key above security. Strange, but I set Users to Full Control on the registry key referenced:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EventLog

I once again tried to start the application and after that, the application started! Note: The good(?) thing is that after first start, one can revert the permissions to default permissions and the application still work. More investigation is needed in this area.

Summary

A few tools, tips and tricks were involved in this troubleshooting, and I hope to inspire others to use these tools and methods in their own troubleshooting in day-to-day work. My next step now is to contact the developers of the application and point out the rather strange problem, and hopefully get the problem fixed.

So to summarize the key areas which you can look into when optimizing performance from an infrastructure point of view here is a summary of the key takeaways from TechEd session WCL326: Five infrastructure changes that will boost performance for the Windows Client.

1. Slow machine boot and login / GPOs and scripts

Use Windows Performance Toolkit (part of Windows 7 SDK) to troubleshoot what is happening during boot. Specifically narrow in one Group Policy in the section in the Generic events and look for and enable only the Group Policy provider to see what’s going on with group policies. Group policies and scripts are most often the bad guys when having performance problems with boot and login.

Also use Event Viewer > Applications and Services > Windows > Group Policy > Operational log to look for instance events with id 5326, 8000, 8001or 5016. In particular the last one is of interest as this will quickly show you which Group policy extension is taking most of the time to finish.

Cleanup, remove unnecessary settings and GPO objects. Convert scripts to Group Policy Preferences as necessary or make scripts running scheduled after startup or login to minimize the boot and login times.

2. Optimizations for RDP

Activate asynchronous login for users to speed up login for Remote Desktop Services and RemoteApp. Go to Administrative templates > Policies > System > Group Policy and set the setting for “Allow asynchronous user Group Policy processing when logging in to Remote Desktop Services”.

Three other really great tweaks found in Administrative templates > Policies > Windows Components > Remote Desktop Services > Remote Desktop Session Host > Remote Session Environment:

Do not allow font smoothing = Enabled
Limit maximum color depth =  Enabled, set it to 32-bit
Set compression algorithm for RDP data = Enabled, set it to Optimized to use less network bandwidth

3. SMB 2.1

To get full use of performance improvements in SMB2.1 protocol you need file servers that are running Windows Server 2008 R2 or if you are running a third party storage solution to activate SMB2.x support as that is not always activated by default and sometimes a firmware upgraded is needed.

Performance increases based on my own performance measuring are varying from 10-80% performance increase.

4. BranchCache

Activate BranchCache feature from Server Manager on the content servers you want to use with BranchCache. Require windows Server 2008 R2 on the content server. For file shares make sure to enable the BranchCache feature on the share(s) you want to use with BranchCache. Also set the group policy “Hash Publication for BranchCache” on the file server(s) found in Administrative templates > Policies > Network > Lanman Server.

To activate BranchCache on the Windows 7 client look in Administrative templates > Policies > Network > BranchCache and activate the required GPO settings.

5. Upgrade key servers to Windows Server 2008 R2

To gain use of RDP improvements, SMB2.1 improvements and actually make performance better for file handling the simple thing to do is to migrate to Windows Server 2008 R2.

BONUS 1. Microsoft tool to measure performance:

WDRAP (Risk and health Assessment Program for Windows Desktop) is a tool designed for enterprise customers that verifies overall performance, including bad drivers, apps that are causing the machine to start slowly etc. Contact your Technical Account Manager at Microsoft to get more information and analyzing the results with this tool. Microsoft themselves used this tool some time ago to improve performance in their environment, more on this in the Microsoft IT Case Study.

BONUS 2. Hotfixes related to infrastructure and performance, Windows 7 Post-SP1:

You experience a long domain logon time in Windows 7 or in Windows Server 2008 R2 after you deploy Group Policy preferences to the computer
http://support.microsoft.com/kb/2561285

Unexpectedly slow startup or logon process in Windows Server 2008 R2 or in Windows 7 (WMI issue)
http://support.microsoft.com/kb/2617858

Slow performance when you browse the My Documents folder in the document library in Windows 7 or in Windows Server 2008 R2
http://support.microsoft.com/kb/2690528

Improved interoperability between the BranchCache feature and the Offline Files feature in Windows 7 or in Windows Server 2008 R2
http://support.microsoft.com/kb/2675611

General Q and A

Q: Can I use this tool to measure performance and troubleshoot on Windows XP?
A: You can run the tool on Windows XP by copying xbootmgr and xperfctrl.dll to an XP machine. You can then analyse the results on a Windows 7 machine. However do not expect the same amount of detailed data as Windows 7 has introduced new features that are not available in Windows XP.

Any further questions around the session or the topics, feel free to leave a comment to the article or send me an email on andreas.stenhall@knowledgefactory.se.

SLIDES: Download the slides from the session WCL326

 

Mapping printers using Group Policy Preferences is a really nice feature and it is supposed to be working much better than using traditional scripting technologies. Let me tell you about an interesting day troubleshooting why printers didn’t want to map using Group Policy Preferences. In the logs it just stated “catastrophic failure” which does not sound good at all nor makes much sense.

Log Name: Application
Source: Group Policy Printers
Event ID: 4098
Description: The user 'X' preference item in the 'Y {3EE4E80F-17CB-4E56-9237-4FC8B9FA090A}' Group Policy object did not apply because it failed with error code '0x8000ffff Catastrophic failure' This error was suppressed.

Logging in as the user on another machine did not produce the same problem and the mapping of printers work fine, as did mapping the printers when logging in using an administrator account.

This got me thinking that I should try mapping a printer manually as the standard user so I did. It got me a “You do not have permission to use the selected printer” which made me turn to the classic tool Process Monitor and to start a trace. It didn’t take long to see that after filtering all logs for a result of “denied” resulted in the following line:

Operation: CreateFile
Result: ACCESS DENIED
Path: \\printserver\print$\w32x86\3\OPLO_UM.dll
Desired access: Generic Read

Note that CreateFile does not necessarily mean to create a file, it can mean “read file” as documented by Microsoft for the CreateFile function. 

So the conclusion was that after investigating with the printer department the printer share “print$” security permissions did not match the user and the user actually did not have read permissions to read the driver which is an absolute requirement for the printer to be mapped (as the print driver is actually installed when the printer is mapped).

Also one setting that affected the behavior was that one have to set “Run in logged-on user’s security context” in the Common tab of the printer mapping Properties. This is an essential part of the solution…

Case closed!

In Windows you might know that a lot of services do not run as separate processes, instead many of them are actually run within the famous svchost.exe processes. To see this in action just go to a cmd.exe and type “tasklist /svc” and it will list all processes and subprocesses. It might look like:

svchost.exe    AeLookupSvc, Appinfo, AppMgmt, BITS,
               Browser, CertPropSvc, gpsvc, IKEEXT,
               iphlpsvc, LanmanServer, MMCSS, ProfSvc,
               Schedule, SENS, SessionEnv,
               ShellHWDetection, Themes, Winmgmt, wuauserv

If you need to do some troubleshooting or investigate something related to a certain process that is running with other services you can split this service to make in run in its own process. This is done by using the command:

sc.exe config servicename wuauserv type= own

Which in this case makes sure that the Windows Update service (wuauserv) will be run in its own process, for you to troubleshoot. Set it back to its original setting by changing “own” to “share” in the above command. You can find more information about this in KB934650.

Opening CHM (Windows Help Files) downloaded from the internet or from a network location will most likely give you “Navigation to the webpage was canceled” and it will refuse to display the contents of the help file. This is due to changes that were introduced some years ago by a security update, and this is the default behavior in Windows Vista and 7 (and their Windows server equivalents).

I experienced this case at a customer site and to resolve the problem you can follow the steps in MS KB article KB896054. However adding the trusted UNC path caused some problems in this case, as the UNC path was sort of hidden because the CHM file was opened from an embedded link within an application.

So to find the UNC path from where the CHM file was opened I had two choices, either to use the famous Sysinternals tool Process Monitor to trace the open file action, but I chose a quicker way. Opened CHM files are logged in a file called hh.dat which can be found at C:\users\\AppData\Roaming\Microsoft\HTML Help. This file, when opened with WordPad contains the path to the opened CHM file, which lead me to the key which was an important part of this solution.