Steve’s previous It’s Tool Time blog about using DebugDiag gives some great advice on how to reactively gather dump files for analysis of issues that have been noticed with Enterprise Vault. There are also some areas of the product that will proactively produce dump files via integration with Windows Error Reporting (referred to as WER in the rest of this post), saving the need for DebugDiag.
Unless you have specifically disabled WER, then it will normally flag its involvement in catching any issues via the Application log. The event log application error 1000 below is an example, of WER in play, on this occasion corresponding to a recurring crash of the Enterprise Vault StorageCrawler.exe process.
And each application error 1000 event was accompanied by WER informational 1001 events, which exposed more useful information for troubleshooting the crash.
Windows does a good effort of bubbling this kind of information up for the admin to notice and pay attention to by duplicating it in Problem Reports which can be found in the server’s Action Center. Recurring problem reports of apparent common root cause are grouped together, and opening each individual report reveals the same information as seen in the event logs with a little more formatting to help understand and analyse the problem reports in more depth.
Some key data to pay attention to in each problem report are
- Application Name, which represents the faulting process
- Exception Code, which represents the hexadecimal error code that was thrown at the time of the issue
- Exception Offset, which reflects the point in the application code where the exception occurred
If you observe the same values repeating in these three pointers across multiple problem reports, then it is a fair indication that the server is experiencing a repeatable / reproducible issue that needs further investigation by Support, and most likely, Engineering.
In its default mode, Windows Error Reporting will produce additional files to help with this investigation as displayed in the report above in the ‘Files that help describe the problem’ section. Clicking on the link to ‘View a temporary copy of these files’ reveals however that the default of WER is to produce mini-dumps in the temporary location
which are subsequently transferred to the default WER ‘ReportQueue’ location
Don’t call in the issue just yet though as, whilst Mini-Dumps can be helpful in determining the root cause of an issue, a Full Dump is far more desirable for debug data hungry Engineers. So, if you have identified a recurring application error / problem report with an Enterprise Vault process, then now is a good time to temporarily enable advanced WER settings as explained by Microsoft here - http://msdn.microsoft.com/en-us/library/windows/desktop/bb787181(v=vs.85).aspx
In the example I have been using through this post, simply setting ‘DumpFolder’ and ‘DumpType’ resulted in Full Dumps being created for each occurrence of the issue in a ‘known’ directory – c:\CrashDumps – which could then be zipped and sent to Engineering for analysis, from which we were quickly able to determine the root cause of the issue and resolve it in a future release. Also be aware that once the required full dumps have been created, it may be a good idea to turn off the WER advanced settings if disk space consumption from subsequent dumps is likely to be a problem.
So keep an eye on your application event logs and your Problem reports and if you do see that WER is trying to flag a recurring issue with an Enterprise Vault process, then hopefully the information above will help you gather the necessary debug information to help us resolve it.