IIS7 (and above) – Using FREB to capture dumps for a long running request

In my previous post, I wrote about using FREB to troubleshoot a simple error message to identify the module which sets the error. For this post, let’s assume we are troubleshooting a slow running page (any page, be it aspx, asmx, html, etc). Not always I would recommend you to collect a dump to understand what’s going on inside. There are a few other ways to troubleshoot a slow running page, say enabling FREB to find out which module is taking long time to process, using RSCA (runtime status control API) data to understand the state of the server, using ETW to understand what’s going on, etc.

Let’s assume that we are at a point where we need to collect dumps and figure out what’s happening inside the process causing the slowness of the page.

FREB in IIS7 (and above) comes with an ability to execute an executable when failure definition is satisfied. For example, you can execute an executable when the page takes more than 15 seconds to come up. We can use the AdPlus.vbs shipped with the Windows Debugging Toolkit to collect manual hang dumps of the processes. Below are the steps to configure the same:

  1. Enable Failure Request Tracing on the Website
  2. Go into the “Failed Request Tracing Rules” feature, and click on “Add..” from the Actions pane
  3. Select the content you want to trace
    • If you want to trace only a specific page, select custom, and give the page name, and click on Next
  4. Uncheck “Status code(s)”, and select “Time taken (in seconds)”, and enter the number of seconds as 15 seconds
    • Note: This number of seconds is a value decided by you who is troubleshooting, and some pages may take more than 20 seconds to come up based on the logic, or DB activity

    image

  5. Click on Next, and choose Finish to activate the rule

The UI does not have an option to specify the EXE to be executed when the page takes 15 seconds to come up. You need to configure the settings using appcmd, or Configuration Editor. Before we see the appcmd commands to configure, let’s see what are those settings that need to be configured.

Take a look at the configuration reference for the system.webServer/tracing/traceFailedRequests/add. You have the customActionExe, customActionsParams, customActionsTriggerLimit, and the path. So, if you want to execute the AdPlus to collect dumps of all the workerprocesses, below are the values you need to set to:

customActionExe                 c:\windows\system32\cscript.exe

customActionParams              c:\debuggers\adplus.vbs -hang -pn w3wp.exe -o c:\dumps-quiet

customActionTriggerLimit        10

Most importantly, you need to set the customActionsEnabled at the website level (under system.applicationHost) to true.

customActionsEnabled            true

Below are the appcmd to configure them:

Set the customActionExe

appcmd.exe set config "Default Web Site" -section:system.webServer/tracing/traceFailedRequests /[path='test.aspx'].customActionExe:"c:\windows\system32\cscript.exe" 

Set the customActionParams

appcmd.exe set config "Default Web Site" -section:system.webServer/tracing/traceFailedRequests /[path='test.aspx'].customActionParams:"’c:\debuggers\adplus.vbs’ -hang -pn w3wp.exe -o c:\dumps -quiet"

Set the customActionTriggerLimit

appcmd.exe set config "Default Web Site" -section:system.webServer/tracing/traceFailedRequests /[path='test.aspx'].customActionTriggerLimit:"5"

or everything at one go

appcmd.exe set config "Default Web Site" -section:system.webServer/tracing/traceFailedRequests /[path='test.aspx'].customActionExe:"c:\windows\system32\cscript.exe" /[path='test.aspx'].customActionParams:"c:\debuggers\adplus.vbs’ -hang -pn w3wp.exe -o c:\dumps -quiet" /[path='test.aspx'].customActionTriggerLimit:"5" 

and, most importantly setting customActionsEnabled to true on the website level

appcmd.exe set config  -section:system.applicationHost/sites /[name='Default Web Site'].traceFailedRequestsLogging.customActionsEnabled:"true"  /commit:apphost

Below is how your web.config would look like:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <tracing>
            <traceFailedRequests>
                <add path="test.aspx" customActionExe="c:\windows\system32\cscript.exe" customActionParams="c:\debuggers\adplus.vbs -hang -pn w3wp.exe -o c:\dumps -quiet" customActionTriggerLimit="5">
                    <traceAreas>
                        <add provider="ASP" verbosity="Verbose" />
                        <add provider="ASPNET" areas="Infrastructure,Module,Page,AppServices" verbosity="Verbose" />
                        <add provider="ISAPI Extension" verbosity="Verbose" />
                        <add provider="WWW Server" areas="Authentication,Security,Filter,StaticFile,CGI,Compression,Cache,RequestNotifications,Module,FastCGI" verbosity="Verbose" />
                    </traceAreas>
                    <failureDefinitions timeTaken="00:00:15" />
                </add>
            </traceFailedRequests>
        </tracing>
    </system.webServer>
</configuration>

Above configuration would collect a dump of all w3wp.exe processes running, and place them in the c:\dumps folder when a request for test.aspx takes more than 15 seconds.

Hope this helps!

No Comments