IIS 6.0: Lessons in Trustworthy Computing
By Jeff Gonzalez
In the past couple of years, Microsoft’s security programming practices have come under attack, due largely in part to IIS 4/5 exploits. Last January Microsoft Chief Software Architect Bill Gates delivered a company wide e-mail that outlined a concept known as Trustworthy Computing. This resulted in significant changes in the way Microsoft builds and designs software. Internet Information Services 6.0 is the first version of IIS with security in mind. I will explain some of the features and architectural changes in this article.
Application Server Role Overview
The server role is a new feature with Windows Server 2003. To configure a server role, you can install the server role using the Configure Your Server Wizard ( fig. 1 ) and manage your server roles by using Manage Your Server Wizard ( fig. 2 ). There are several built in roles to choose from such as File server, Print server, Mail server, Terminal server, and the Application Server role which I explain further. An application server is a technology that provides core infrastructure services to applications hosted on the server.
Some typical application servers include the following:
- Resource pooling (COM+)
- Distributed transaction management (DTC)
- Asynchronous communication (Message Queuing Services)
- Automatic Web Service interfaces for accessing business objects (UDDI)
- Failover and server health monitoring (Network Load Balancing)
- Integrated Security (Active Directory)
- Web Server (IIS and ASP.NET)
By configuring your server as an application server, you will be installing Internet Information Services and optional technologies such as ASP.NET and COM+.
Application Platform Improvements IIS 6.0 includes several new platform development features such as:
- New COM+ Services for ASP
- Worker Process Recycling
- Dynamic Cache support via VectorSend
- Improved ISAPI Unicode support
- ISAPI Support for custom errors
- Internal Redirection via ExecuteURL Method
- Native ASP.NET Integration
IIS 6.0 and Windows 2003 Server have made significant improvements on the Windows 2000 platform with such features as:
The entire Windows 2003 Server family code base can be compiled for either 32bit or 64bit platforms.
Windows 2003 now implements a production ready, native TCP/IP version 6 protocol stack.
Granular file level compression
In previous versions of IIS, compression was implemented as an ISAPI filter and could only be enabled globally. It can now be configured at the file level.
Quality of Service via resource monitoring
QoS features for IIS 6.0 come in the form of connection limits, timed out connections, application pool queue limits, cpu throttling, bandwidth throttling and memory based process recycling.
Logging improvements address issues such as international support, large sites, or troubleshooting by adding the following features:
- UTF-8 Logging support
- Binary Logging support
- HTTP Error status subcodes
Improved patch management
The patch process for IIS 5.0 was often a headache due to the downtime involved. This has been addressed in IIS 6.0, allowing for less downtime, and automatic patching. All IIS 6.0 application dlls have been compiled resource free to allow greater internalization and localization.
IIS 6 Architecture Overview
IIS 6.0 has several new features that dramatically improve reliability and scalability, such as a new process isolation model much like the worker process isolation mode of ASP.NET, stable request processing architecture, and a revamped and actively managed runtime. Performance improvements like kernel mode queuing and caching have been added. Kernel mode operation allows the system to react to the incoming requests much faster. IIS 5.0 was designed around one process architecture. The one process known as inetinfo.exe functioned as the Web server process. IIS 6.0, in contrast, was redesigned into two new components, the kernel mode HTTP protocol stack (HTTP.sys) and a user-mode administration and monitoring component. This newly designed architecture allows IIS 6.0 to separate the functionality and operational activity from the processing of application code, without sacrificing performance. The two key components in the IIS 6.0 architecture are:
HTTP.sys is a kernel mode protocol stack that queues and parses incoming HTTP requests and caches and returns application content. This component does not load any application code, making it more secure. HTTP.sys listens for requests and queues them as needed. There is a one-to-one relationship between request queues and application pools. HTTP.sys provides a more reliable method of handling requests, because it never executes any application code. HTTP.sys can not be affected by failures in user-mode application code that would normally affect the health of the Web service. HTTP.sys continues to accept and queue requests in the correct queue until one the following happens: the process has been respawned and begins to accept requests, there are no more available queues, all space has been expended in each queue, or the Web service has been shut down. Due to the nature of kernel-mode processing, HTTP.sys is a very efficient request processor.
WWW Service Administration and Monitoring is a user mode process and configuration manager that delegates server operations and monitors the execution of application code. Like HTTP.sys this component does not load or process any application code.
The WWW Service Administrator and monitoring services play two major roles in the IIS 6.0 architecture, server configuration, and worker process management. When the system is initialized, the configuration manager uses the memory resident configuration metabase to initialize the HTTP.sys namespace routing table. Each entry in this table contains reference information that routes incoming URLs to the application pool that contains the corresponding application code. These steps inform the HTTP.sys component that there is an application pool that responds to requests in a part of the namespace, and that prompts HTTP.sys to request a worker process be started for the application pool if any requests come in. All of this registration is done before HTTP.sys begins to route requests. This routing information can be changed dynamically without requiring a service restart.
Applications pools are for managing Web sites and application code. Application pools are separated by Windows 2003 process boundaries. You can configure up to 2,000 application pools per server. An example configuration would be to have Customer A’s applications configured to run inside “Customer A Application Pool ” and Customer B’s applications configured for “Customer B Application Pool”. This provides each application with a significant operational isolation boundary. This allows application pools to operate independently of each other. A request from one application can not be routed to another application pool while being serviced by the current pool. Applications can, however, be reassigned to other pools while the server is running.
Worker process services and isolation
Worker process services are the second new feature we will discuss. This model will seem very familiar to you, if you have had exposure to ASP.NET’s process architecture. A worker process handles all application execution, including authentication and authorization, as well as ISAPI filter and extension loading. This new executable process is called W3WP.exe. When acting as the worker process manager, the www service is responsible for controlling the lifetime of all worker processes that are processing requests. This is very similar to the machine.config section #&060;processModel#&062;, but allows this information to be managed from a GUI interface ( figure 3 ). This management console allows us to configure options such as when to start or recycle a worker process, how many requests to serve before recycling, and what to do if the worker becomes blocked or unable to continue processing requests.
IIS 6.0 supports an all new application isolation mode for processing application and Web sites. This is referred to as worker process isolation mode. This allows all application code to be executed in a completely isolated environment without suffering any performance penalty incursion. Requests are pulled from the kernel directly, instead of having a user mode process pull them into the process, and then recommunicate the information to another user-mode process. Transferring kernel objects is vastly faster than incurring interprocess communication performance hits. This completely eliminates the process hops encountered when sending a request to DLLHost.exe and returning it back again. It is noteworthy that there is no longer an in-process application. All application runtime services are supported equally. Again this reinforces the separation boundary that allows several Web sites and applications to run concurrently without adverse affect on each other. Another great feature of worker process isolation is that if a worker process has to come down, another can be spawned in its place to deal with subsequent requests. This had the added benefit of being able to support other OS level services such as CPU throttling.
Some features of worker process isolation include:
Recycling of worker process
When a worker process becomes unstable the monitoring service can automatically respawn processes based on pre-configured criteria such as elapsed time, process request limit, resource usage and on demand. This allows administrators and developers to troubleshoot and manage applications by reducing the amount of downtime caused by an error or faulty application.
Orphaning of worker processes
The monitoring service can mark worker processes gone awry for “orphaning”. It can start a new worker process in the place of the “terminally ill” worker process. The monitoring service can also run a command (like attaching a debugger) on orphaned processes.
Worker process time out
An application pool can be configured to shutdown a worker process to reclaim unused resources if the application has been idle for a specified amount of time. Additional worker processes will be created as demanded.
Multiple application pools
By allowing the configuration of 2,000 application pools, Microsoft has greatly improved the amount of concurrent processes that can run without adverse affect on server health. Each application pool can be configured separately.
Better partitioning of applications across clustered environments
The advent of application pools creates a very real physical boundary between applications. This allows for a greater deal of isolation and flexibility.
Web garden support and processor affinity
Multiple processes can be configured to service requests for individual application pools. By default each application pool has only one worker process but can be configured to support x number of processes. This is referred to as a Web garden because it is similar in nature to a Web farm, but the difference being that it is localized to one single server. The worker processes respond to requests in round-robin fashion. The benefit comes from having multiple worker processes being able to process requests even if an application service (such as the scripting engine) becomes unresponsive. Tying a particular process to a specific CPU is also available as an option.
Kernel mode caching
By providing a high performing request broker running in kernel mode, IIS 6.0 is able to achieve a previously unparalleled level of efficiency. HTTP.sys has been tuned to boost performance on both static and dynamic contents.
Benefits of new request processing architecture include: fewer restarts, increased stability and reliability, higher application availability, and strong application platform support.
There have been several improvements to the IIS 6.0 security model. One of the factors that has been a concern with previous version of IIS is default configuration. IIS was installed and enabled by default in Windows 2000 Server family installations. This caused problems when exploits were released and people were affected without realizing that IIS was even installed. Microsoft has taken several steps to decrease the attack surface of IIS significantly. IIS 6.0 is not installed by default on Windows 2003 Server family installations. When IIS 6.0 is installed, it is configured in a locked-down state. It is little more than a static content Web server. You must enable Web service extensions in order to service dynamic content. ( figure 4 ) In this example I have enabled ASP.NET, ASP Classic, and server-side includes.
In addition, both ASP Classic and ASP.NET have been configured to run under the lowest privileged account possible. Some other improvements include disabling the ability to install IIS via group policy, disabling of IIS on upgrade, buffer overflow detection, and write protection for content. IIS 6.0 also allows configurable identities for each worker process.
Authentication and Authorization
Authentication and Authorization are often confused with each other. Authentication is the method of defining who you are, and authorization determines what privileges you have. With ASP.NET Microsoft has implemented a new authentication method focused on their Single Sign On goals. IIS 6.0 now supports native Passport integration. A Passport account can also be mapped to a Windows account. Authorization has been vastly revamped as well. In the past authorization has been utilized as ACL (Access Control Lists). The problem with this method is that it is very object driven, focused on files, directories, etc … . Most Web-based applications are more task or operation based. The new authorization framework with Windows 2003 Server provides a solution. IIS 6.0 uses this tool by providing gatekeeper authorization to specific URLs. Web applications can also use URL authorization in tandem with the Authorization manager to control access from within the same policy to control application specific tasks or operations. Constrained, Delegated authentication is the last security feature. Delegation is the act of letting a server impersonate as a user on the network. This allows Web applications to act as a network user for the purpose of gathering information from another server in the enterprise and bringing it back to the Web server in a consolidated manner. Configured correctly this can be a very powerful tool in designing application suites because applications can make use of RPC or DCOM. There are a few caveats with this feature, such as malicious server administrators impersonating clients to gain access to unauthorized servers and services.
In order to improve scalability many applications have moved to a clustered or Web farm model. This can prove difficult from a management and administrative point of view in a large environment. Microsoft has added several features to improve the manageability of multiple servers. In previous version of IIS, the metabase has been configured as a binary store of settings. While there was an IIS Admin object model to access this configuration store, it often proved ungainly for the average administrator. The metabase has been converted to a plain text, human readable XML file for configuration with IIS 6.0. The metabase supports historical change versioning, meaning if the metabase is changed and saved, IIS automatically makes a copy of the original metabase and copies it into a history folder. This makes rollback operations of Web servers very easy. Another feature is the ability to modify the metabase while the Web services are running. An administrator can modify settings, such as creating a virtual directory, while the Web site is operational. When the file is saved, the changes are automatically detected and validated, and if valid, applied to the Web server with no restart or recompilation. This significantly improves uptime for configuration changes.
I hope I have been able to convey the very exciting improvements made to the Microsoft Internet Server platform with IIS 6.0 and Windows 2003. I feel Microsoft has made leaps and bounds with this new version. I look forward to working with this technology in the future.
IIS 6.0 Performance Tuning
by Peter A. Bromberg, Ph.D.
“Some forms of reality are so horrible we refuse to face them, unless we are trapped into it by comedy. To label any subject unsuitable for comedy is to admit defeat.” — Peter Sellers (1925 – 1980)
In the Patterns and Practices Group’s “Improving .NET Application Performance and Scalability“, which is available in full text online and as a PDF download from the above link, as well as in softcover through MSPress and major booksellers, there are over 1000 pages and appendixes of detailed information about how to improve .NET application performance and scalability, written by the top experts in the business. One area that is both little understood and potentially confusing is the tuning of Internet Information Services 6.0.
The skinny about all this is that the PAP group says the default settings shipped with IIS and the .NET Framework should be changed. They provide detailed information in pages 332 through 342 in Chapter 6 on ASP.NET, and they provide even more information in Chapter 17. I’ll summarize some of the more important points here, since I know that, human nature being what it is, most people running IIS and reading this article probably have not waded through this lengthy but excellent publication. Once you see the quality of the information you can get from it, it may encourage you to do so; it is an investment of your time as a professional ASP.NET developer that I highly recommend. The fact that the book is made available free online by Microsoft should not in any way diminish its importance or value to developers who are interested in achieving the absolute best performance and scalability from their .NET Applications.
NOTE: This “helper” article focuses almost totally on the IIS-related issues and settings. However, Chapter 6 and additional information in the various checklists and in Chapter 17 address many other issues that are related to, but do not specifically involve IIS 6.0 settings. Some of these can be addressed at the machine.config level, others are “best practices” coding techniques, and some can be addressed in web.config. Paragraphs marked “Discussion” are my individual comments. The rest is (mostly) untouched snippets from the PAP publication itself.
Formula for Reducing Contention
The formula for reducing contention can give you a good empirical start for tuning the ASP.NET thread pool. Consider using the Microsoft product group-recommended settings that are shown in Table 6.1 if the following conditions are true:
- You have available CPU.
- Your application performs I/O bound operations such as calling a Web method or accessing the file system.
- The ASP.NET Applications/Requests In Application Queue performance counter indicates that you have queued requests.
Table 6.1: Recommended Threading Settings for Reducing Contention
Configuration setting Default value (.NET Framework 1.1) Recommended value
12 * #CPUs
88 * #CPUs
76 * #CPUs
To address this issue, you need to configure the following items in the Machine.config file. Apply the recommended changes that are described in the following section, across the settings and not in isolation. For a detailed description of each of these settings, see “Thread Pool Attributes” in Chapter 17, “Tuning .NET Application Performance.”
- Set maxconnection to 12 * # of CPUs . This setting controls the maximum number of outgoing HTTP connections that you can initiate from a client. In this case, ASP.NET is the client. Set maxconnection to 12 * # of CPUs.
- Set maxIoThreads to 100 . This setting controls the maximum number of I/O threads in the .NET thread pool. This number is automatically multiplied by the number of available CPUs. Set maxloThreads to 100.
- Set maxWorkerThreads to 100 . This setting controls the maximum number of worker threads in the thread pool. This number is then automatically multiplied by the number of available CPUs. Set maxWorkerThreads to 100.
- Set minFreeThreads to 88 * # of CPUs . This setting is used by the worker process to queue all the incoming requests if the number of available threads in the thread pool falls below the value for this setting. This setting effectively limits the number of requests that can run concurrently to maxWorkerThreads � minFreeThreads . Set minFreeThreads to 88 * # of CPUs. This limits the number of concurrent requests to 12 (assuming maxWorkerThreads is 100).
- Set minLocalRequestFreeThreads to 76 * # of CPUs . This setting is used by the worker process to queue requests from localhost (where a Web application sends requests to a local Web service) if the number of available threads in the thread pool falls below this number. This setting is similar to minFreeThreads but it only applies to localhost requests from the local computer. Set minLocalRequestFreeThreads to 76 * # of CPUs.
Discussion: The proviso above indicates that these settings should be used when your application has I/O bound operations and the Applications/Requests In Application Queue perfcounter indicates you have queued requests. However, I have found that settings approaching those indicated can improve performance on ASP.NET apps that do not exhibit these conditions. I recommend using the “Homer” web stress tool from at least one remote machine (and preferably more than one machine, with the supplied ASP controller page), or the .NET ACT Application Center Test application, to throw a good solid load at your app and carefully measure the performance statistics with each set of both the default and the above settings. In particular, pay close attention to the Requests per second and the time to last byte readings. This baseline testing scenario should provide the basis for further tuning if it is necessary, and it doesn’t take long at all. You can only improve something if you have metrics, and the way you get the metrics is to take the time to get them! You can easily script all kinds of “user paths” through your ASP.NET application with testing software such as is mentioned here, and get the important baseline metrics you need. One more thing– rule number 1 of software testing and debugging:
“When you are going to change something, ONLY CHANGE ONE THING AT A TIME!” Test it, get the metrics, and only then, proceed.
Kernel Mode Caching
If you deploy your application on Windows Server 2003, ASP.NET pages automatically benefit from the IIS 6.0 kernel cache. The kernel cache is managed by the HTTP.sys kernel-mode device driver. This driver handles all HTTP requests. Kernel mode caching may produce significant performance gains because requests for cached responses are served without switching to user mode.
The following default setting in the Machine.config file ensures that dynamically generated ASP.NET pages can use kernel mode caching, subject to the requirements listed below.
<httpRunTime enableKernelOutputCache=”true” . . ./>
Dynamically generated ASP.NET pages are automatically cached subject to the following restrictions:
- Pages must be retrieved by using HTTP GET requests. Responses to HTTP POST requests are not cached in the kernel.
- Query strings are ignored when responses are cached. If you want a request for http://contoso.com/myapp.aspx?id=1234 to be cached in the kernel, all requests for http://contoso.com/myapp.aspx are served from the cache, regardless of the query string.
- Pages must have an expiration policy. In other words, the pages must have an Expires header.
- Pages must not have VaryByParams .
- Pages must not have VaryByHeaders .
- The page must not have security restrictions. In other words, the request must be anonymous and not require authentication. The HTTP.sys driver only caches anonymous responses.
- There must be no filters configured for the W3wp.exe file instance that are unaware of the kernel cache.
Discussion: The “enableKernelOutputCache = “true” setting IS NOT present in the default machine.config “httpRunTime” element. Since it is not present, we should be able to expect that the default setting of “true” is automatic. Personally, I feel better explicitly putting the attribute in there, and setting it to “true”. As an aside, I have found that it is ALWAYS a good idea to KEEP A BACKUP COPY of your machine.config stored somewhere safe.
Tuning the Thread Pool for Burst Load Scenarios
If your application experiences unusually high loads of users in small bursts (for example, 1000 clients all logging in at 9 A.M. in the morning), your system may be unable to handle the burst load. Consider setting minWorkerThreads and minIOThreads as specified in Knowledge Base article 810259, “FIX: SetMinThreads and GetMinThreads API Added to Common Language Runtime ThreadPool Class,” at http://support.microsoft.com/default.aspx?scid=kb;en-us;810259 .
Discussion: The .NET Threadpool is somewhat limited in its flexibility and is specifically limited in terms of how many instances you may have per process, since it is static. If you have ASP.NET applications that specifically need to run background thread processing, you may wish to investigate using a custom threadpool class. I have used Ami Bar’s SmartThreadPool with great success, and have even modified it to provide a ThreadPriority overload. You can have more than one instance of this pool, and each can be custom configured. This type of approach provides maximum flexibility while simultaneously permitting individual threadpool tuning of critical resources.
Tuning the Thread Pool When Calling COM Objects
ASP.NET Web pages that call single-threaded apartment (STA) COM objects should use the ASPCOMPAT attribute. The use of this attribute ensures that the call is executed using a thread from the STA thread pool. However, all calls to an individual COM object must be executed on the same thread. As a result, the thread count for the process can increases during periods of high load. You can monitor the number of active threads used in the ASP.NET worker process by viewing the Process:Thread Count (aspnet_wp instance) performance counter.
The thread count value is higher for an application when you are using ASPCOMPAT attribute compared to when you are not using it. When tuning the thread pool for scenarios where your application extensively uses STA COM components and the ASPCOMPAT attribute, you should ensure that the total thread count for the worker process does not exceed the following value.
75 + ((maxWorkerThread + maxIoThreads) * #CPUs * 2)
Evaluating the Change
To determine whether the formula for reducing contention has worked, look for improved throughput. Specifically, look for the following improvements:
- CPU utilization increases.
- Throughput increases according to the ASP.NET Applications\Requests/Sec performance counter.
- Requests in the application queue decrease according to the ASP.NET Applications\Requests In Application Queue performance counter.
If this change does not improve your scenario, you may have a CPU-bound scenario. In a CPU-bound scenario, adding more threads may increase thread context switching, further degrading performance.
When tuning the thread pool, monitor the Process\Thread Count (aspnet_wp) performance counter. This value should not be more than the following.
75 + ((maxWorkerThread + maxIoThreads) * #CPUs)
If you are using AspCompat, then this value should not be more than the following.
75 + ((maxWorkerThread + maxIoThreads) * #CPUs * 2)
Values beyond this maximum tend to increase processor context switching.
Discussion: There is a long list of attention items that revolve around and are tightly woven into the IIS tuning issue for ASP.NET application tuning and scalability. These include, but are not limted to the following:
- Improving page response times.
- Designing scalable Web applications.
- Using server controls efficiently.
- Using efficient caching strategies.
- Analyzing and applying appropriate state management techniques.
- Minimizing view state impact.
- Improving performance without impacting security.
- Minimizing COM interop scalability issues.
- Optimizing threading.
- Optimizing resource management.
- Avoiding common data binding mistakes.
- Using security settings to reduce server load.
- Avoiding common deployment mistakes.
You can find detailed treatment of most of these issues in Chapter 6 of the above-captioned publication.
I hope this brief synopsis of IIS tuning parameters is useful to you. Once again, I strongly recommend reading all this in the bigger context of the book, and mapping out an optimization plan that includes code review, refactoring, and optimization tuning both at the ASP.NET application and IIS webserver levels. One of the great things about the lessons learned from IIS / ASP.NET testing and tuning optimizations is that they can be carried forward to new applications and will improve your skills and value as a professional developer. I spent nearly three weeks at the Microsoft Testing Lab in Charlotte, NC under the tutelage of Dennis Bass and his fine crew, and the lessons learned there were invaluable. If this book were avalable then, I may not have needed to spend so many nights in hotel rooms