I was wondering what authentication settings SharePoint 2010 will use when creating a new Web Application. Because Windows Server 2008 introduced Kernel-mode Authentication and Windows Server 2008 R2 introduced a Kerberos-only authentication provider.

So first a few lines on the what and the where of these settings and after that what SharePoint 2010 uses as default.

Kernel-mode Authentication

This is a new setting in IIS7 (Windows 2008 and Vista SP1) that when enabled increases authentication performance by requiring less switching between user-mode and kernel-mode code. 

You can find this setting in Authentication section in IIS.

Then right click "Windows Authentication" and click "Advanced Settings..."

And there you have the toggle box to switch Kernel-mode Authentication on or off.

And according to the description we want to have it enabled: "...improve authentication performance and prevent authentication problems..." ;-)

Kerberos-only authentication provider (Negotiate:Kerberos provider)

Before this setting was introduced if you wanted to use Kerberos, you would use the Negotiate authentication provider instead of NTLM. However the Negotiate provider will fall back to NTLM authentication if Kerberos authentication fails. So there was no way to enforce Kerberos. With this new provider you can enforce Kerberos.It will not fall back to NTLM.

You can find that setting in the same Authentication section as above. However you choose "Providers..." instead of "Advanced Settings...".

And that will give you option to select which Authentication Provider to use:

So there are 3 options: NTLM, Negotiate and Negotiate:Kerberos.

When you select Negotiate:Kerberos and have Kernel-mode authentication turned on, you will get the following warning however:

So this new provider is not compatible with Kernel-mode authentication. NTLM and Negotiate are compatible and will not give this warning. 

Negotiate provider and Kernel-mode authentication

Note that the setting of Kernel-mode Authentication will change the behaviour of the Negotiate provider.

If Kernel-mode authentication is enabled, then the Kerberos tickets will be decrypted with the local system account. That means that your SPN must be register to the machine account!!!
If Kernel-mode authentication is disabled, it will show the normal behaviour of decrypting Kerberos tickets with the application pool identity. The SPN must be registered to the used application pool identity then.

"So you change the account you register the SPN on when using Kernel-mode, so what's the deal?"
Well, if you are using a (loadbalanced) multi web frontend SharePoint farm (which is like 95% of the SharePoint farm configurations I do), you must run the application pool on each web server with the same domain account if you want to use Kerberos. The SPN for the loadbalanced url can only be registered to 1 single account and not to 2 or more machine accounts, otherwise Kerberos authentication will fail.
So this would mean that you can't use Kernel-mode authentication when doing Kerberos on a webfarm.

There is a way however to use Kernel-mode authentication and Kerberos authenticated webfarms.
There is a setting which is NOT in the GUI interface of IIS to enable the use of the Application Pool Identity to decrypt the Kerberos tickets. It's explained in the OCS 2007 R2 documentation, which basically tells us to set the useAppPoolCredentials attribute to true of the WindowsAuthentication element in the file %windir%\system32\inetsrv\config\ApplicationHost.config. This look like some undocumented feature because it's not in the documentation

Example of configuration:
<system.webServer>
   <security>
      <authentication>
         <windowsAuthentication enabled="true" useAppPoolCredentials="true" />
      </authentication>
   </security>
</system.webServer>

What are the SharePoint 2010 defaults?

When you create a new webapplication you have the choice of using Negotiate (Kerberos) or NTLM as the authentication provider.

Negotiate (Kerberos)
Before I tested this I was wondering if on a Windows Server 2008 R2 environment the Negotiate:Kerberos (Kerberos only) or the normal Negotiate provider was used.
I created a new web application and it enabled the normal Negotiate and NTLM authentication  provider and Kernel-mode authentication was turned off.
That the Kernel-mode was turned off makes some sense: a lot of SharePoint farms will use multiple web frontends and setting a hidden and lesser documented setting in IIS to make it work might create extra support calls But if Kernel-mode was turned off anyway, they could have used the Kerberos only provider. But maybe there is some kind of compatibilty issue?

I have tested SharePoint 2010 in a webfarm with the following 2 scenario's:
1) Negotiate:Kerberos (Kerberos only) and Kernel-mode authentication turned off
2) Negotiate (Normal Negotiate) and Kernel-mode authentication plus the useAppPoolCredentials setting in the ApplicationHost.config file

And they both worked without problems with SharePoint 2010. So I am not sure if there was some design reason behind this. I'll ask some friends at Microsoft about this soon. 

NTLM
So now NTLM. When you choose it, it will only enable the NTLM provider and turns off Kernel-mode authentication. Normally Kernel-mode is on by default. But SharePoint turns it off. I have no idea why. I tested it when turned on and it worked with SharePoint 2010. So the SharePoint team probably decided somewhere to turn it off.

Conclusion

SharePoint 2010 (beta 2) does not use the new authentication features offered by Windows Server 2008 and R2. It will turn off the use of Kernel-mode authentication. Hopefully I have some answers soon if this was a design choice. In my testing scenario's SharePoint 2010 can work in Kernel-mode. I haven't discovered any issues yet. 

So I am stuck now with a very important question: How much performance improvement are we missing if SharePoint 2010 is not using Kernel-mode authentication?  
I already did some testing. Have to write a blog post about it, because the results surprised me! So stay tuned!