Michel Barneveld's Blog

WaitForWspJob Tool

Michel Barneveld — Mon, 05 Apr 2010 12:08:00 GMT

WaitForWspJob.exe is a tool for SharePoint 2007 that will wait for the Admin Service to finish the deployment or retraction of a SharePoint solution (wsp). Download page: http://blog.michelbarneveld.nl/media/p/5623.aspx

Usage

The tool accepts the solution file name or the solution id (guid) and you can give it a timeout value. By default it waits 5 minutes for the solutiion job to finish before it returns control.

Usage:
WaitForWspJob.exe <wsp> [<timeout>]

Arguments:
wsp The name or guid of the SharePoint solution
timeout The maximum wait time in minutes. Default: 5

Examples:
WaitForWspJob.exe ExampleSharePointSolution.wsp
WaitForWspJob.exe ExampleSharePointSolution.wsp 10
WaitForWspJob.exe 3ce2bc5d-69b2-47bd-bd89-80d138f4faba

Why do you need it

Well, you don't have to use it ;-) There are other good (GUI) SharePoint solution installers available or you can even use the Solution Management screens in Central Admin or the STSADM commands. But when working with batchfiles and deploying/retracting solutions you might run into errors/weird behaviour. I wrote this tool in a few minutes to solve my issues with deployments with batchscripts.

After deploying or retracting a SharePoint solution, you have to wait till the Admin service has executed the jobs to deploy/retract the solution, before you can delete the solution (in case of retract scenario) or activate the features in the solution (deploy scenario).
Otherwise you will get an error like in below example of deleting the solution directly after retracting it.

D:\>stsadm -o retractsolution -name Sample_Solution_v1.0.wsp -allcontenturls -im
mediate

Timer job successfully created.


D:\>stsadm -o deletesolution -name Sample_Solution_v1.0.wsp

The solution "Sample_Solution_v1.0.wsp" has been deployed in the farm. Please re
tract the deployment before removing the solution.You can also use the -override
parameter to forcibly remove the solution, but you will not be able to retract
the solution deployment.

You can force running the jobs by executing the STSADM command ExecAdmSvcJobs, however that STSADM operation doesn't check if the Admin Service is also running that solution job. So most of the times the job gets executed twice at the same time! This can lead to all kinds of problems. In the deployment scripts I was working on a few weeks ago, this resulted in solutions that couldn't be deployed because they were already deployed according to STSADM, however the Solution Management in Central Admin showed them as not deployed!?! You can use the -force parameter in such a case, but I only want to do that if nothing else works.

Using the ExecAdmSvcJobs after a deployment can lead to messages like: "The job completed successfully, but could not be properly cleaned up. This job may execute again on this server."
Like in this example:

D:\>stsadm -o deploysolution -name Sample_Solution_v1.0.wsp -url http://sp07dev
-immediate -allowgacdeployment

Timer job successfully created.


D:\>stsadm -o execadmsvcjobs

Executing .
Executing solution-deployment-sample_solution_v1.0.wsp-0.
The solution-deployment-sample_solution_v1.0.wsp-0 job completed successfully, b
ut could not be properly cleaned up. This job may execute again on this server.

Operation completed successfully.

When you get a message as above you can also see that the job was executed twice in the Solution Manager in Central Admin. The message that the solution was deployed is shown twice!

My solution was simple:

Make sure that the Admin Service is running
Don't use the STSADM command ExecAdmSvcJobs
Use WaitForWspJob.exe to wait for the jobs to finish before executing any other command

Example usage in a script:

stsadm -o retractsolution -name Sample_Solution_v1.0.wsp -allcontenturls -immediate
WaitForWspJob.exe Sample_Solution_v1.0.wsp
stsadm -o deletesolution -name Sample_Solution_v1.0.wsp

Example output:

D:\>stsadm -o retractsolution -name Sample_Solution_v1.0.wsp -allcontenturls -im
mediate

Timer job successfully created.


D:\>WaitForWspJob.exe Sample_Solution_v1.0.wsp
Solution (Sample_Solution_v1.0.wsp) found in the solution store.
Wating for solution job to finish
...................
Solution job finished!
Last Operation Result: RetractionSucceeded

D:\>stsadm -o deletesolution -name Sample_Solution_v1.0.wsp

Operation completed successfully.

Internals of the tool

So how does the tool determine if there is still a job running?

You can get a SPSolution object to your solution from the solution collection from SPFarm.Local.Solutions. The SPSolution has a property JobExists that returns true if there is still a job running or waiting.
There is also a property JobStatus, but be carefull when you use it, because it will throw an exception the moment the job is finished.

So the tool is nothing more than a loop checking for the JobExists property to go false. In short:

SPSolution wsp = SPFarm.Local.Solutions[wspNameOrGuid];
while (wsp.JobExists)
{
System.Threading.Thread.Sleep(1000);
}

The SPSolution class has also properties to determine if the job went successful or not: LastOperationResult, LastOperationDate and LastOperationDetails.

Full source is downloadable at: http://blog.michelbarneveld.nl/media/p/5624.aspx

Kerberos Authentication Tester

Michel Barneveld — Sat, 05 Dec 2009 20:25:00 GMT

Quite often I am wondering if a site is using Kerberos or NTLM. You can use tools like Fiddler, Network Monitor and such for that. But sometimes I just want to have a simple tool without installation like when working on computers where you can't install such software but are allowed to run executables. For that I have created a tool: Kerberos Authentication Tester.

Kerberos Authentication Tester Features:

It shows what authentication method is used in a web request: None, Basic, NTLM or Kerberos
It shows the SPN used in case of Kerberos
It shows the HTTP status
It shows the HTTP Headers of the request.
It shows the version of NTLM used (v1 or v2)
It has a detailed view with a complete breakdown of the Authorization header. (Yep, went through all the RFCs to dissect the Kerberos and NTLM packages)
It shows your current Kerberos tickets and allows you to remove them (like klist.exe)

Some Screen shots:

The main form:

The blue details link at the top right shows a new screen with the details of the request, including a breakdown of the Authorization HTTP Header. You can also save this XML.

Use different credentials or a proxy:

And it also has quick access to the Kerberos Ticket of the current users. And you can even delete them.

It's still in beta, so your mileage may vary ;-)

How to create a custom DocumentIdProvider for SharePoint 2010 (including source code)

Michel Barneveld — Sat, 05 Dec 2009 12:49:00 GMT

[4/26/2010] Updated with interface changes from beta2 to rtm

In SharePoint 2010 there is a new feature called the Document ID. When that feature is enabled, it will assign an unique ID and a permanent url to the document in a document library. Documents can be found using that Document ID, even if the document is moved to another location within the document library, site, or farm.

The default structure of the DocumentID is: <PrefixString>-<ListId>-<ListItemId>
Example: E2E4NK52YQC4-2-4

That prefix string is created as a unique string for every Site Collection. The first way to customize the DocumentID is to use the Document ID Settings page in Site Settings.
In this page you can change the prefix string:

But if that is not enough customization, there is also an other way: Your own DocumentIDProvider implementation.

Just derive your class from the abstract class Microsoft.Office.DocumentManagement.DocumentIdProvider and implement 1 property and 3 methods,

But before we take a look at the methods we need to implement, let's take a look how Document IDs work.

How does a Document ID work

When a document is added to a document library and the Document ID feature is enabled in that site collection, the event ItemAdded is trapped and the registered Document ID Provider is asked to generate a Document ID. This ID is then assigned to the DocumentID column. Note that the asynchronous event ItemAdded is used and not ItemAdding to be able to access metadata for the Item when assigning Document IDs. This page in the documentation explains more about the different events that are used: Document IDs and the DocID Service.

When you use a view that includes the DocumentID column or when you view the properties of the item, you will notice that the Document IDs are rendered as links to: _layouts/DocIdRedir.aspx

There is also a Document ID webpart to search for documents (The normal search also finds documents with Document IDs). That webpart also uses the DocIdRedir.aspx page.

If you want to add this webpart, it's in the Documents Category and not in the Search Category:

How does DocIdRedir.aspx work?

The DocumentID column is an indexed property of Search. So when your document is indexed it can be found by using the Search API. But the Out-of-the-box DocumentIdProvider does something smarter first.

Remember that the default structure of the DocumentId is: <Prefix>-<ListID>-<ListItemID>. So before asking SharePoint Search to find the document, it will first check the prefix if that is the prefix of the current Site Collection and if it is it can quite easy get the Item based on the ListID and ListItemID. If the item exists and it has the same DocumentID then the users is redirected to this document. If the document can't be found via this optimized way, then the Search API is used. If that also can't find the document then an error is shown.

Example of error message:

That is the behaviour of the Out-of-the-box DocumentIdProvider. Now we have enough information to look on how to custimize the provider.

Custom DocumentIdProvider

The generating of Document IDs can be custimized by creating a class that derives from Microsoft.Office.DocumentManagement.DocumentIdProvider

The definition of DocumentIdProvider:

namespace Microsoft.Office.DocumentManagement
{
  public abstract class DocumentIdProvider
  {
    protected DocumentIdProvider();

    public abstract bool DoCustomSearchBeforeDefaultSearch { get; }

    public abstract string GenerateDocumentId(SPListItem listItem);
    public abstract string[] GetDocumentUrlsById(SPSite site, string documentId);
    public abstract string GetSampleDocumentIdText(SPSite site);
}
}

You have to implement the 3 abstract methods and 1 abstract property. But how are they meant to be used?

DoCustomSearchBeforeDefaultSearch

So what does this property do? Remember that the ootb provider first tries to find the document based on the ListId and ListItemId and if that doesn't give a result it will revert to using SharePoint Search. This boolean controls that behaviour. Notice that there is also a GetDocumentUrlsById method in this interface.

If you return true, then you let the DocIdRedir.aspx know that you want it to call the GetDocumentUrlsById first before calling Search. If you let this Property return false, you want to use SharePoint Search first, before using the GetDocumentUrlsById method. Note that it will only tries the 2nd search method when the first search method doesn't return a result!

GenerateDocumentId

This method is needed to generate the DocumentId. The SPListItem is passed as an argument in case you want to generate a Document ID based on the list item metadata. You have to return your DocumentId as a string.

GetDocumentUrlsById

This method is used to return the urls of the document based on a Document ID. This can be used as an alternative when the document isn't indexed yet and therefor the Search way didn't returned a result. Or if you have beter way to retrieve the document in the Site Collection.

If you don't want to return any urls, just return an empty string array.

return new string[0];

If DoCustomSearchBeforeDefaultSearch was true, then returning the empty string array will signal the DocIdRedir.aspx to try it again with the SharePoint Search.
If DoCustomSearchBeforeDefaultSearch was false then the SharePoint Search was called first but didn't gave a result and the GetDocumentUrlsById was called. Returning an empty string array in that case will generate a message that the file could not be found.

An other possible scenario is that you set DoCustomSearchBeforeDefaultSearch to false and rely on the SharePoint Search. And if that can't find your document, you use the GetDocumentUrlsById to return a fixed url to a page that informs the user that the document can't be found (e.g. The document might not be indexed at this time if it was recently added.)

GetSampleDocumentIdText

This method is used to populate the Document ID webpart with an example.

The text E2E4NK52YQC4-1-1 is returned by this method in above example.

Registering the provider?

So now that you have implemented your custom DocumentIdProvider, how do you get it registered to a Site Collection (or unregistered)?

I didn't find any STSADM or PowerShell command for that, but the class Microsoft.Office.DocumentManagement.DocumentId has 2 methods that we can use:

public static void SetDefaultProvider(SPSite site);
public static void SetProvider(SPSite site, DocumentIdProvider iProvider);

The method SetProvider will register your provider to SPSite and removing it is done by calling SetDefaultProvider.

A good way to call these methods is in the events of a Feature event receiver.

Example:

  public override void FeatureActivated(SPFeatureReceiverProperties properties)
{
   SPSite site = (SPSite)properties.Feature.Parent;
    DocumentId.SetProvider(site, new GuidDocumentIdProvider()));
}

public override void FeatureDeactivating(SPFeatureReceiverProperties properties)
{
   SPSite site = (SPSite)properties.Feature.Parent;
    DocumentId.SetDefaultProvider(site);
}

Once the custom provider is registered, you will not be allowed to change the prefix in the Document ID Settings page in Site Settings.Which is logical, since you are no longer using the OOTB DocumentIdProvider.

Example source code

I also uploaded an example implementation to the Files section. It includes a SharePoint 2010 project for Visual Studio 2010 (beta2). That project contains the Custom DocumentIdProvider called GuidDocumentIdProvider that uses GUIDs as Document IDs and a feature GuidDocumentIdFeature to enable and disable the GuidDocumentIdProvider.

Code can be found here: Custom DocumentIdProvider Example

When experimenting with this example, make sure you crawl your SharePoint site after adding documents, otherwise the documents can not be found by the DocIdRedir.aspx page.

Kernel-mode authentication performance benefits

Michel Barneveld — Wed, 02 Dec 2009 20:30:00 GMT

2 Weeks ago I wrote a post about kernel-mode authentication and SharePoint 2010. One of my questions was: What is the performance benefit of using kernel-mode authentication?

I did some testing last week and now I had time to analyze the results and create some graphs out of those results.

The test setup

I defined some requirements for my tests:

Test the authentication with a minimum of overhead
Test without IIS or client caching
Automated and repeatable tests
Test with multiple users
Test with multiple threads
Test with different amounts of group memberships
Test with separate client and web server
Test with full stressed web server and not full stressed client
Test with different application pool identities
Test Kerberos and NTLM

Based on those requirements I used the following setup:

Users & Groups:

Created 750 AD users (TestUser0001 to TestUser0750)
Created 100 AD groups (TestGroup0001 to TestGroup0100)
TestUser0001-0250 are member of 100 groups
TestUser0251-0500 are member of 50 groups
TestUser0501-0750 are member of 0 groups
Created account for application pool identity (using Kerberos AES 256 bit encryption).

So I have 3 sets of users with different group memberships (0, 50 and 100 groups)

Web server:

Hyper-V VM
Single CPU with limit of 20% CPU usage on host*
windows 2008R2
4 sites with 4 application pools
each application pool with a different authentication setting
- Negotiate:Kerberos (Kerberos only provider and can only work with kernel-mode off)
- Negotiate with kernel-mode off
- Negotiate with kernel-mode on (SPN registered on machine account)
- Negotiate with kernel-mode on (SPN registered on app pool identity + useApplicationPoolCredentials setting)
1 single HTML file in the root**

*) The web server was way more efficient in decrypting Kerberos tickets than the client was in encrypting them. I needed to lower the CPU speed of the web server to make sure it was at 100% CPU usage while the client was not completely stressed out. Setting it to 20% in Hyper-V slowed down the web server enough to get it at 100% CPU usage, while keeping the CPU usage of the client far enough from 100%.

**) To isolate the performance of authentication I wanted to eliminate the overhead of the .Net framework so I didn't use any aspx files but a plain .HTML file.

Client:

Windows 2008R2 in the same domain (I know that the 10 connection limit were removed in Vista SP2 and Windows 7, but just to be sure I used a server OS as client)
Physical machine with dual core CPU
Custom written .Net test application
- Used a queue of 5000 requests
- Used 32 threads to fire those 5000 requests (I tested with 1,2,4 and 8 threads first, but that couldn't get the server at 100%)
- Used the credentials of 30 different users (I wanted to use the whole set of 250 users, but using more than 40 users made the client app using 100% CPU, so with 30 users I was on the safe side)
- Each thread would ask the queue for the next request and based on the request number (using modulo) it used a different user for the request: User = RequestNumber % 30 (examples: RequestNumber 21->User21, RequestNumber 33 -> User3, etc)
- Request are of form: http://site1.mb.local/test.html?<GUID> (The GUID makes the request unique so prevent client or server side caching)
- The time was captured to complete those 5000 requests
- Each test was run 6 times. The first run was discarded. In the presented results the average of the 5 runs was used and converted to requests per second.
- The user set was configurable, so the tests could be batched with different group memberships.

Test 1 - Kerberos authentication using the ApplicationPoolIdentity

For the first test I configured the 4 application pools to use the built-in ApplicationPoolIdentity as the identity of the application pool. And I registered the SPN's of the 4 URLs to the machine account.

Result:

2 Observations from above graph:

Using kernel-mode authentication (green and purple bars) has a significant improvement over the disabled kernel-mode bars (blue and red)
When the users are member of more groups the overall performance decreases. That is expected because the Kerberos ticket size will increase since each group membership is included in a Kerberos ticket and with more tickets there is more data to decrypt.

Note:
This scenario is not compatible with web farm scenario's. When you have multiple web servers you must use domain accounts for the application pool identity and register the SPN's to that account.

Test 2 - Kerberos authentication using a domain account

For the second test I configured the 4 application pools to use a custom domain account as the identity of the application pool. And I registered the SPN's of 3 URLs to that domain account. The 4th SPN was registered to the machine account, because the Kerberos tickets are decrypted by the machine account when using kernel-mode authentication without the useAppPoolCredentials setting.

The domain account used as the application pool identity didn't have any delegations configured to other services.

Result:

*) Kernel-mode on (the green bar) is not compatible with web farm scenario's, because the SPN is registered to the machine account.

3 Observations from above graph:

Also with this test there is more performance when using kernel-mode.
And also the performance decreases with more group memberships.
The performance of the kernel-mode enabled scenario's is the same as in the first test. However the performance of the kernel-mode disabled scenario's is significant less compared to test 1!

I also captured the request lenght and noticed something odd: The request length of the green bar scenario was much larger (almost doubled) compared to the other 3. The only difference was that this scenario has the SPN registered to the machine account instead of the domain account. After inspection of the Kerberos tickets I discovered the difference: The ticket for the machine account was almost twice as large compared to the ticket for the domain account and there was also an other difference: it has the ok_as_delegate flag. That flag means that the user credentials can be delegated to any other service (TRUSTED_FOR_DELEGATION).

So tickets for SPN's registered to the machine account has the ok_as_delegate flag set. I didn't confirm if it actually allows to delegate to any service (the normal TRUSTED_FOR_DELEGATION setting). But I did test what happens when I gave my domain account the same "delegate to any service" setting. The Kerberos tickets for the domain account became twice as large, just like for the machine account!

So when a service account has the "Delegate to any service" setting set, the Kerberos tickets for SPN's registered to that account will get twice as large. That's an interesting finding!
To put this in perspective: The Kerberos tickets at my current customer are around the 7KB in size (yep large organization with a lot of groups), So 90% of the web request size is this Kerberos tickets. And if you trust this domain account to delegate to any service it is even doubling the request size. That has impact on performance when dealing with large volumes of clients. But since we are only using constrained delegation and don't have any accounts with the delegate to any service option, we don't have these larger tickets.

To see if that delegate to any service has an impact on the performance of the authentication I used a 3rd test:

Test 3 - Kerberos authentication using a domain account with delegate to any service

For the third test I used the same configuration as the second test with the exception that the domain account was trusted for delegation to any service.

Result:

*) Kernel-mode on (the green bar) is not compatible with web farm scenario's, because the SPN is registered to the machine account.

3 Observations from above graph:

Same results as previous tests (kernel-mode is faster and being a member of more groups is slower)
The performance of the green bar scenario is the same as in Test 2. Remember that this scenario has the SPN registered to the machine account. The ticket size remains the same as in Test 2, so this is expected behavior.
The performance of the 3 other scenario's is decreased! So the change to the domain account being trusted for delegation has increased the ticket size and effected the performance, which is expected when you have to decrypt a larger ticket.

Test 4 - NTLM authentication using the ApplicationPoolIdentity

Kerberos was tested in the first 3 tests. But will NTLM authentication have the same result? That's what test 4 and 5 is about.

For test 4, I used 2 sites with 2 application pools and used the NTLM authentication provider. One site is using kernel-mode enabled the other one is disabled. Both application pool use the built-in ApplicationPoolIdentity as application pool identity for this test.

Result:

2 Observations from above graph:

Kernel-mode authentication has also improved performance for NTLM.
The group membership doesn't effect NTLM authentication performance. Which is expected because the groups are not included in the NTLM authorization header as is the case with Kerberos.

Test 5 - NTLM authentication using a domain account

So is there a difference using a domain account as the application pool identity as we saw in the Kerberos tests? This test will give the answer.

The application pools from test 4 were now configured to use a domain account.

Result:

1 Observations from above graph:

Using a domain account as the application pool identity doesn't effect the authentication performance. It's the same result as with test 4.

Conclusions

Using Kernel-mode authentication can give up to a 20% to 25% performance boost.
Note: In real-world scenario's this will most likely be much lower, because above test scenario's were optimized to show the impact of kernel-mode authentication.
Using an application pool identity account that is trusted for delegation will lower the performance boost. Use constrained delegation for maximum performance ( and is more secure btw)
The more groups users are member of the lower the performance of Kerberos authentication.
Using the Kerberos only authentication provider is a good way to enforce the use of Kerberos authentication, but it can't profit of the performance boost since it's not compatible with kernel-mode authentication.
When you can't use kernel-mode authentication (like in the Kerberos only scenario), you will get the most performance with Kerberos using the built-in ApplicationPoolIdentity as the application pool identity instead of a domain account. However this is only possible for single web server scenario's. Web farm Kerberos scenario's require domain accounts as application pool identities.

How to enable Intel VT on Acer Aspire 7720G laptop

Michel Barneveld — Thu, 26 Nov 2009 22:10:00 GMT

My laptop, an Acer Aspire 7720G, has an Intel T7500 Core 2 Duo Mobile CPU. This CPU has support for hardware-assisted Intel® Virtualization Technology (VT). That's the reason why I bought this laptop some years ago. It was a big disappointment when I discovered that Acer turned that option off in the BIOS and didn't offer a way to turn it on. So I couldn't use this laptop to test Hyper-V on Windows 2008 (R2) or XP mode on Windows 7, which both require hardware virtualization. But now there is a solution!!!

I have been searching the Internet for a solution for quite some time now. But today I discovered the solution on this blog: Enabling Intel VT on the Aspire 8930G (and other InsydeH2O-based laptops)
Bigted gave the steps I needed. His steps were for the 5720G, but they also worked for my 7720G.

The steps (credits go to bigted for the original steps and marrio for updated instructions in the comments):

Download the latest BIOS from Acer and flash it to make sure the laptop is running the latest version of the BIOS before proceeding. (link to 7720G bios)
Extract BIOS_v1.45.exe from that zip file.
Now extract the files in BIOS_v1.45.exe using winrar, 7-zip, etc. to a new directory. (right click the file -> extract to... or something like that)
Edit the extracted file platform.ini and change BackupName= to BackupName=acer7720Gdump.fd
Run InsydeFlash.exe as Administrator but CANCEL the flash. We only need it to create a backup of the BIOS in the file acer7720Gdump.fd.
Download the latest version of the 2.6.x version of python and install it. The next steps will not work with a 3.x version of python!!!
Copy MaxL’s vtenable.py (http://pastebin.com/f604e244a) (archived version)into same folder as dump file.
Run “vtenable.py acer7720Gdump.fd acer7720GdumpVT.fd”
Check with a binary comparison tool (like vbindiff) that only 1 byte changed value, in my case from 00 to 01, between the files acer7720Gdump.fd and acer7720GdumpVT.fd
Open the platform.ini again and ...
Change BackupName=acer7720Gdump.fd back to BackupName=
And in the section [Option] change Flag=0 to Flag=1 (Flag 1-> User option mode, including option, start, exit buttons.)
Save the platform.ini
Run InsydeFlash.exe as Administrator.
Click on the button Option
In the Option tab select the new acer7720GdumpVT.fd as the filename
Go to the tab ROM Protection List and select Flash all
Click on OK to close the option screen.
Click on Start
After it's finished it will restart the computer.
Congrats! VT is now enabled!

[edit: 02-Aug-2010]
Update to instructions with improvements from Marrio.
Verified the procedure with the laptop of a friend. It's working!

SharePoint 2010 and Kernel-mode Authentication

Michel Barneveld — Mon, 16 Nov 2009 13:16:00 GMT

I was wondering what authentication settings SharePoint 2010 will use when creating a new Web Application. Because Windows Server 2008 introduced Kernel-mode Authentication and Windows Server 2008 R2 introduced a Kerberos-only authentication provider.

So first a few lines on the what and the where of these settings and after that what SharePoint 2010 uses as default.

Kernel-mode Authentication

This is a new setting in IIS7 (Windows 2008 and Vista SP1) that when enabled increases authentication performance by requiring less switching between user-mode and kernel-mode code.

You can find this setting in Authentication section in IIS.

Then right click "Windows Authentication" and click "Advanced Settings..."

And there you have the toggle box to switch Kernel-mode Authentication on or off.

And according to the description we want to have it enabled: "...improve authentication performance and prevent authentication problems..." ;-)

Kerberos-only authentication provider (Negotiate:Kerberos provider)

Before this setting was introduced if you wanted to use Kerberos, you would use the Negotiate authentication provider instead of NTLM. However the Negotiate provider will fall back to NTLM authentication if Kerberos authentication fails. So there was no way to enforce Kerberos. With this new provider you can enforce Kerberos.It will not fall back to NTLM.

You can find that setting in the same Authentication section as above. However you choose "Providers..." instead of "Advanced Settings...".

And that will give you option to select which Authentication Provider to use:

So there are 3 options: NTLM, Negotiate and Negotiate:Kerberos.

When you select Negotiate:Kerberos and have Kernel-mode authentication turned on, you will get the following warning however:

So this new provider is not compatible with Kernel-mode authentication. NTLM and Negotiate are compatible and will not give this warning.

Negotiate provider and Kernel-mode authentication

Note that the setting of Kernel-mode Authentication will change the behaviour of the Negotiate provider.

If Kernel-mode authentication is enabled, then the Kerberos tickets will be decrypted with the local system account. That means that your SPN must be register to the machine account!!!
If Kernel-mode authentication is disabled, it will show the normal behaviour of decrypting Kerberos tickets with the application pool identity. The SPN must be registered to the used application pool identity then.

"So you change the account you register the SPN on when using Kernel-mode, so what's the deal?"
Well, if you are using a (loadbalanced) multi web frontend SharePoint farm (which is like 95% of the SharePoint farm configurations I do), you must run the application pool on each web server with the same domain account if you want to use Kerberos. The SPN for the loadbalanced url can only be registered to 1 single account and not to 2 or more machine accounts, otherwise Kerberos authentication will fail.
So this would mean that you can't use Kernel-mode authentication when doing Kerberos on a webfarm.

There is a way however to use Kernel-mode authentication and Kerberos authenticated webfarms.
There is a setting which is NOT in the GUI interface of IIS to enable the use of the Application Pool Identity to decrypt the Kerberos tickets. It's explained in the OCS 2007 R2 documentation, which basically tells us to set the useAppPoolCredentials attribute to true of the WindowsAuthentication element in the file %windir%\system32\inetsrv\config\ApplicationHost.config. This look like some undocumented feature because it's not in the documentation

Example of configuration:
<system.webServer>
   <security>
      <authentication>
         <windowsAuthentication enabled="true" useAppPoolCredentials="true" />
      </authentication>
   </security>
</system.webServer>

What are the SharePoint 2010 defaults?

When you create a new webapplication you have the choice of using Negotiate (Kerberos) or NTLM as the authentication provider.

Negotiate (Kerberos)
Before I tested this I was wondering if on a Windows Server 2008 R2 environment the Negotiate:Kerberos (Kerberos only) or the normal Negotiate provider was used.
I created a new web application and it enabled the normal Negotiate and NTLM authentication provider and Kernel-mode authentication was turned off.
That the Kernel-mode was turned off makes some sense: a lot of SharePoint farms will use multiple web frontends and setting a hidden and lesser documented setting in IIS to make it work might create extra support calls But if Kernel-mode was turned off anyway, they could have used the Kerberos only provider. But maybe there is some kind of compatibilty issue?

I have tested SharePoint 2010 in a webfarm with the following 2 scenario's:
1) Negotiate:Kerberos (Kerberos only) and Kernel-mode authentication turned off
2) Negotiate (Normal Negotiate) and Kernel-mode authentication plus the useAppPoolCredentials setting in the ApplicationHost.config file

And they both worked without problems with SharePoint 2010. So I am not sure if there was some design reason behind this. I'll ask some friends at Microsoft about this soon.

NTLM
So now NTLM. When you choose it, it will only enable the NTLM provider and turns off Kernel-mode authentication. Normally Kernel-mode is on by default. But SharePoint turns it off. I have no idea why. I tested it when turned on and it worked with SharePoint 2010. So the SharePoint team probably decided somewhere to turn it off.

Conclusion

SharePoint 2010 (beta 2) does not use the new authentication features offered by Windows Server 2008 and R2. It will turn off the use of Kernel-mode authentication. Hopefully I have some answers soon if this was a design choice. In my testing scenario's SharePoint 2010 can work in Kernel-mode. I haven't discovered any issues yet.

So I am stuck now with a very important question: How much performance improvement are we missing if SharePoint 2010 is not using Kernel-mode authentication?
I already did some testing. Have to write a blog post about it, because the results surprised me! So stay tuned!

The reason why KB911149 and KB908209 are not the solution!

Michel Barneveld — Sat, 14 Nov 2009 21:26:00 GMT

"Do not use CNAME dns records and non default web ports when using Kerberos!"

The last 2 years I have done quite some Kerberos review and troubleshooting sessions (usually in combination with SharePoint) for big customers of Imtech but also for some Microsoft customers. And quite often I tell them the above sentence to not use CNAME dns records and non default web ports when doing Kerberos authentication. So, does that mean that Kerberos doesn't work with CNAME dns records and non default web ports? No, Kerberos can work with that ... in some cases. But usually it complexes the setup and introduces different behaviour for different versions of Windows, Internet Explorer and other client.
And that is the key take away of this blog entry:

If you use HOST A dns records and default web ports (80 and 443) than all version of the Windows OS, Internet Explorer and other Web Clients will show the same consistent behaviour. When you introduce CNAME records or non default ports than different applications or versions will show different behaviour.

A lot of my customers are unaware of the different behaviour, but some are and give a response like:
Customer: "Yeah, we know that. But we have implemented KB911149, KB938305 and/or KB908209 (and maybe even some others) on all our desktops. So we shouldn't have a problem."

So what are these KB articles?

KB911149 and KB938305

KB911149 and KB938305 are fixes for Internet Explorer to use CNAME records with Kerberos authentication. On some versions you need to apply a hotfix, but you always need to enable the feature by setting a registry key.

So what is the problem with CNAME dns records and Kerberos?
A CNAME dns record is an alias to another dns record. So it doesn't point to an IP address but to another record.
Let's say vanity.domain.com is a CNAME for server123.domain.com which is a HOST A record for 1.2.3.4.
Something like: vanity.domain.com -> server123.domain.com -> 1.2.3.4

When you go to the url http://vanity.domain.com you would expect you will request a Kerberos ticket for http/vanity.domain.com, but because it's a CNAME most applications (like IE) will try to get a ticket for http/server123.domain.com.
So if you didn't have the spn http/server123.domain.com registered to the application pool identity it will fail the authentication.

Register the spn to the machinename and finished right?
Well most of the time it's not that simple. If you have multiple websites with each have it's own CNAME dns name pointing to the same hostname and they are using different application pool identities than this will not work. A SPN must be registered to one and only one account. If you have the SPN registered to Account A and request a ticket for that SPN, it will be encrypted by the password hash of Account A. So when your application pool identity is Account B it will receive the ticket from the client, but it can't decrypt it, because it is encrypted with the password hash of Account A and not of Account B.

The KB articles mentioned above will change this behaviour for Internet Explorer. So when you have a CNAME it will request a SPN for that CNAME.
This is a client fix, you need to deploy this to all your clients within the organisation. And note that it will only fix Internet Explorer! Other webclients might not pick this up
Also does this fix not work anymore on Windows 7 with IE8. So IE8 shows the original behaviour. (Microsoft might have changed the way to turn this on. To be honest I haven't looked for a new way. But I did confirm that the way mentioned in the KB articles doesn't work anymore.)

KB908209

This KB908209 article is about fixing Wininet to include the port number when requesting a Kerberos ticket. This is also a client fix and you need to specify in the registry which application should get the new behaviour.
An SPN consists of 3 parts: a service class (http, mssqlsvc, host, etc.) part, a hostname part and an optional port part. If you are using default ports, you don't specify the port.

So if you are running an application on a non default port, for example: http://intranet.domain.local:81 which runs on port 81, you would expect you to register the SPN http/intranet.domain.local:81, which is how you should do it according to the specifications, however Internet Explorer will not request a ticket for that SPN. It ignores the port part and requests it for http/intranet.domain.local which is a different spn and will lead to failed Kerberos authentication.

Then why not register it also without the port part?
This will not work when you have multiple websites running on the same hostname but with different port numbers and using different application pool identities. Because of the same reasons mentioned in the other KB articles above. (See section: Register the spn to the machinename and finished right?)

Why aren't these KB articles the solution?

Let's go back to my dicussion with my customer:

Me: "Do not use CNAME dns records and non default web ports when using Kerberos!"
Customer: "Yeah, we know that. But we have implemented KB911149, KB938305 and/or KB908209 (and maybe even some others) on all our desktops. So we shouldn't have a problem."
Me: "First: You have to deploy that to all your desktops. I rather have a server side fix. But second: it will only fix Internet Explorer (CNAME issue) and some WinInet applications (non default ports issue) including IE."
Customer: "We only use Internet Explorer and no other browsers and everyone is using the same version. So what's the problem?"
Me: "The problem is that you think that Internet Explorer is your only client!"

Be aware here that I usually operate in SharePoint environments. It might be different when dealing in other ecosystems.
With SharePoint we are not dealing with Internet Explorer only. But also with Office Applications, SharePoint designer, WebDAV Redirector, etc. Some of these applications will not be fixed with above articles. But there is even another very import framework that doesn't get fixed by this: The .NET Framework!!!

Every .NET application that does a webrequest to a Kerberos enabled website will not be influenzed by these KB articles. And the .Net framework has the same problems: In most scenario's it doesn't use CNAME records to request SPN's and it will never use the port numbers in the SPN request.

Customer: "But we don't have 3rd party applications or even custom .Net applications! So we are ok, right?"

SharePoint itself was build on the .NET framework. So when you are using Excel Services,SharePoint integrated Reporting Services (with integrated security), RSS Viewer Webpart to view authenticated RSS feeds from different SharePoint farms or from the same farm in a loadbalanced scenario you will run into these problems. Also scenario's where commercial applications written in .NET are connected to SharePoint like for instance K2 workflow will have these problems. And most customers at one point will use tools that will use the webservices of SharePoint and usually those are build in .NET as well.

Are you sure that the .Net Framework are not effected by these KB articles?

Yep. All web requests in .NET will use the System.Net.WebRequest class or descendant classes eventually. This class will not use the WinInet functions to request a Kerberos ticket, but will call the InitializeSecurityContext function and related functions from secur32.dll directly. One of the parameters of that function is pszTargetName. When Kerberos is used that parameter is used to pass the SPN to the function.

So where does the .Net Framework get the SPN from?
It will get that from the method GetComputeSPN(HttpWebRequest httpWebRequest) of the class System.Net.AuthenticationState, which is an internal class. That method will build the SPN based on the HttpWebRequest. There are however 3 distinct code paths that will return different SPN's based on input like: is the url a CNAME or HOST A, do we use a proxy server and is the url a Fully Qualified Domain Name (FQDN).

Below is the source code of this function in version 3.5 of .NET (latest version) to show the different paths. (Thanks to source code tracing of .Net Framework code in Visual Studio )
The 3 paths are:

Not using a proxy server.
Using a proxy server and the Url of the request is an IP Address (e.g.: http://1.2.3.4) or a FQDN (e.g.: http://portal.mb.local)
Using a proxy server and the Url of the request is a short name (e.g.: http://portal)

Each of these 3 code path's uses a different assignment of the returning spn value:

1 (red): spn = httpWebRequest.ServicePoint.Hostname;
2 (blue): spn = httpWebRequest.ChallengedUri.Host;
3 (yellow): spn = Dns.InternalGetHostByName(spn).HostName;

Path 1 and 3 will NOT USE the CNAME. They do a GetHostByName to get the name the CNAME is pointing at. (You can't see it here that path 1 is using a GetHostByName, but this is done in another function)
Path 2 however DOES USE the CNAME.

So below table will summarize which SPN's are created based on the parameters: FQDN, Proxy, CNAME

Url	DNS Type	Using Proxy	Code Path	SPN
http://portal.mb.local	HOST A	No	1	http/portal.mb.local
http://portal.mb.local	HOST A	Yes	2	http/portal.mb.local
http://portal.mb.local	CNAME	No	1	http/hostname.mb.local
http://portal.mb.local	CNAME	Yes	2	http/portal.mb.local
http://portal	HOST A	No	1	http/portal.mb.local
http://portal	HOST A	Yes	3	http/portal.mb.local
http://portal	CNAME	No	1	http/hostname.mb.local
http://portal	CNAME	Yes	3	http/hostname.mb.local

So from above code and table we can at least conclude a few things for the .Net framework:

Ports are never used to generate the SPN! Always uses default ports.
Short names in URL's lead to FQDN names in the SPN!
If you are using a CNAME it will create a SPN for the hostname, unless you are using a FQDN and a proxy server, than it will use the CNAME!

The last point from above list is also the reason why CNAMES sometimes work when they have a proxy server configured, Internet Explorer has the same behaviour! That's also the reason why you shouldn't use Fiddler to debug Kerberos problems, cause it changes the behavour, since Fiddler register itself as a Proxy Server within WinInet. Use a network sniffing tool like the great Network Monitor from Microsoft.

Conclusion

If you use CNAMES and non default ports for web applications you have to be aware of the different behaviours of the different applications. You usually end up doing a lot more configuration and patching clients and servers and having more complex environment. And still it can go *boom* when you introduce new versions or new applications.

When you use HOST A records and default ports all the applications have the same behaviour and your life as administrator just became easier.

Login failed. (Microsoft SQL Server, Error: 18452)

Michel Barneveld — Wed, 11 Nov 2009 19:33:00 GMT

Today I created a new virtual machine to do some SharePoint 2010 testing. First installed Windows 2008 R2, then added and configured the DNS and Active Directory roles. Installed Sql Server 2008 R2. Started the install of SharePoint 2010. Untill this point everything went very smooth. But during the installation you have to give the database name where the SharePoint databases will be created. And since I didn't like the hostname, I wanted a dns entry for it with a better name. Should be simple, right?

Started the dns server management console and added the entry sql.mb.local pointing to my fixed ip. Used the new name in the SharePoint installation screen and got login failures.
Hmm, what is going on here? Is Sql service running? Did I used the wrong ip? Those all checked out fine. So I started the SQL Server Management Studio. I could connect using the hostname and ip addresses but not through the dns name. If I did that I got a nice little box saying: "Login Failed. The login is from an untrusted domain and cannot be used with Windows authentication. (Microsoft SQL Server, Error: 18452)"

Searched for the errors on internet and found some blog postings about enabling Mixed Mode authentication, instead of Windows only authentication. But I was sure it must also work with just Windows authentication. Next step was to look in the eventviewer and there was an event 17806 in the Application Log saying "SSPI handshake failed with error code 0x8009030c while establishing a connection with integrated security; the connection has been closed." and in the Security Log there was an event 4625 saying "An Error occured during Logon".

I also tried it with a CNAME record as alias for the host name and that worked perfectly. I also checked if there were SPN's registered to the sql service account. None were registered. Also checked for MSSQLSvc spn in the domain, also none found. But by this time I was convinced it had to be some kind of Kerberos issue or NTLM reflection protection. But in both cases I am still puzzled why using a HOST A record doesn't work, while the hostname or CNAME does work. But I tried the standard fixes for both problems and they both work (Technically both solutions work for the NTLM reflection protection problem):

Solution 1 - Register a SPN for the SQL Service

Execute the following command to register a SPN

SetSpn -A MSSQLSvc/sql.mb.local:1433 MB\sql_service

(Where MSSQLSvc is the service class for SQL Server, sql.mb.local is my dns entry, 1433 is the port my sql instance is listening on and MB\sql_service is the account sql server is running as)

Solution 2 - Disable NTLM reflection protection

Create a new DWORD with the name DisableLoopbackCheck and value 1 in the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa and restart the computer.

This will disable NTLM reflection protection on the whole machine, which opens it for Man In The Middle attacks, but on my test machine this is ok ;-)
For production machine reference these KB articles for other ways to disable the protection: Users experience authentication issues... and MS08-068: Vulnerability in SMB could allow remote code execution

X-SharePointHealthScore: a new SharePoint 2010 HTTP header

Michel Barneveld — Sun, 08 Nov 2009 14:29:00 GMT

I was testing something with SharePoint 2010 and needed a deeper look on what was actually send over the wire. So I started fiddler and noticed a few HTTP headers in the response that were new to me.

The new headers:

MicrosoftSharePointTeamServices
This header is not really new. It's been there in previous versions of SharePoint. But the version is new ;-)

SPRequestGuid
This turns out to be the Log Correlation Id. With this correlation id you can search the ULS logs for the log lines belonging to this request. Very handy!
It's also mentioned in the Developer Dashboard:

X-SharePointHealthScore
This one got me puzzled for some time where it's getting it's value from. At first I thought it was getting it's value from the new Health Analyzer Report in Central Admin. But changing the amount of problems in the Health Analyzer didn't effect the HealthScore. After finding no references on Internet or the SharePoint SDK, it was time to open Reflector and see what is going on.

What is the X-SharePointHealthScore HTTP Header?

It's a header that returns the systems health (duh) based on these 3 performance counters:

Memory - Available MBytes
ASP.NET - Requests Queued
ASP.NET - Request Wait Time

There are references in the code to the CPU Usage performance counter, but no performance counters are actually being used for that. Maybe some leftover from an earlier build or maybe something that will be added later. And also in the final version of SharePoint there might be other performance counters as well. And maybe we can even add our own counters and thresholds to it.

So how does it calculate the Health Score value?
Every Performance Counter has an table with 10 score buckets associated with it:

For example the Memory Score (the top one of the above 3). If you have more than 1000MB available than the Memory gets a score of 0. If the amount of memory available is between 1000MB and 500MB it will get a score of 1, etc. And if there is less than 20MB of memory available it will get a score of 10. The same applies to the other 2 counters. And the score of these 3 counters are added together and returned as the X-SharePointHealthScore. (So currently it will be a value between 0 and 30)

What does the Health Score mean or do?

One of the new features of SharePoint 2010 is resource throttling when the system is a bit busy handling requests. This can be tuned on at Web Application level in Central Admin. That throttling uses the health score value. If the value becomes 10 or above it will go into throttling mode. At that point it will give GET requests a lower priority than other types like POST. So people can finish the form they are filling in, but new request will be denied.

What can we do with this Health Score?

Besides the obvious for using it to monitor the server. It can also be used in load balancing scenarios (if your loadbalancer can be configured to support this). Based on the healthscores the loadbalancer is able to determine which web front-end has the most ammount of resources left and forward the request to that WFE.

So how do we test the loadbalancer if we set it up in this fashion? Do we need to DDoS a WFE?
There is a better way ;-) There is a registry setting that will override the Health Score and it will return the value from the registry.

Create a DWORD with the name ServerHealthScore in the following location: HKLM\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\14.0\WSS and give it the value you want.

And as a little disclaimer. All above is discovered using reflectoring the beta version of SharePoint. This might be different on the RTM version. ;-)

First post!

Michel Barneveld — Sun, 08 Nov 2009 14:26:00 GMT

Ok, ok, I am finally convinced. Even I should blog ;-)
So the software is setup and ready, let the inspiration begin.

Regards,
Michel.