STS Background:

In SharePoint 2010, 2013, 2016, etc, the Security Token Service (STS) is a web service hosted under the “SharePoint Web Services” IIS site on HTTP port 32843 and HTTPS port 32844, in a virtual directory called SecurityTokenServiceApplication.

In SharePoint 2010, it contains 2 web services:
Securitytoken.svc
Windowstokencache.svc

In SharePoint 2013 and 2016, it contains 3 web services:
Appsts.svc
Securitytoken.svc
Windowstokencache.svc

• In SharePoint, the STS is used to generate security tokens for claims-based authentication purposes.
• It is only invoked locally, meaning it must be running and healthy on every server in the farm.
• Even when using “classic” authentication for your web applications, claims authentication is still used for intra-farm communication. So even if you’re not using claims to authenticate your users, you still need a healthy STS.

Investigating the STS:

When investigating STS problems, you should look at the ULS logs, Application Event log (event viewer), and even Central Administration.

The built-in SharePoint Health Analyzer has a rule for testing STS functionality. It’s called “The Security Token Service is not available”. It runs hourly and will report if the STS fails on any of your servers.

In the Application Event log, you may see event ID 8306 with the following description:

An exception occurred when trying to issue security token: The server was unable to process the request due to an internal error.  For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs.

In the SharePoint ULS logs, you may see similar generic errors like this:

OWSTIMER.EXE (0x0FF8) 0x0FC8 SharePoint Foundation Claims Authentication fsq7 High Request for security token failed with exception: System.ServiceModel.FaultException: The server was unable to process the request due to an internal error.  For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs.     at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.ReadResponse(Message response)     at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst, RequestSecurityTokenResponse& rstr)     at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst)     at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForContext(Uri context, Boolean bearerToken, SecurityToken onBehalfOf, SecurityToken actAs, SecurityToken delegateTo)

OWSTIMER.EXE (0x0FF8)              0x0FC8  SharePoint Foundation Claims Authentication    8306                Critical   An exception occurred when trying to issue security token: The server was unable to process the request due to an internal error.  For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs.

To get a more descriptive error, you need to update the STS web.config (typically stored in C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\WebServices\SecurityToken) to give you more detail.
You do that by adding the following within the <behavior name=”SecurityTokenServiceBehavior” > section:

<serviceDebug includeExceptionDetailInFaults="True" httpHelpPageEnabled="True"/>

Example:

Testing the STS:

You should start by just browsing to the STS web service.  This won’t test if the STS can actually issue tokens, but will validate whether or not the web service is “up” in IIS and available for requests.

Open IIS Manger, expand SharePoint Web Services, click SecurityTokenServiceApplication. 
Switch to Content View.  Right click on securitytoken.svc and choose Browse.
It should open the following in a browser:

http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc

You should also try the HTTPS version:
https://localhost:32844/SecurityTokenServiceApplication/securitytoken.svc

Note: In SharePoint 2010, when you browse to the STS web service, it will throw this error:
The server was unable to process the request due to an internal error. For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs.

This is completely normal for SharePoint 2010 servers and does not indicate a problem.

In SharePoint 2013 and 2016, it should return a web service page with links to the wsdl and examples like this:

Testing token creation:

Now that you’ve tested general connectivity and IIS configuration for the STS, you can move on to testing if the service is able to issue security tokens.  You can do that by running this PowerShell script:

Add-PSSnapin *sharepoint*
$farm = [Microsoft.SharePoint.Administration.SPFarm]::Local 
$webServiceCollection = new-object Microsoft.SharePoint.Administration.SPWebServiceCollection($farm) 
foreach ($service in $webServiceCollection) 
{ foreach ($webApp in $service.WebApplications) 
{ $firstWebApp = $webApp 
#Get the context 
$context = $firstWebApp.GetResponseUri([Microsoft.SharePoint.Administration.SPUrlZone]::Default) 
Write-Host "Web Application Context:" $context.AbsoluteUri 
#Call the token generator function 
$token = [Microsoft.SharePoint.SPSecurityContext]::SecurityTokenForContext($context) 
Write-Host "Token:" $token.InternalTokenReference 
Write-Host "**************************" } }

This script will get each web application in the farm, and then contact the STS to issue a new security token within the context of that web app.
When successful, it should produce output like this:

Web Application Context: https://teams.contoso.com/
Token: SamlAssertionKeyIdentifierClause(AssertionId = ‘_d47eb32c-4ff5-4b6c-86e1-12c18c9be8fa’)
**************************
Web Application Context: http://mysite.contoso.com/
Token: SamlAssertionKeyIdentifierClause(AssertionId = ‘_78bc4df4-062f-45ed-978f-0fc646ffccda’)
**************************
Web Application Context: http://ourweb.contoso.com/
Token: SamlAssertionKeyIdentifierClause(AssertionId = ‘_a7e31b81-0ec5-4bc1-9ef0-f757647a74c5’)
**************************

If this script throws errors, that means the STS on that server cannot issue security tokens.

If you enabled IncludeExceptionDetailInFaults in the STS web.config like mentioned above, it should give you a decent error to research and fix, like the one detailed in the section below.

A common issue:

I’ve seen this a few times. A custom claims provider is installed in the farm, but is only deployed to the servers that are running the “Microsoft SharePoint Foundation Web Application” service.

When other servers call their STS, it tries to load the custom claims provider, but fails because the custom DLL is not available on the box. Here’s an example of what you’d see in the ULS logs, or Application Event log after enabling the IncludeExceptionDetailInFaults flag in the STS web.config.

w3wp.exe (0x46AC) 0x4634 SharePoint Foundation Claims Authentication fo1t Monitorable STS Call: Failed to issue new security token. Exception: System.IO.FileNotFoundException: Could not load file or assembly ‘LDAPCP, Version=1.0.0.0, Culture=neutral, PublicKeyToken=c0e8a57fc919aedb’ or one of its dependencies. The system cannot find the file specified.  File name: ‘LDAPCP, Version=1.0.0.0, Culture=neutral, PublicKeyToken=c0e8a57fc919aedb’     at <stack truncated>

In this case, custom claims provider LDAPCP was deployed, but not to all servers in the farm.

Ideally, the farm solution would be updated to deploy to all servers in the farm and be redeployed, but there are cases where you want to skip all that headache and just get the DLL on the box so that the STS will work.

The custom DLL needs to be copied to the box and added to the Global Assembly Cache (GAC). In the past we’ve used Gacutil.exe, and that works, but it can be hard to find and get on the box.

Instead, you can use the following PowerShell to add a DLL to the GAC:

#Note that you should be running PowerShell as an Administrator to run this.
[System.Reflection.Assembly]::Load("System.EnterpriseServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a")
$publish = New-Object System.EnterpriseServices.Internal.Publish
$publish.GacInstall("C:\temp\LDAPCP.dll")

After adding the custom DLL to the GAC, you’ll want to reset IIS or recycle the application pool for the STS to make sure it gets loaded.

You may ask: Where can I get the custom claims provider DLL in the first place?

You can get it from another box in the farm, or you can actually extract it from the installed farm solution. You can export a farm solution back to a WSP file on the file system like this:

$Farm = get-spfarm
$solution = $farm.solutions | ? {$_.name -eq "LDAPCP.wsp"}
$solution.SolutionFile.SaveAs("c:\" + $solution.Name) 

Then you can rename the .wsp file to .cab, extract the files and grab the DLL.