SharePoint: FIM / MIM fails on Import with stopped-extension-dll-exception or read-error

Posted On January 28, 2019

Consider the following scenario:

You have one of the following User Profile Synchronization configurations for SharePoint:
- SharePoint 2010, which utilizes Forefront Identity Manager (FIM) for User Profile Synchronization.
- SharePoint 2013, using the “Use SharePoint Profile Synchronization” option, which also uses FIM.
- SharePoint 2016 or 2019, using the “Enable External Identity Manager” option, which (typically) uses an external implementation of Microsoft Identity Manager (MIM).
  - Note: MIM is the successor to FIM, and works similarly. However, there are differences in how this problem is reported in each version. See “errors reported” below.
You have configured profile synchronization to either import or export user profile pictures, typically by mapping the pictureURL property to the thumbnailPhoto attribute in Active Directory.
You notice that profile synchronization is not importing new users or updating existing users.

Errors Reported:

SharePoint 2010 / 2013 with FIM:

In the FIM client (miisclient.exe), you notice that the MOSS_FullImport / MOSS_DeltaImport steps fail with “stopped-extension-dll-exception“.
If you look at the Application Event Log on the Synchronization server, you should find an error like this:

Log Name: Application

Source: FIMSynchronizationService

Event ID: 6801

Task Category: Server

Level: Error

Keywords: Classic

Computer: The_Sync_Server

Description:

The extensible extension returned an unsupported error.

The stack trace is:

System.Net.WebException: The remote server returned an error: (404) Not Found.

at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)

at System.Net.WebClient.DownloadData(Uri address)

at Microsoft.Office.Server.UserProfiles.ManagementAgent.ProfileImportExportExtension.DownloadPictures(ProfileChangeData[] profiles)

at Microsoft.Office.Server.UserProfiles.ManagementAgent.ProfileImportExportExtension.Microsoft.MetadirectoryServices.IMAExtensibleFileImport.GenerateImportFile(String fileName, String connectTo, String user, String password, ConfigParameterCollection configParameters, Boolean fFullImport, TypeDescription

Notes:

The error is not always “404 Not Found”. That’s just the most common scenario. It could be any exception for the FIM service trying to complete an HTTP GET for a profile picture thumbnail. As a rule, if the MOSS_FullImport or MOSS_DeltaImport steps are failing with “stopped-extension-dll-exception” and you see a 6801 error from FIMSynchronizationService in the app log with ProfileImportExportExtension.DownloadPictures in the stack trace, you are running into some version of this issue.

Here are some alternate stack traces for the 6801 FIMSynchronizationService error that could be thrown:

System.Net.WebException: The remote server returned an error: (401) Unauthorized.

at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)

at System.Net.WebClient.DownloadData(Uri address)

at Microsoft.Office.Server.UserProfiles.ManagementAgent.ProfileImportExportExtension.DownloadPictures(ProfileChangeData[] profiles, Boolean delayErrors)

System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)

at System.Net.WebClient.DownloadData(Uri address)

at Microsoft.Office.Server.UserProfiles.ManagementAgent.ProfileImportExportExtension.DownloadPictures(ProfileChangeData[] profiles, Boolean delayErrors)

System.Net.WebException: The remote server returned an error: (403) Forbidden.

at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)

at System.Net.WebClient.DownloadData(Uri address)

at Microsoft.Office.Server.UserProfiles.ManagementAgent.ProfileImportExportExtension.DownloadPictures(ProfileChangeData[] profiles)

Note: For the above 403 – Forbidden condition, see my other post here: https://joshroark.com/sharepoint-issues-with-profile-pictures-when-mysite-uses-saml-auth/

SharePoint 2016 / 2019 with MIM:

With MIM, the errors are much more subtle. There is no outright failure of an entire Sync step. Instead, you’ll see:

“Completed-discovery-errors” for either the FULLIMPORT or DELTAIMPORT steps for the SharePoint Management Agent (SPMA).
In the Discovery Errors pane, it doesn’t even list which users had the problem. It just shows generic entries (ex: entry 5) with a “read-error“.

Cause:

During the DeltaImport and FullImport steps, FIM / MIM needs to get each users profile picture. It does so by making a regular HTTP GET call to the URL listed in the PictureURL property for that user. For example: http://MySiteRoot/User Photos/Profile Pictures/UserName_LThumb.jpg. If this HTTP GET fails for any reason, the user will fail to be updated. MIM appears to be more forgiving and moves on to process other users.

However, with the version of FIM built into SharePoint 2010 and 2013, if it fails to get picture thumbnail, even for a single user, it causes the entire Sync step to fail. Since the entire step fails to import users from the SharePoint side, no user profiles can be updated. As such, a problem with a single picture thumbnail can cause the entire sync to fail for all users.

In the most common scenario (404), the web request to download the picture fails with a 404 (not found) error because the picture thumbnail no longer exists at the location the link is pointing to. It may have been deleted, or the host name part of the URL may have changed. For example, if the mysite host URL was changed from http://MySiteRoot to https://my.contoso.com, any users that still have a profile picture pointing to the old URL will fail (assuming the old URL no longer resolves).

How do we fix this?

You need to identify the user profile or profiles that have this problem. The Application Event Log will not tell you which users are problematic, and neither will the SharePoint ULS logs. You could review the IIS logs for the Mysite web application and look for 404 responses to LThumb.jpg and MThumb.jpg requests, or configure Fiddler to capture the FIM / MIM traffic, but if there are multiple users with this problem, you will likely only be able to find the first problem profile, and be forced to keep running Syncs and capturing new data until you find them all.

To detect the problematic profiles, there are a couple of options:

You can use this SQL query on the Profile database to see which user profiles have a picture set in SharePoint:

2010 / 2013 version:

select RecordID, Bdeleted, NTName, PreferredName, Email, PictureUrl from UserProfile_Full where PictureUrl like ‘%http%’ order by PictureURL

2016 / 2019 version:

select RecordID, Bdeleted, NTName, PreferredName, Email, PictureUrl from upa.UserProfile_Full where PictureUrl like ‘%http%’order by PictureURL

Since it’s sorted by pictureURL, any URLs that use a different host name will be at the top or bottom of the query results. If there are any that use a host name other than your current MySite host name, those are highly suspect. To test a given Picture URL, you can just paste it in your browsers address bar. The browser should be able to fetch the picture thumbnail and display it. For the most accurate results, and to account for any network problems, you would want to browse to the picture URL from the Sync server.

If you find that the problem is that the mysite host name was changed, and the old host name no longer resolves, you can update those pictureURL links with the Update-SPProfilePhotoStore commandlet, using the “OldBaseURI”, and “NewBaseURI” switches. Example:

Update-SPProfilePhotoStore -MySiteHostLocation https://my.contoso.com -OldBaseUri http://MySiteRoot -NewBaseUri https://my.contoso.com

If nothing sticks out as wrong after reviewing the results from the SQL query above and testing a few URLs, you’ll have to test each thumbnail individually to find the problematic ones. I wrote the PowerShell script shown in the “Script” section below for this purpose. The idea is that it mimics what the Sync engine does. It does an HTTP GET for each pictureURL that is currently set in SharePoint. This is how you use it:

1. Save the PowerShell script as “CheckProfilePictures.ps1” and move it on to your FIM Sync server at the root of the C: drive.

2. Log into the Sync server as the Farm account.

3. Run the script like this: .\CheckProfilePictures.ps1 -url http://mysite -filepath c:\logs

— Where “http://mysite” is the URL for your MySite host, and “c:\logs” where you want it to create the output log file.

Important: Normally, we’d want to run this script from the Sync server. However, if you are using MIM with SharePoint 2016 or 2019, the MIM server is not part of the SharePoint farm, and the script will not work. In that case, you will need to choose a SharePoint server to run it from. Please note that the results may be less valid because, in that case, the script cannot account for networking problems between MIM and SharePoint.

Expected output:

When it’s done, the console will give you a status update and alert you to any problems:

And it will automatically open the log file ex: “c:\logs\ProblemPicLog.txt” in Notepad.

If it finds any profiles with problem picture URLs, it will prompt you to decide if you want to automatically remove the picture for those profiles. Just answer Y or N.

WARNING: It generally only makes sense to remove the identified picture thumbnails if there are a small number of them failing with 404 – Not Found. If they are failing with any other error, it could be a permission, networking, or configuration problem. Removing the thumbnails in those cases not only will not solve the problem, it will result in data loss.

Once all the identified problem picture thumbnails have been corrected, you’ll want to run a Full Profile Sync.

Script:

WARNING: Please do not just run this script and have it delete all of the picture thumbnails it finds as being problematic without further consideration (It will pop up a log file and prompt you Y or N to do the delete). It marks picture thumbnails as being problematic if it fails to get them for any reason. Having the script delete the pictures can result in data loss. Please review the log file it creates and verify that it makes sense to have the script delete the identified picture thumbnails before doing so. You can always answer “N” on the first run, review the log file and run the script again.

# Title: CheckProfilePictures.ps1
# Disclaimer: This script is provided as-is with no warranties expressed or implied. Please make sure you have good and current backups. Use at your own risk.
# Author: Joroar
# You need to specify the mysite URL and path for log file. You run it like so:
# .\CheckProfilePictures.ps1 -url http://mysite -filepath c:\logs
# When it’s done, it will automatically open the log file ex:”c:\logs\ProblemPicLog.txt” in Notepad (if it finds any problems)
# If it finds any profiles with problem picture thumbnail URLs, it will prompt you to decide if…
# …you want to automatically remove the picture for those profiles. Just answer Y or N.
# WARNING: It generally only makes sense to remove the identified picture thumbnails if they are failing with 404 – Not Found.
Param(
[Parameter(Mandatory=$True,Position=1)]
[string]$URL,
[Parameter(Mandatory=$True)]
[string]$filePath
)
$logfile = “$filePath\ProblemPicLog.txt”
[array]$problemProfiles = @()
[array]$problemProfileReport = @()
[array]$ProfilesWithPics = @()
$site = Get-SPSite $url
if ($site) {Write-Host “Successfully obtained site reference”}
else {Write-Host “Failed to obtain site reference. Check the URL you entered”}
$serviceContext = Get-SPServiceContext($site)
if ($serviceContext) {Write-Host “Successfully obtained service context”}
else {Write-Host “Failed to obtain service context”}
$upm = new-object Microsoft.Office.Server.UserProfiles.UserProfileManager($serviceContext)
if ($upm) {Write-Host “Successfully obtained user profile manager”}
else {Write-Host “Failed to obtain user profile manager. Make sure the site you entered is associated with a UPA”}
# Go get all the profiles with a picture set and store their login name and picture URL in an array
$numpics = 0
Write-Host “Working on grabbing the profiles. This may take a while if you have a large number of profiles with pictures…” -ForegroundColor green
$profiles = $upm.GetEnumerator()
foreach($user in $profiles)
{
if ($user[“PictureURL”].Value)
{$numpics++
$pic = $user[“PictureURL”].Value
$ProfilesWithPics += ,@($user.MultiloginAccounts, $pic)
}
}
Write-Host “There appears to be ” $numpics ” profiles with pictures set” -ForegroundColor green
Function Hit-AllProfilePictures-Full
{
Write-Host “Doing a FULL check for all Mthumbs and Lthumnbs by doing an HTTP GET request” -ForegroundColor green
Write-Host “Working on it…” -ForegroundColor green
$date = Get-Date -Format U
“Started Full picture check at ” + $date + ” (UTC time)” | out-file $logfile -append
“===============================================” | out-file $logfile -append
$progressCounter = 0
$countProblemProfiles = 0
#Loop through the array and do the file name comparison
for ($a=0; $a -lt $ProfilesWithPics.length; $a++)
{ $progressCounter++
Write-Progress -Activity “Checking Picture Thumbnails” -status “Checking $progressCounter” -percentComplete ($progressCounter / $ProfilesWithPics.length*100)
$UName = $ProfilesWithPics[$a][0]
$PUrl = $ProfilesWithPics[$a][1]
$mthumbURL = $PUrl
$lthumbURL = $mthumbURL.Replace(“MThumb”, “LThumb”)
# Do a web request to the Mthumb URL we found for the profile
“Hitting picture ” + $mthumbURL | out-file $logfile -append
$date = Get-Date -Format U
$date + ” (UTC time)” | out-file $logfile -append
Try
{ $request = [System.Net.WebRequest]::Create($mthumbURL)
$request.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
$request.proxy = [System.Net.WebRequest]::DefaultWebProxy
$request.ContentType = “application/x-www-form-urlencoded”
$request.Method = “GET”
“Response = ” + $request.GetResponse().StatusCode | out-file $logfile -append
” ” | out-file $logfile -append
}
catch [Exception]
{ “ERROR!!! for user: ” + $UName + ” — ” + $_.Exception.Message | out-file $logfile -append
” ” | out-file $logfile -append
$problemProfiles += $UName
$problemProfileReport += $UName, $mthumbURL, $_.Exception, ” ”
$countProblemProfiles++
}
# Now do a web request to the Lthumb URL we found for the profile
“Hitting picture ” + $LthumbURL | out-file $logfile -append
$date = Get-Date -Format U
$date + ” (UTC time)” | out-file $logfile -append
Try
{ $request = [System.Net.WebRequest]::Create($LthumbURL)
$request.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
$request.proxy = [System.Net.WebRequest]::DefaultWebProxy
$request.ContentType = “application/x-www-form-urlencoded”
$request.Method = “GET”
“Response = ” + $request.GetResponse().StatusCode | out-file $logfile -append
” ” | out-file $logfile -append
}
catch [Exception]
{ “ERROR!!! for user: ” + $UName + ” — ” + $_.Exception.Message | out-file $logfile -append
” ” | out-file $logfile -append
$problemProfiles += $UName
$problemProfileReport += $UName, $LthumbURL, $_.Exception, ” ”
$countProblemProfiles++
}
}
if ($problemProfileReport)
{“In summary, these are your problem profiles: ” | out-file $logfile -append
“———————————————-” | out-file $logfile -append
ForEach ($profileReport in $problemProfileReport)
{$profileReport | out-file $logfile -append
}
Write-Host “Done. ” $countProblemProfiles “missing thumbnails were identified. The log file you specified in” $logfile “should open in Notepad. Please review the results before proceeding. A summary of all the errors is at the bottom.” -foregroundcolor red
Write-Host “Total number of profiles with pictures processed = ” $progressCounter -foregroundcolor green
Start-Process notepad -WindowStyle normal -FilePath $logfile
[String]$YN = Read-Host “Want to remove the picture for the problem profiles that were identified? Y\N”
if($YN -ieq “y” -or $YN -ieq “yes” )
{write-host “You chose Yes” -foregroundcolor red
write-host “Removing problematic profile picture links…” -foregroundcolor red
Remove-ProblemPictures
}
else{write-host “You chose No” -foregroundcolor red
write-host “No pictures were automatically removed. You should manually verify the profile picture URLs for the identified problem profiles are correct. The Sync service needs to be able to access these with an HTTP GET.” -foregroundcolor red
}
}
else
{Write-Host “Good News! There were no problem profiles detected.” -foregroundcolor green
Write-Host “Total number of profiles with pictures processed = ” $progressCounter -foregroundcolor green
Write-Host “You can have a look at the log file you specified in” $logfile -foregroundcolor green
}
}
Function Remove-ProblemPictures()
{if ($problemprofiles)
{
ForEach ($problemprofile in $problemProfiles)
{write-host “Removing the picture link for profile: ” $problemprofile
$userProfile = $upm.GetUserProfile($problemprofile)
$ProfilePictureUrl = “”
$userProfile[“PictureURL”].Value = $ProfilePictureUrl
$userProfile.Commit()
}
Write-Host “Done. The broken picture links should be cleared up. You can wait few minutes (it takes a minute or two to remove the values from cache) and then try running this script again, or just run a Full Sync.” -foregroundcolor green
}
Else{Write-Host “No problem picture links to remove!” -foregroundcolor red}
}
Hit-AllProfilePictures-Full($url)

Add a Comment

You must be logged in to post a comment.