SharePoint: All about one-time timer jobs

Posted On April 7, 2019

Update 9/15/19: Added some clarifying info, and some PowerShell that can be used to delete any “stuck” one-time timer jobs.

One-time timer jobs are created on the fly, should run immediately, and then disappear when they are done doing whatever they were supposed to do.  If you have one-time timer jobs hanging around, you have a SharePoint Timer service (owstimer.exe) problem.

You can easily find any one-time timer jobs by doing this:
Go to Central Administration | Monitoring | Review Job Definitions
Click the Schedule Type column to sort by that column.
Click it again to sort descending.
Scroll down past the Weekly timer jobs.  Any one-time jobs should be between Weekly and Monthly.


You can also use this PowerShell to identify one-time timer jobs:

Get-SPTimerJob | ?{$_.schedule.description -eq "One-time"} |select displayname,server,locktype,lastruntime | fl

Here’s some PowerShell you can use to remove all of the currently pending (“stuck”) one-time timer jobs. However, before you do that, you should first run through all the steps in this article. You may be able to get the timer service working properly again, which may (depending on the job) be able to pick up the pending jobs and complete them.

# Delete all one-time timer jobs:
$tjs = Get-SPTimerJob | ?{$_.schedule.description -eq "One-time"} 
foreach ($tj in $tjs)
{$tj.delete()}

In most cases when you complete some action in Central Administration that must be replicated to all the servers in the farm, a one-time timer job is used to do it. I don’t pretend to know all the situations where one-time timer jobs are used, but here are some common scenarios, listed with the name of the one-time timer job it creates:

  • Starting the User Profile Synchronization Service: “ProfileSynchronizationSetupJob”
  • Extending a Web Application or creating a new one: “Provisioning Web Application <web app name>”
  • Creating new service applications: “Service Application Instance Provisioning Job”
  • Updating service account passwords: “Password Change Event”
  • Changing the SharePoint Logging levels: “Microsoft SharePoint Foundation Diagnostics Service Configuration”
  • Deploying farm-level solutions, including using the Install-SPSolution command: Microsoft SharePoint Foundation Solution Deployment for “solutionName.wsp”
  • Retracting farm-level solutions: Microsoft SharePoint Foundation Solution Retraction for “solutionName.wsp”
  • Starting or stopping services on any server other than the Central Admin box: “Provisioning <the name of the service> service on <serverName>” Ex: “Provisioning App Management Service service on ContosoAPP01”

— In general, most farm administration tasks that include work that must be completed on other servers in the farm will use a one-time timer job to do it.

So what to do if one-time timer jobs are NOT running?

If one-time timer jobs are not running, it’s usually one of three problems:

1.  The Timer and / or Administration service “service instance object” is offline / disabled:

Public article about this issue for reference:
https://support.microsoft.com/en-us/help/2616609/administrative-timer-jobs-not-running-after-upgrade

— Here’s my take on a PowerShell Script to enable the Timer and Administration service instances. It also outputs the status of the AllowServiceJobs and AllowContentDatabaseJobs properties for each box, which is useful for SharePoint 2016 and above:

#Title: CheckTimerAndAdminServices.ps1
#Author: Joroar; Et al
#Date: 4/7/19
#Make sure the Timer service and Administration service instances are all online
Add-PSSnapin microsoft.sharepoint.powershell -ErrorAction SilentlyContinue
$farm = Get-SPFarm
#Check SharePoint TIMER
write-host "Checking SharePoint TIMER Service..."
$FarmTimers = $farm.TimerService.Instances
foreach ($FT in $FarmTimers){write-host "Server: " $FT.Server.Name.ToString(); write-host "Status: " $FT.status; write-host "Allow Service Jobs: " $FT.AllowServiceJobs; write-host "Allow Content DB Jobs: " $FT.AllowContentDatabaseJobs;"`n"}
#Set any that are Disabed to Online
$disabledTimers = $farm.TimerService.Instances | ? {$_.Status -ne "Online"}
if ($disabledTimers -ne $null)
{foreach ($timer in $disabledTimers)
{Write-Host -ForegroundColor Red "Timer service instance on server " $timer.Server.Name " is NOT Online. Current status:" $timer.Status
Write-Host -ForegroundColor Green "Attempting to set the status of the service instance to online…"
$timer.Status = [Microsoft.SharePoint.Administration.SPObjectStatus]::Online
$timer.Update()
write-host -ForegroundColor Red "You MUST now go restart the SharePoint Timer service (in services.msc) on server " $timer.Server.Name "`n"}}
else{Write-Host -ForegroundColor Green  "All Timer Service Instances in the farm are online. No problems found with SPTimerV4."}    
#Check SharePoint ADMIN
Write-host "Now checking SharePoint ADMINISTRATION Service...`n"
$servers = $farm.Servers
foreach ($server in $servers)
{$AdminService = $server.serviceinstances | ? {$_.TypeName  -eq "Microsoft SharePoint Foundation Administration"}
if($AdminService)
{write-host “Server: ” $AdminService.Server.Name.ToString(); write-host “Status: ” $AdminService.status; "`n"}}
#Set any that are Disabed to Online
$disabledAdmins = $farm.servers.serviceinstances | ? {$_.TypeName  -eq "Microsoft SharePoint Foundation Administration" -and $_.Status -ne "Online"}
if($disabledAdmins)
{foreach ($disabledAdmin in $disabledAdmins)
{Write-Host -ForegroundColor Red "Administration service instance on server " $disabledAdmin.Server.Name.ToString() " is NOT Online. Current status:" $disabledAdmin.status
Write-Host -ForegroundColor Green "Attempting to enable the Administration service..."
$disabledAdmin.Status = [Microsoft.SharePoint.Administration.SPObjectStatus]::Online
$disabledAdmin.update()
write-host -ForegroundColor Red "You MUST now go restart the SharePoint Administration service (in services.msc) on server " $disabledAdmin.Server.Name.ToString() "`n"}}
else {write-host -ForegroundColor Green "All Administration Service Instances in the farm are online. No problems found with SPAdminV4."}
#End Script

2. A problem with the SharePoint Administration service.

The SharePoint Administration service (aka: SPAdminV4, WSSADMIN.EXE) is also involved in the execution of one-time timer jobs.  Make sure that this service is running on every server in the farm and the log on account must be “Local System”.  
— It doesn’t hurt to restart the Administration service and the Timer service just to make sure nothing is hung up there.

You may also be able to force one-time timer jobs to run on the local server by temporarily stopping the SharePoint Administration service and then running this STSADM command:
stsadm -o execadmsvcjobs
You’ll want to restart the Administration service after that completes.

3. Bloated TimerJobHistory table (like millions of rows).

If the TimerJobHistory table (in the Configuration database) accumulates a large number of records (millions) over time and the “Delete Job History” timer job is not able to keep up with the number of records being added, one-time timer jobs will fail to run.

To check for this issue, run the following two SQL queries against the Configuration database:

-- Show total rows:
select COUNT(*) from TimerJobHistory (nolock)

-- Show oldest records
select top 10 ID, StartTime, EndTime from TimerJobHistory (nolock)
order by StartTime

Or, if you don’t have access to your SQL server to run direct queries, you can run this PowerShell to give you the same information.  Be sure to be logged on as the Farm service account, or another account that has access to the Configuration database:

# Check Timer job history table via SQL query using PowerShell
# This PowerShell script is provided “as-is” with no warranties expressed or implied. Use at your own risk.
Add-PSSnapin microsoft.sharepoint.powershell -ErrorAction SilentlyContinue
$conn = New-Object System.Data.SqlClient.SqlConnection
$conn1 = New-Object System.Data.SqlClient.SqlConnection
$cmd = New-Object System.Data.SqlClient.SqlCommand
$cmd1 = New-Object System.Data.SqlClient.SqlCommand
$configDb = Get-SPDatabase | ?{$_.TypeName -match “Configuration Database”}
$connectionString = $configDb.DatabaseConnectionString
$conn.ConnectionString = $connectionString
$conn.Open()
$cmd.connection = $conn
$conn1.ConnectionString = $connectionString
$conn1.Open()
$cmd1.connection = $conn1
Write-Host “Issuing Query on: ” $configDb.Name “”
“”
$cmd.CommandText = “select COUNT(*) from TimerJobHistory (nolock)”
$rows = $cmd.ExecuteReader()
if($rows.HasRows -eq $true)
{while($rows.Read())
{“Timer Job History Table contains: ” + $rows[0]+ ” rows”}
“”}
$rows.Close()
$conn.Close()
$cmd1.CommandText = “select top 10 ID, StartTime, EndTime from TimerJobHistory (nolock) order by StartTime”
$rows1 = $cmd1.ExecuteReader()
if($rows1.HasRows -eq $true)
{“Top 10 Oldest Records in Timer Job History Table: ”
“”
while($rows1.Read())
{$rows1[0].ToString() + ” ~ ” + $rows1[1].ToString() + ” ~ ” + $rows1[2].ToString()
}}
$rows1.Close()
$conn1.Close()

Then, run the following PowerShell on one of the SharePoint servers to get information about the job that is supposed to keep TimerJobHistory cleaned up:

Get-SPTimerJob | ?{$_.name -eq "job-delete-job-history"} | fl

Notes:

  • Getting a record count (SQL query 1) is a good indicator.  If it’s several million rows, that’s a pretty sure bet it’s causing problems.*
    • * In most farms anyway… in very large farms with many servers, it can be normal to have a few million rows in that table after just a few days.
  • The second query will show you the oldest records in that table.  By default, they should be no older than 7 days**.
  • The PowerShell output of “job-delete-job-history” is to verify that the cleanup job is enabled and what “DaysToKeepHistory” is set to.  It should be 7**.

If this is the problem, you have a couple options:

  • Figure out what is wrong with the “job-delete-job-history” timer job and hopefully it will be able to catch up and clean up that table to a reasonable number of rows.
  • Contact Microsoft Support.  We have ways of manually cleaning it up.  I’m not going to detail that method here because it’s unsupported for anyone except support engineers with the proper approvals to complete.

Update! The November 2017 CU (build 15.0.4981.1002) changed the behavior around TimerJobHistory table cleanup. Essentially, it’s done as part of every timer job. However, if you already have a bloated TimerJobHistory table at time of upgrade, it can initially cause more problems than it solves. See Stefans blog post about it.

**Update 2! The April 2018 CU (build 15.0.5023.1001) changed the behavior back to the original reliance on “job-delete-job-history”. However, now the maximum history is 3 days instead of 7. See Stefans other blog post about that.

One Comment

Add a Comment