Maintenance Mode

Looking for a good way to automate putting machines in maintenance mode while patching, firmware upgrades, software maintenance is being performed. The list can be quite large and changes from month to month. We typically on the SCCM side get a list of machines and a window every month when we can perform maintenance we use right click tools to add that list to a collection and then perform our patching and after that hardware team is free to do any firmware updates and then app owners any application work so just using the check boxes in SCCM is not practical. Having maybe hundreds of servers going at once makes having individuals put them in maintenance mode is Squared Up is impractical. My last job I did a runbook in scorch but I do not currently have SCORCH here. Looking for ideas and thoughts

1 Like

Hi Rick,

I read something about it, but did not try it yet:

https://systemcenter.wiki/?GetCategory=SCCM+Service+Window+to+SCOM+Maintenance+Mode+Automation

SCCM Service Window to SCOM Maintenance Mode Automation
This management pack was designed with two parts. These parts, contained in a single Rule and one Data Source Method, were broken up to provide System Center Operations Manager with the ability of automatically placing a system into Maintenance Mode using System Center Configuration Manager Service Windows.

These two parts enable the ability to automatically trigger the Operations Manager Maintenance Mode function based upon the Configuration Manager Service Windows found in the Configuration Manager Agent’s WMI stack. The scheduled “SCOM Agent Initiated Maintenance Mode Rule for Active SCCM Service Windows” rule, accounts for Service Window overlap, calculates the total service window duration and sends property bag items from the Operations Manager Agent directly to the Management Server that the agent is connected to.

 

Please share your experience :slight_smile:

1 Like

If your SCCM team uses SCCM right click tools, I have a GUI I made with PowerShell studio you can adapt. Basically we compile the script into an exec and the SCCM team sends parameters to it from an SCCM collection via the right click tools.

It has a few other features like create/remote VMware snapshots, etc. Of course, the code comes with no support or warranty (use at your own risk).

I’m pretty sure we have a script which reads the MW config from SCCM and creates maintenance schedules in SCOM which match them.

I’ll see if I can track down the script tomorrow.

We actually have three different scripts that we use to achieve this goal, but this is partly because of how our Mainenance Windows are targetted in SCCM.

In our case we use AD Groups to control which MW a Server falls under. These groups are then used within SCCM to ensure that the appropriate schedule is adhered to. For us the goal we to make sure that we applied the same maintenance schedule in SCOM to make sure we avoided false positives whilst patching was taking place.

The scripts therefore do the following:

Script 1: Retrieve Data From SCCM - Export lists of maintenance windows and collection member from the SCCM database, and store them as CSV files.
Script 2: Populate Groups - Import collection members from a CSV, then create or update matching Groups in SCOM.
Script 3: Configure Maintenance Schedules - Import maintenance windows from a CSV, then create SCOM Maintenance Schedules for any windows in the future,

The scripts we’re using might not be a good fit if you’re working in a different way in SCCM. But, perhaps components of them could be used as the basis for something which would work for you.

You can schedule Maintenance mode in SCOM 2016+ - anyone made good use of this with patching windows?

I have created SQL Query based groups using Kevin Holman fragments https://gallery.technet.microsoft.com/SCOM-Management-Pack-VSAE-2c506737 and these populate from the SCCM Database. Our patching collection names never change, only the members so I’ve been able to stay with this one set of groups.

We use MMWeb Scheduler as still running SCOM 2012 R2 https://www.scom2k7.com/new-version-scom-2016-maintenance-mode-scheduler/ and can place the entire group in MM, in the background this tool is simply creating scheduled tasks with powershell so you could do it the manual way there too :slight_smile:

some work in process (not yet quite modular…) but perhaps some inspiration: seems to work pretty well for us the last monthly maintenances (environment (450 win agent, 30 linux, 290 home made distributed applications).

(Please note that I’m not a real coder)

# 0.1 List all objects already in maintenance before Monthly Maintenance Window(preOIM)
# 1. List all User Created Distributed Applications Components (UCDACom) already in MaintenanceMode (IM) and Save Maintenance settings
# 2. List all User Created Distributed Applications Components Upper Objects (UCDAUO) not in maintenance
# 3. Put all UCDAUO in MaintenanceMode (SCOM CLI start-SCOMMaintenanceMode ALWAYS does this recursive)
# 4. Get list of planned maintenance servers and put them in maintenance 
# 5. Reapply UCDACom Maintenance Settings for each object which was already in maintenance
# 6. Check if preOIM are put back 


$StartDate = (Get-Date)
$StartUniversal = $StartDate.ToUniversalTime()
$EndDate = (Get-Date).AddMinutes(100)
$EndUniversal = $EndDate.ToUniversalTime()
$Reason = "PlannedOther"
$Comment = "MonthlyMaintenance" 
#$TraversalDepth = "OneLevel"
#$TraversalDepth = "Recursive"

# 0.1 List all objects already in maintenance before Monthly Maintenance Window(preOIM)
$PreOIMFull = @()  
$preOIM = Get-SCOMMaintenanceMode
$preOIM | Export-Clixml c:\windows\temp\PreOIM.xml


# 1. List all User Created Distributed Applications Components (UCDACom) already in MaintenanceMode (IM) and Save Maintenance settings
 # $UCDAComClassName: "User Created Distributed Application Components"
	$UCDACompClassName = "Microsoft.SystemCenter.ServiceDesigner.ServiceComponentGroup"
	$UCDAComClass = get-scomclass |where {$_.Name -eq $UCDACompClassName}  
	$UCDAComClassInst = Get-SCOMClassInstance -Class $UCDAComClass
	$UCDAComClassInstNIM = $UCDAComClassInst|where {$_.InMaintenanceMode -like "False"}
	$UCDAComClassInstIM = $UCDAComClassInst|where {$_.InMaintenanceMode -like "True"}
 # Save PreMaintenance Ojects In Maintenance Settings	
	$UCDAObjPreMaintenance = $UCDAComClassInstIM | Get-SCOMMaintenanceMode
	$UCDAObjPreMaintenance |select * |ogv
	$UCDAObjPreMaintenance | Export-Clixml c:\windows\temp\UCDAObjPreMaintenance.xml



	
# 2. List all User Created Distributed Applications Components Upper Objects (UCDAUO) not in maintenance
 # $UCDAUOClassName: "User Created Distributed Applications"
	$UCDAUOClassName = "Microsoft.SystemCenter.ServiceDesigner.Service"
	$UCDAUOClass = get-scomclass |where {$_.Name -eq $UCDAUOClassName}  
	$UCDAUOClassInst = Get-SCOMClassInstance -Class $UCDAUOClass
	$UCDAUOClassInstNIM = $UCDAUOClassInst|where {$_.InMaintenanceMode -like "False"}
	$UCDAUOClassInstIM = $UCDAUOClassInst|where {$_.InMaintenanceMode -like "True"}	

# 3. Put all UCDAUO in MaintenanceMode (SCOM CLI start-SCOMMaintenanceMode ALWAYS does this recursive)
	$TraversalDepth = "OneLevel"
	$UCDAUOClassInstNIM.ScheduleMaintenanceMode($StartUniversal,$EndUniversal,$Reason,$Comment,$TraversalDepth)

# 4. Get list of planned maintenance servers and put them in maintenance, except scom servers and scom db clusternodes/instances
	$Classname = 'Microsoft.Windows.Computer'
	$Class = get-scomclass -name $Classname
	$allwin = get-scomclassinstance -class $Class |where {$_.DisplayName -NotMatch "vscomm" -AND $_.DisplayName -NotMatch "vcompanycl05p"}
	$TraversalDepth = "Recursive"
	$allwin.ScheduleMaintenanceMode($StartUniversal,$EndUniversal,$Reason,$Comment,$TraversalDepth)
	
	$Classname = 'Microsoft.Windows.Cluster.VirtualServer'
	$Class = get-scomclass -name $Classname
	$allvs= get-scomclassinstance -class $Class |where {$_.DisplayName -NotMatch "companysql7" -AND $_.DisplayName -NotMatch "companysql8"}
	$TraversalDepth = "Recursive"
	$allvs.ScheduleMaintenanceMode($StartUniversal,$EndUniversal,$Reason,$Comment,$TraversalDepth)	
	
	$Groupname = 'companyGENT.prd.SqlServers'
	$Group = Get-SCOMGroup $Groupname
	$allSqlsv= $Group |get-scomclassinstance |where {$_.DisplayName -NotMatch "companysql7" -AND $_.DisplayName -NotMatch "companysql8"}
	$TraversalDepth = "Recursive"
	$allSqlsv.ScheduleMaintenanceMode($StartUniversal,$EndUniversal,$Reason,$Comment,$TraversalDepth)	
		
	$Classname = 'Microsoft.Windows.Cluster'
	$Class = get-scomclass -name $Classname
	$allcl = get-scomclassinstance -class $Class |where {$_.DisplayName -NotMatch "companysql7" -AND $_.DisplayName -NotMatch "companysql8"  -AND $_.DisplayName -NotMatch "companycl05p"}
	$TraversalDepth = "Recursive"
	$allcl.ScheduleMaintenanceMode($StartUniversal,$EndUniversal,$Reason,$Comment,$TraversalDepth)	
	

	$Classname = 'Microsoft.Linux.Computer'
	$Class = get-scomclass -name $Classname
	$alllnx = get-scomclassinstance -class $Class |where {$_.DisplayName -NotMatch "vscomm" -AND $_.DisplayName -NotMatch "vcompanycl05p"}
	$TraversalDepth = "OneLevel"
	$alllnx.ScheduleMaintenanceMode($StartUniversal,$EndUniversal,$Reason,$Comment,$TraversalDepth)	

	$Classname = 'System.NetworkDevice'
	$Class = get-scomclass -name $Classname
	$allnwd = get-scomclassinstance -class $Class |where {$_.DisplayName -NotMatch "vscomm" -AND $_.DisplayName -NotMatch "vcompanycl05p"}
	$TraversalDepth = "OneLevel"
	$allnwd.ScheduleMaintenanceMode($StartUniversal,$EndUniversal,$Reason,$Comment,$TraversalDepth)	

	
	# 5. Reapply all Maintenance setting on objects which were already in maintenance
	$Reason = 'PlannedOther'
	$preOIM = import-Clixml c:\windows\temp\PreOIM.xml
	$TraversalDepth = "OneLevel"
	$OUT = @()
	Foreach ($monobj in $preOIM){
	$obj = Get-SCOMClassInstance -ID $monobj.MonitoringObjectId
	$Comment = $monobj.Comments
	$EndUniversal1 = $monobj.ScheduledEndTime.ToUniversalTime()
	$EndUniversal2 = $EndUniversal1.AddMinutes(60)
	$obj.UpdateMaintenanceMode($EndUniversal2,$Reason,$Comment)
	}

	# 6. Check if preOIM are put back 
	$postOIM = @()
	Foreach ($monobj in $preOIM){
	$obj = Get-SCOMClassInstance -ID $monobj.MonitoringObjectId
	$ttt = $obj | Get-SCOMMaintenanceMode
	$postOIM += $ttt
	}
	$postOIM | Export-Clixml c:\windows\temp\postOIM.xml

$preOIM = import-Clixml c:\windows\temp\PreOIM.xml
foreach ($i in $preOIM) {
$monobj = Get-SCOMMonitoringObject -id $i.MonitoringObjectId
$p1 = $monobj.DisplayName
$p2 = $monobj.FullName
Add-Member -InputObject $i -NotePropertyName "DisplayName" -NotePropertyValue $p1
Add-Member -InputObject $i -NotePropertyName "FullNameName" -NotePropertyValue $p2
}
$preOIM |Export-Clixml c:\windows\temp\PreOIM_ext.xml

$preOIM |select * |ogv
$postOIM |select * |ogv

Have you tried creating a script within one of your SCOM management packs to querry the SCCM datatabase, identifing servers that are due to enter a patching window and then trigger SCOM maintenance mode itself for that period of time (we add 5 mins either side) to ensure the SCOM places the Server into true maintenance Mode rather than SCCM simply pausing the agent.

Kevin Holman post this

 

https://kevinholman.com/2019/07/28/scom-agent-initiated-maintenance-mode-with-sccm-maintenance-windows/

 

I had just deployed this MP out. It worked great in my test environment but I am concerned with how well it will scale. I should find out shortly

Checking this out to see what it would take to implement in our environment.

Sorry, the maintenance mode GUI is a stand alone one I created. It doesn’t tie into the SCCM right click tools.

But you are welcome to adapt it to your environment. As always, it comes with no support, or warranty and used at your own risk.

I did fool with these at my last job and they worked but their were some down sides. There is no link between SCCM controlling the patching and SCOM doing he monitoring. So if a server got added, removed, or moved from a patch window I had to go make he same change in SCOM. With these products being in a suite I would love to see them talk to each other in a better way so that when I have a maintenance window open in SCCM it puts a server in maintenance mode in SCOM even more ideal would be if it was a check box option to do that in SCCM. I know there are check boxes there currently but they do not perform as they should. I did see several people commenting on it in user voice.

My last job I had a similar set up were collection names never changed just the members. I don’t currently have that luxury here but we are working towards it. That said maybe I can set up a collection in SCCM that we could use to put machines in maintenance mode and just do an include of the patching collection while patching is going on.

This looks interesting if I could wrap a UI around it similar to the right click tools where I could paste a list of machines in set a number of hours to keep the list in maintenance mode and then just hit a run button I think that would be a great tool.

This sounds great! Is it available to download somewhere?

I have test Kevin’s version of this in my test environment and it works great so far. Getting ready to move it to prod. One thing to watch for is unlike the original the rules are disabled by default. It does how ever give you the ability to say if maintenance window is over so many minutes to ignore it. Which works great if you have any long windows that exist but you don’t want scom to react to machines being in them.