Suppress alert storms in SCOM 2012R2 for network outage

Dashboard Server
,
1

Is there a way to suppress unneeded notifications or alert storms generated by SCOM when a Switch or Router goes down?

If one of the racktop switches goes down, we’ll get 100’s of notifications emailed to the Service Desk for servers not reachable, which we don’t want.

We’d like notifications emailed that the switch is down only

we don’t want a server down notification emailed for each of the servers attached to the switch. And following the same logic, we don’t want notifications emailed that multiple racktop switches & servers are down if the core switch has a faulty card or goes down itself.

Any suggestions or guidance on fixing this in SCOM will be greatly appreciated.

 

Can anyone point me to the right resource or help me out with the steps?

Regards
SR

2

This is an old SCOM issue. Don’t think there is a solution for this natively in the product.

You can vote for the issue on the SCOM User Voice forum

https://systemcenterom.uservoice.com/forums/293064-general-operations-manager-feedback/suggestions/9358647-logical-chain-of-components-alert-storm-preventio

1 Like
3

This PowerShell script reads in a file called ScheduledGroups.txt and writes it out.

Add a scheduled task to run every 5 minutes that calls this PowerShell Script.

Have it run with an account that can affect SCOM.

Action is to run C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe

and the argument would be the path to the script.

Options are

Frequency - Once, Daily, weekly

Enabled - true, false or test

Make sure the folder and file are writable

First line of script should be:

“Enabled”,“Group”,“Frequency”,“Length”,“LastExecution”,“NextExecution”,“Comment”

Next line should contain the values you want like this:

“false”,“SquaredUp - Windows Computers”,“Once”,“315”,“12/19/2017 11:58 PM”,“12/20/2017 12:01 AM”,“HD Case 1131295”

change the location for the text file (2 places)

change the FQDN of the SCOM management server you want to connect to as well.

import-module OperationsManager
New-SCOMManagementGroupConnection -ComputerName “Server.domain.com”
$Settings = Import-Csv <location of script>\ScheduledGroups.txt

ForEach($Setting in $Settings)
{
#Check the next execution time… allow for 2.5 minutes since the task cannot run more often than every 5 minutes.
If((Get-Date).AddMinutes(10) -gt (Get-Date($Setting.NextExecution)) -and (Get-Date).AddMinutes(-10) -lt (Get-Date($Setting.NextExecution)) -and ($Setting.Enabled -eq “true” -or $Setting.Enabled -eq “test”))
{

#Start Maintenance Mode
$Instances = Get-SCOMGroup -DisplayName $Setting.Group

foreach($monitorObject in $Instances)
{
If($Setting.Enabled -eq “test”)
{ Write-Host $monitorObject }
Else
{ Start-SCOMMaintenanceMode -EndTime:(Get-Date).AddMinutes($Setting.Length) -Instance:$monitorObject -Comment:$Setting.Comment }
}

#Set Last Execution
$Setting.LastExecution = Get-Date -format g

#Set Next Exeuction
$LastEx = Get-Date
If($Setting.Frequency.StartsWith(“Once”))
{
$Setting.Enabled = “false”
}
If($Setting.Frequency.StartsWith(“Daily”))
{
$Frequency = $Setting.Frequency.Split("|") #Expecting 2: Daily | <time>
$Setting.NextExecution = Get-Date((Get-Date($LastEx.AddDays(1)) -format d).ToString() + " " + ($Frequency[1])) -format g
}
If($Setting.Frequency.StartsWith(“Weekly”))
{
$Frequency = $Setting.Frequency.Split("|") #Expecting 3: Weekly | <NumDayOfWeek Sun=0, Mon=1, etc> | <time>
$Setting.NextExecution = Get-Date((Get-Date($LastEx.AddDays(7 + $Frequency[1] - [int]$LastEx.DayOfWeek)) -format d).ToString() + " " + ($Frequency[2])) -format g
}
}
}

$Settings | Export-Csv <location of script>ScheduledGroups.txt -notype

4

We have made our own solution to cope with this issue. We do not use scoms builtin notification channels.

We built a sma-job that fetches open alerts from scom and then we have created filters that tags the alerts and sends alerts if criterias are met and does not send alerts if anotger criteria is met.

Another solution we be to have a seperate notificitaion channel just for the switch alert and one for the others. And then have ps-script disable the notification channel temporarilly while the alert is open.

1 Like
5

By Class and object name using wildcards

ie.

“Enabled”,“Class”,“Names”,“Frequency”,“Length”,“LastExecution”,“NextExecution”,“Comment”

“True”,“Microsoft.Windows.Computer”,“BMOJKW*”,“Weekly|0|6:55”,“60”,“10/18/2015 6:50 AM”,“10/25/2015 6:55 AM”,“HD Case 987874”

import-module OperationsManager
New-SCOMManagementGroupConnection -ComputerName “server.domain.com”
$Settings = Import-Csv C:\Scheduled.txt

ForEach($Setting in $Settings)
{
#Check the next execution time… allow for 2.5 minutes since the task cannot run more often than every 5 minutes.
If((Get-Date).AddMinutes(5) -gt (Get-Date($Setting.NextExecution)) -and (Get-Date).AddMinutes(-10) -lt (Get-Date($Setting.NextExecution)) -and ($Setting.Enabled -eq “true” -or $Setting.Enabled -eq “test”))
{

    #Start Maintenance Mode
    $MonitorClass = Get-SCOMClass -Name:$Setting.Class;
    If($Setting.Names -eq "All")
        { $Instances = Get-SCOMClassInstance -Class:$MonitorClass }
    Else
        { $Instances = Get-SCOMClassInstance -Class:$MonitorClass | Where-Object {$_.DisplayName -match $Setting.Names} }
        
    foreach($monitorObject in $Instances)
    {
        If($Setting.Enabled -eq "test")
            { Write-Host $monitorObject }
        Else
            { Start-SCOMMaintenanceMode -EndTime:(Get-Date).AddMinutes($Setting.Length) -Instance:$monitorObject -Comment:$Setting.Comment }
    }
    
    #Set Last Execution
    $Setting.LastExecution = Get-Date -format g
    
    #Set Next Exeuction
    $LastEx = Get-Date
    If($Setting.Frequency.StartsWith("Once"))
    {
        $Setting.Enabled = "False"
    }
    If($Setting.Frequency.StartsWith("Daily"))
    {
        $Frequency = $Setting.Frequency.Split("|")  #Expecting 2: Daily | 
        $Setting.NextExecution = Get-Date((Get-Date($LastEx.AddDays(1)) -format d).ToString() + " " + ($Frequency[1])) -format g
    }
    If($Setting.Frequency.StartsWith("Weekly"))
    {
        $Frequency = $Setting.Frequency.Split("|")  #Expecting 3: Weekly |  | 
        $Setting.NextExecution = Get-Date((Get-Date($LastEx.AddDays(7 + $Frequency[1] - [int]$LastEx.DayOfWeek)) -format d).ToString() + " " + ($Frequency[2])) -format g
    }
}

}

$Settings | Export-Csv C:\Scheduled.txt -notype