SCOM – Servers without Perf Data

By | April 22, 2020
0 Comment

Hi,

Sometimes it happens that a server stops reporting performance data to SCOM. But we still get alerted when a monitor gets critical.

When do you notice this problem? Right, when you want to dive into performance data to troubleshoot something. Then you see that the server hasn’t sent performance data for the last couple of hours, days, weeks,…

To solve (or rather workaround) this issue I’ve managed to create a SQL query to point me toward servers that haven’t sent performance data in the last 2 hours (excluding servers that are in maintenance mode): 

with _CTE
AS
(select bme.[Path],
pcv.ObjectName,
pcv.CounterName,
pcv.InstanceName,
pdv.SampleValue,
pdv.timesampled,
convert(varchar(max),pdv.TimeSampled,103)+ ' ' + convert(varchar(max),pdv.TimeSampled,108) as [Last Sample],
ROW_NUMBER() OVER (PARTITION BY bme.Path ORDER BY pdv.TimeSampled desc) AS RowNumber
from PerformanceDataAllView pdv with (NOLOCK)
inner join PerformanceCounterView pcv with (NOLOCK) on pdv.performancesourceinternalid = pcv.performancesourceinternalid
inner join BaseManagedEntity bme with (NOLOCK) on pcv.ManagedEntityId = bme.BaseManagedEntityId
where objectname = 'Memory' AND
countername = 'Available MBytes'
AND bme.BaseManagedEntityId NOT IN (SELECT M.BaseManagedEntityId FROM MaintenanceMode AS M WITH (NOLOCK) WHERE IsInMaintenanceMode = 1)
)
select * from _CTE WHERE RowNumber = 1
and timesampled < DATEADD(HOUR, -2, GETUTCDATE())
order by timesampled DESC

I’ve added this query the SquaredUp dashboard I check every morning.

But I also want to get alerted, so I created a powershell monitor to include this SQL query. As soon as a server stops sending performance data I get notified and I can (manually) solve it:

  • Stop Microsoft Monitoring Agent
  • Cleanup Health Service State folder
  • Start Microsoft Monitoring Agent

Hope this helps,

Bert

SCOM – Forward Alerts to Microsoft Teams

By | April 21, 2020
0 Comment

Today I want to talk about sending Alerts from SCOM to Microsoft Teams. Due to the current situation a lot of companies (that didn’t use Microsoft Teams previously) start using Microsoft Teams and want to have as much data as possible there. Thus it might be interesting to have the (critical) open alerts there as well.

I found some interesting posts on the internet:

I wanted to try this out myself as one of my customers also wanted to see how far we can go with this. Let’s go!

First things first, setup the necessary components in Microsoft Teams:
– Setup a dedicated Team in Microsoft Teams and create a channel. I called the team “Monitoring” and the channel “SCOM Alerts. I will give all relevant users necessary rights on this team

– Right click on the “SCOM Alerts” channel and select “Connectors”

– Search for “incoming webhook” and click Configure

– Give the webhook a name, I choose “SCOM”

– Click Create (make sure to copy the URL, we will need this later on in the powershell script needed for the subscription)

As Microsoft Teams is now configured, we can dive into powershell.

To post a message to Teams we will use the “Invoke-Webrequest” cmdlet from powershell. The message itself is in JSON format. In the message I would like to include: alertname, source, description, alert time, resolutionstate,…
As something extra I also added a “SquaredUp” button that enables you to go directly to the alert in SquaredUp.

I did struggle a bit to include the alert description in the JSON file, as for some alerts including the description resulted in a “Bad request”. This had to do with a “\” included in the description, which is a reserved character. adding a string replace “\” –> “\\” solved the problem. Thanks Cole for pointing that out!

The core of the script is:
# Fill in the Webhook URL here
$SCOM_Hook = ‘[URL goes here]’

# Build the JSON object for the POST – using a “stringwich” ( @”…”@ )to keep it readable.
$content_JSON = @”
{
“title”: “Critical SCOM Alert”,
“text”: “Source: $alertsource”,
“themeColor”: “FF0000”,
“sections”: [{
“title”: “Details”,
“facts”: [{
“name”:”Time Raised”,
“value”:”$alerttime”

},
{
“name”:”ResolutionState”,
“value”:”$alertresolutionstate”
},
{
“name”:”Monitor/Rule”,
“value”:”$alert”
},
{
“name”:”Description”,
“value”:”$alertdescription”
}
]
},
{
“potentialAction”: [
{
“@context”: “http://schema.org”,
“@type”: “OpenUri”,
“name”: “SquaredUp”,
“targets”: [{
“os”: “default”,
“uri”: “$alerturl”
}
]
}
]
}
]
}
“@

Invoke-WebRequest -Uri $SCOM_HOOK -Method POST -Body $content_JSON -ContentType application/json

When your script is finished we can create the necessary components in SCOM (make sure your SCOM servers have internet access to https://outlook.office.com)

  • Create a channel:
    • Full Path of the command file: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
    • Command line parameters: \\myserver\myfolder\sendtoteams.ps1 ‘$Data/Context/DataItem/AlertId$’
    • Startup folder: C:\Windows\System32\WindowsPowerShell\v1.0\
  • Create a subscriber for that channel
  • Create a subscription for the subscriber
    • You can include any criteria like you would on any other subscription

When you’re finished, just wait for an alert to fire the subscription
An alert message looks like this in Teams:

Looks even cooler in dark theme:

Extra tip: When using the Teams app on Android or iOS setup Notifications for the SCOM Alerts channel and you will get a push notification when an alert is posted.

If you have any questions, just drop me a line and I’ll be happy to help!

Best regards,
Bert

SCOM – Uptime Robot Management Pack

By | April 3, 2020
2 Comments

Hi,

As I posted earlier SquaredUp can be integrated with all kinds of APIs, however if you know about management pack authoring an API can also be used as datasource in a management pack. In this post I’ll show some information about an Uptime Robot Management pack I created to create classes and instances of those classes in SCOM by using the API of Uptime Robot. To accomplish this I worked together with fellow SCOM consultant Jasper Vandamme

First things first: create the API using the Uptime Robot website. (detailed steps: https://www.bpitconsulting.be/2019/11/12/squared-up-uptimerobot-api/ )

I created the management pack using Visual Studio Authoring Extensions.

Classes (with properties):

  • Uptime Robot WatcherNode
    • URL
    • APIKey
  • Uptime Robot Website
    • FriendlyName
    • ID
    • URL
    • Type
    • Interval

Uptime Robot WatcherNode Discovery:

All webrequests to the API are targeted on the watchernode. For this Discovery a registry discovery is used. It looks for a key HKLM\SOFTWARE\SCOM\Uptimerobot with values APIKey and URL. Both parameters are used afterwards in the scripts.

Uptime Robot Website Discovery:

After the WatcherNode is discovered the Website discovery kicks off, the core of the script is these lines of code:

$body = “api_key=$ApiKey&format=json”

Invoke-webrequest -method post -uri “$scripturl/v2/getMonitors” -body $body -UseBasicParsing

Monitors:

  • Website availability ( up or down)
  • API availability (up or down)

Rules:

  • API responsetime (ms)
  • Website responsetime (ms)
  • Website uptime last 7 days (%)
  • Website uptime last 30 days (%)
  • Website uptime last 90 days (%)

A dashboard can be created in SquaredUp to show the performance data

Drop me a comment below if you have any suggestions, feedback,..

Best regards,

Bert

Squared Up – Uptimerobot (API)

By | November 12, 2019
1 Comment

Hi,

Today I want to talk about the integration of SquaredUp with external APIs. As an example I want to integrate date coming from Uptimerobot (https://uptimerobot.com/)  in SquaredUp. Uptimerobot enables you to aadd 50 websites to check for free.

First of all create an account for Uptimerobot

To be able to access data we need to create an API key, therefore go to “My Settings”. Go all the way down until you see “Read-Only API Key”. Click “Show/Hide it”. Create the API key (or copy the one that is already listed)

The Uptimerobot API is documented here: https://uptimerobot.com/api

Next step is adding the API in SquaredUp. Open Squaredup > System > Web Api. Click “Add New Provider”. Fill in Service name, base URL should be the following: https://api.uptimerobot.com/v2

Click Add provider, it should look like this:

Now it’s time to create a dashboard. I want to show the status of all monitors and the responsetimes as well, let’s go.

Create a new empty dashboard select the “Web API” tile

Select “Web API (grid)”

Under Scope, click Next

 Select “Uptime Robot” Provider

Click Next

Select “Post” and add the url /getmonitors

Provide the necessary parameters

Under Key path enter monitors

Add any columns you want in the grid eg. Friendly Name

I wanted to show the response times as well but in seconds. The value is gathered in milliseconds so it should be divided by 1000.

The output looks like this:

Hope this helps to set this up if you might need it!

Best regards,

Bert

SCOM – Powershell Recovery Action – Stopped Windows Service

By | November 12, 2019
0 Comment

Hi,

Today I was at a customer who had a really specific question regarding monitoring of Windows Services with Operations Manager (SCOM).

We had already set up some basic recovery actions which restart the service automatically after it was stopped.

For some other services the customer wanted to add extra functionality: The recovery action should retry starting the service a maximum of 3 times, if the service wasn’t started after 3 tries the customer wanted to receive an email telling them the recovery action failed. Out-of-the-box SCOM is unable to do stuff like that, therefore I used Powershell to accomplish this.

Sidenote: To be able to use Powershell as a recovery action you can use the free management pack provided by the community & SquaredUp, it can be downloaded from this website: https://squaredup.com/free-powershell-management-pack/. This management pack adds Powershell everywhere it is missing in Operations Manager, this is one of the default management packs I always install at customers.

To be fully functional different components are needed:

  • A monitor that checks the status of the service
    • This monitor can be created from the Authoring pane of the SCOM console using the Windows Service template
3
  • A recovery action for the monitor created previously
    • The recovery action can be created from health explorer
  • A rule that picks up the event created by the recovery action Powershell script
    • This is an Alert Generating Rule (NT Event Log), the configuration is linked to the type and location of the event logged during the script
  • A subscription on the rule to send the email.

The powershell script:

# Fill in the service name here

$ServiceName = “LPD Service”

$ServiceStarted = $False

$i =0;

#Create Eventlog source, erroraction Ignore is neededbecause once the source is created an error is thrown because the source already exists

New-Eventlog -LogName Application -Source “Powershell – Restart Service” -ErrorAction Ignore

Do{

# In second or third run, wait a minute before trying
to start the service

if ($i -gt 0){Start-Sleep -s 60}

#Try to start the service

Start-Service $ServiceName

$Service = Get-Service -Name $ServiceName

     if($Service.Status -eq “Running”)

    {

    $ServiceStarted = $true

     }

    $i++

    if (($i -eq 3) -and ($ServiceStarted = $false))

    {

    $eventmessage = “$Servicename failed to restart after $i attempts, exiting script”

    #Log error event in eventviewer

    Write-Eventlog -LogName Application -Source “Powershell – Restart Service” -EntryType Error -Eventid 101 -Message $eventmessage

    exit

    }

 }

Until ($ServiceStarted = $true)

 $eventmessage = “$ServiceName restarted after $i attempt(s)”

Write-Eventlog -LogName Application -Source “Powershell – Restart Service” -EntryType Information -Eventid100 -Message $eventmessage

 If you have any difficulties doing this, don’t hesitate to drop a comment below.

If you find this post useful, please consider buying me a virtual beer with a bitcoin donation: 3QhpQ5z5hbPXXRS8x6R5RagWVrRQ5mDEZ1

Best regards,

Bert

SCOM – File Count Management Pack

By | November 12, 2019
0 Comment

Hi,

I come at a lot of customers to implement or support SCOM. Sometimes the same questions or troubles come up.

One of that questions is: “Is it possible to monitor the count of files (with a specific extension) in a share?”

The answer to this question is yes and no. There is a possibility to count files on Windows Servers that have an agent installed using this management pack: http://www.systemcentercentral.com/pack-catalog/file-system-management-pack-2/ but for shares located on non-Windows Servers, let’s say on a SAN for example I haven’t found a solution available.

Therefore I created my own management pack to monitor the file count, independent of the location of the file share (Windows Server or not).

In this post I describe how the management pack works. With the management pack you can count files with a specific extension (or no extension if everything should be counted) in a share (optionally also subfolders included).

There is also the ability to add a specific age zo the given scenario is possible: Count if there are more then 20 files in a share (subfolders included) that are older then 10 minutes.

First of all we need a seed discovery which is targeted to a registry key located on a SCOM agent monitored Windows Server.

The value in the registry is located under SOFTWARE\Filecount. The value is “CSV” and it should contain the path to a CSV file. The server will be discovered as a “File Count Watcher Node”

Next stop is the csv file itself, for every share to be monitored it should contain a line with a specific syntax shown in the screenshot below

Different parameters are added:

  • ID
    • Must be unique per share
  • Share
    • UNC path of the share
  • Extension
    • The extension of the files that needs to be counted, leave empty to count all files in the share
  • Count
    • How many files must be present for a critical state
  • Time
    • This is the time in minutes of the maximum file age of file count
  • Recurse
    • 0 = No need to count files in subfolders
    • 1 = Count also files in subfolders

When the info is filled in, SCOM will discover every line as a “File Count Share”. The properties are used to configure the monitoring.

A monitor is also defined based on the properties filled in the csv file, but it’s basically a powershell script with necessary parameters.

The core of the script is this command:

$count  = Get-ChildItem -Recurse $strShare\$strExtension | where{$_.LastWriteTime -le (Get-Date).AddMinutes($strAge)}|Measure-Object |%{$_.Count}

The file count is also gathered as a performance counter so it can be included in reporting or in a Squared Up dashboard for example.

The management pack is also configured to use a specific Run As account. This account needs rights on the shares: at least Read-only Share rights and Read-Only NTFS rights.

I’ve been able to help some customers already by using this management pack.

The first customer where I set this up is a big hospital in Belgium where they use this management pack to monitor shares which are used to store (and process) images and movies made during surgery.

The content should be processed from the network share and transferred somewhere else but sometimes the processing hangs and the share is getting full without anyone knowing. Since they have the management pack in place this hasn’t happened anymore.

If you have interest in the management pack, I’ve made it available via GitHub: https://github.com/bpinoy/ManagementPacks/tree/master/File%20Count%20MP

Best regards,

Bert

Visualize Windows Update info (from SCCM) in Squared Up

By | November 12, 2019
2 Comments

Hi,

Today I want to show you guys a nice example of how we can use custom scripts to visualise data coming from SCCM in Squared Up dashboards on top of a SCOM platform. And provide a strong integration between SCCM, SCOM & Squared Up.

Squared Up is my favorite tool to create dashboards, check it out here: https://demo.squaredup.com 

I’m currently in the process of setting up a brand new SCOM 1801 environment at a customer and an SCCM CB environment at the same customer with the focus on servers and server patching. The customer wanted also an easy way to check the following parameters:

  • How many updates are pending on each server?
  • Is there a pending reboot due to the installation of Windows Updates?
  • When was the server last updated? And what Windows Updates have been applied then?
  • Are there any maintenance windows applied to a server?

Lots of questions to be answered! All this information can be found somewhere hidden in the SCCM reporting or we could create some custom reports containing that info. However, due to the fact I’m not a SQL reporting guru I wanted to see if we could show all this information in an easy way using Squared Up. As all the servers also have a SCOM agent installed we could get our information directly from there. And I have to say, it was really fun and easy to do!

I’ve used some custom powershell scripts which I added as SCOM tasks, these tasks are then added to a “Windows Updates” perspective I created for each Windows Computer object and this is the result:

All the components you see in the above perspective are based on powershell scripts put in SCOM tasks.

The “Pending Updates” counter is a custom performance counter reading information from WMI, this is the script:

# Any Arguments specified will be sent to the script as a single string.
# If you need to send multiple values, delimit them with a space, semicolon or other separator and then use split.
param([string]$Arguments)

$ScomAPI = New-Object -comObject “MOM.ScriptAPI”

# Collect your performance counter here. Note that in order to support the DW this script MUST only return a single
# object/counter and those must be static values. You can return multiple instances just fine though.

$Instances = @(“Total”)
$Metric = @(Get-WmiObject -Class CCM_SoftwareUpdate -Filter ComplianceState=0 -Namespace root\CCM\ClientSDK).Count

Foreach ($Instance in $Instances)
{
$PropertyBag = $ScomAPI.CreatePropertyBag()
$PropertyBag.AddValue(“Metric”, $Metric)
$PropertyBag.AddValue(“Total”,$Instance)

# Send output to SCOM
$PropertyBag
}

On top of the perspective I’ve also added a button to install all pending updates directly from the dashboard without having to logon to the server and start the installation from Software Center. When clicking the button, Software Center kicks of the installation within minutes.

There is also an additional dashboard where they can review the servers with the most updates pending (top 20) per domain.

To finish things of I added some of the above information to the “Windows Server” perspective as well. In this way the customer can immediately see additional info (highlighted in red in the below screenshot):

  • How many pending updates?
  • When was the server last rebooted?
  • Is there a pending reboot due to Windows Updates?
  • If the server is a VM, what is the Hyper-V host the VM is running on?

Now they are using Squared Up as the go to tool for reviewing the health of the environment and for following-up on patching!

If you would like to accomplish the same or have any questions, please feel free to drop a comment below and I’ll be happy to share my scripts.

Hope this helps!

Best regards,

Bert

SCOM – Quick Tip: Exchange Queue Length Monitor

By | November 12, 2019
0 Comment

Hi,

I’m currently busy at several customers setting up SCOM infrastructures. At one of those customers there were complaints by users that email messages were queued. When looking at the queues on the exchange servers they were actually HUGE.

Due to the fact Exchange was already monitored in SCOM I thought we would have seen this but that was not the case. In the Exchange Management pack there is NO monitor to check the actual queue length although there are rules which collect the queue lengths. I think this is not OK as it is really important to be notified when messages get queued in Exchange.

Therefore I created a custom Powershell monitor to check this. I created the monitor using the SquaredUp Powershell management pack, this management pack adds Powershell support everywhere throughout SCOM (where you would expect that by default :)) The management pack can be downloaded here: https://squaredup.com/content/management-packs/free-powershell-management-pack/

This is the powershell script itself:

# Any Arguments specified will be sent to the script as a single string.
# If you need to send multiple values, delimit them with a space, semicolon or other separator and then use split.
param([string]$Arguments)

$ScomAPI = New-Object -comObject “MOM.ScriptAPI”
$PropertyBag = $ScomAPI.CreatePropertyBag()

# Example of use below, in this case return the length of the string passed in and we’ll set health state based on that.
# Since the health state comparison is string based in this template we’ll need to create a state value and return it.
# Ensure you return a unique value per health state (e.g. a service status), or a unique combination of values.

Add-PSSnapin Microsoft.Exchange.Management.PowerShell.SnapIn;

$queue = ((Get-Queue | Select-Object @{ n = “MessageCount”; e = { [int]($_.MessageCount) } }).MessageCount | measure-object -sum).sum

$PropertyBag.AddValue(“Length”,$queue)

if($queue -gt 500) {
$PropertyBag.AddValue(“State”,”OverThreshold”)
}
else
{
$PropertyBag.AddValue(“State”,”UnderThreshold”)
}

# Send output to SCOM
$PropertyBag

The monitor will become critical when the messagecount exceeds 500 messages, this can be increased or decreased if needed by changing “$queue -gt 500” accordingly.

This monitor has helped my customer to prevent this issue from happening again.

Hope this helps!

Best regards,

Bert

SCOM – HTML Report: Certificate Expiration

By | November 12, 2019
0 Comment

Hi,

Today I want to give an answer to the question I often get at customers: “We want an overview of all certificates in our environment (on Windows Servers) that are expiring within X amount of days”.

This approach is based on this management pack: http://www.systemcentercentral.com/pack-catalog/pki-certificate-verification-mp/

The management pack discovers certificates on all servers (in specific stores if you will) and adds monitors to detect expiration, validity, etc. It also adds some views in the Monitoring pane of the SCOM console.

I wanted to provide a (daily) report to the customer to show them the list of certificates expiring within X amount of days. Therefore I created a custom group in SCOM where I put in all “critical” certificates with for example a specific CA. Afterwards I created a powershell script which uses the members of the group to create a report and mail it to the different stakeholders.

The report looks like this:

As you can see the report shows in a glance when certificates will expire and on what server they are located. As this is fully custom all layout, colours etc. can be set at will.

Mostly I choose to use red, orange and yellow to highlight the urgency in which the certificates need to be replaced.

Also keep in mind that after replacing the certificates in eg. IIS, they need to be deleted from the server(s) as well because they will keep on being discovered by SCOM if they stay on the server(s).

If you are interested in the script, please drop a comment below and I’ll be happy to assist.

Hope this helps!

Best regards,

Bert