I was working on a custom monitor at one of my customers using a powershell script. It involved creating a dataset to store some variables coming from a SQL database.
When trying to save the monitor I got an error stating: Incorrect expression specified: $Data = New-Object System.Data.dataset
A quick search pointed at that it was regarding to the variable $Data, apparently that is a reserved “name”. Getting rid of that solved the problem and it works like a charm.
Sometimes it happens that a server stops reporting performance data to SCOM. But we still get alerted when a monitor gets critical.
When do you notice this problem? Right, when you want to dive into performance data to troubleshoot something. Then you see that the server hasn’t sent performance data for the last couple of hours, days, weeks,…
To solve (or rather workaround) this issue I’ve managed to create a SQL query to point me toward servers that haven’t sent performance data in the last 2 hours (excluding servers that are in maintenance mode):
with _CTE
AS
(select bme.[Path],
pcv.ObjectName,
pcv.CounterName,
pcv.InstanceName,
pdv.SampleValue,
pdv.timesampled,
convert(varchar(max),pdv.TimeSampled,103)+ ' ' + convert(varchar(max),pdv.TimeSampled,108) as [Last Sample],
ROW_NUMBER() OVER (PARTITION BY bme.Path ORDER BY pdv.TimeSampled desc) AS RowNumber
from PerformanceDataAllView pdv with (NOLOCK)
inner join PerformanceCounterView pcv with (NOLOCK) on pdv.performancesourceinternalid = pcv.performancesourceinternalid
inner join BaseManagedEntity bme with (NOLOCK) on pcv.ManagedEntityId = bme.BaseManagedEntityId
where objectname = 'Memory' AND
countername = 'Available MBytes'
AND bme.BaseManagedEntityId NOT IN (SELECT M.BaseManagedEntityId FROM MaintenanceMode AS M WITH (NOLOCK) WHERE IsInMaintenanceMode = 1)
)
select * from _CTE WHERE RowNumber = 1
and timesampled < DATEADD(HOUR, -2, GETUTCDATE())
order by timesampled DESC
I’ve added this query the SquaredUp dashboard I check every morning.
But I also want to get alerted, so I created a powershell monitor to include this SQL query. As soon as a server stops sending performance data I get notified and I can (manually) solve it:
Today I want to talk about sending Alerts from SCOM to Microsoft Teams. Due to the current situation a lot of companies (that didn’t use Microsoft Teams previously) start using Microsoft Teams and want to have as much data as possible there. Thus it might be interesting to have the (critical) open alerts there as well.
I wanted to try this out myself as one of my customers also wanted to see how far we can go with this. Let’s go!
First things first, setup the necessary components in Microsoft Teams:
– Setup a dedicated Team in Microsoft Teams and create a channel. I called the team “Monitoring” and the channel “SCOM Alerts. I will give all relevant users necessary rights on this team
– Right click on the “SCOM Alerts” channel and select “Connectors”
– Search for “incoming webhook” and click Configure
– Give the webhook a name, I choose “SCOM”
– Click Create (make sure to copy the URL, we will need this later on in the powershell script needed for the subscription)
As Microsoft Teams is now configured, we can dive into powershell.
To post a message to Teams we will use the “Invoke-Webrequest” cmdlet from powershell. The message itself is in JSON format. In the message I would like to include: alertname, source, description, alert time, resolutionstate,…
As something extra I also added a “SquaredUp” button that enables you to go directly to the alert in SquaredUp.
I did struggle a bit to include the alert description in the JSON file, as for some alerts including the description resulted in a “Bad request”. This had to do with a “\” included in the description, which is a reserved character. adding a string replace “\” –> “\\” solved the problem. Thanks Cole for pointing that out!
The core of the script is:
# Fill in the Webhook URL here
$SCOM_Hook = ‘[URL goes here]’
# Build the JSON object for the POST – using a “stringwich” ( @”…”@ )to keep it readable.
$content_JSON = @”
{
“title”: “Critical SCOM Alert”,
“text”: “Source: $alertsource”,
“themeColor”: “FF0000”,
“sections”: [{
“title”: “Details”,
“facts”: [{
“name”:”Time Raised”,
“value”:”$alerttime”
Invoke-WebRequest -Uri $SCOM_HOOK -Method POST -Body $content_JSON -ContentType application/json
When your script is finished we can create the necessary components in SCOM (make sure your SCOM servers have internet access to https://outlook.office.com)
Create a channel:
Full Path of the command file: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
Command line parameters: \\myserver\myfolder\sendtoteams.ps1 ‘$Data/Context/DataItem/AlertId$’
You can include any criteria like you would on any other subscription
When you’re finished, just wait for an alert to fire the subscription
An alert message looks like this in Teams:
Looks even cooler in dark theme:
Extra tip: When using the Teams app on Android or iOS setup Notifications for the SCOM Alerts channel and you will get a push notification when an alert is posted.
If you have any questions, just drop me a line and I’ll be happy to help!
As I posted earlier SquaredUp can be integrated with all kinds of APIs, however if you know about management pack authoring an API can also be used as datasource in a management pack. In this post I’ll show some information about an Uptime Robot Management pack I created to create classes and instances of those classes in SCOM by using the API of Uptime Robot. To accomplish this I worked together with fellow SCOM consultant Jasper Vandamme
I created the management pack using Visual Studio Authoring Extensions.
Classes (with properties):
Uptime Robot WatcherNode
URL
APIKey
Uptime Robot Website
FriendlyName
ID
URL
Type
Interval
Uptime Robot WatcherNode Discovery:
All webrequests to the API are targeted on the watchernode. For this Discovery a registry discovery is used. It looks for a key HKLM\SOFTWARE\SCOM\Uptimerobot with values APIKey and URL. Both parameters are used afterwards in the scripts.
Uptime Robot Website Discovery:
After the WatcherNode is discovered the Website discovery kicks off, the core of the script is these lines of code:
$body = “api_key=$ApiKey&format=json”
Invoke-webrequest -method post -uri “$scripturl/v2/getMonitors” -body $body -UseBasicParsing
Monitors:
Website availability ( up or down)
API availability (up or down)
Rules:
API responsetime (ms)
Website responsetime (ms)
Website uptime last 7 days (%)
Website uptime last 30 days (%)
Website uptime last 90 days (%)
A dashboard can be created in SquaredUp to show the performance data
Drop me a comment below if you have any suggestions, feedback,..
Today
I want to talk about the integration of SquaredUp with external APIs. As an
example I want to integrate date coming from Uptimerobot (https://uptimerobot.com/) in SquaredUp. Uptimerobot
enables you to aadd 50 websites to check for free.
First of
all create an account for Uptimerobot
To be
able to access data we need to create an API key, therefore go to “My
Settings”. Go all the way down until you see “Read-Only API
Key”. Click “Show/Hide it”. Create the API key (or copy the one
that is already listed)
Next step
is adding the API in SquaredUp. Open Squaredup > System > Web Api. Click
“Add New Provider”. Fill in Service name, base URL should be the
following: https://api.uptimerobot.com/v2
Click Add
provider, it should look like this:
Now it’s
time to create a dashboard. I want to show the status of all monitors and the
responsetimes as well, let’s go.
Create a
new empty dashboard select the “Web API” tile
Select
“Web API (grid)”
Under
Scope, click Next
Select “Uptime Robot” Provider
Click
Next
Select
“Post” and add the url /getmonitors
Provide
the necessary parameters
Under Key
path enter monitors
Add any
columns you want in the grid eg. Friendly Name
I wanted
to show the response times as well but in seconds. The value is gathered in
milliseconds so it should be divided by 1000.
The
output looks like this:
Hope this helps to set this up if you might need it!
Today I was at a customer who had a really specific question regarding monitoring of Windows Services with Operations Manager (SCOM).
We had already set up some basic recovery actions which restart the service automatically after it was stopped.
For some other services the customer wanted to add extra functionality: The recovery action should retry starting the service a maximum of 3 times, if the service wasn’t started after 3 tries the customer wanted to receive an email telling them the recovery action failed. Out-of-the-box SCOM is unable to do stuff like that, therefore I used Powershell to accomplish this.
Sidenote: To be able to use Powershell as a recovery action you can use the free management pack provided by the community & SquaredUp, it can be downloaded from this website: https://squaredup.com/free-powershell-management-pack/. This management pack adds Powershell everywhere it is missing in Operations Manager, this is one of the default management packs I always install at customers.
To be fully functional different components are needed:
A monitor that checks the status of the service
This monitor can be created from the Authoring pane of the SCOM console using the Windows Service template
A recovery action for the monitor created previously
The recovery action can be created from health explorer
A rule that picks up the event created by the recovery action Powershell script
This is an Alert Generating Rule (NT Event Log), the configuration is linked to the type and location of the event logged during the script
A subscription on the rule to send the email.
The powershell script:
# Fill in the service name here
$ServiceName = “LPD Service”
$ServiceStarted = $False
$i =0;
#Create Eventlog source, erroraction Ignore is neededbecause once the source is created an error is thrown because the source already exists
I come at a lot of customers to implement or support SCOM. Sometimes the same questions or troubles come up.
One of that questions is: “Is it possible to monitor the count of files (with a specific extension) in a share?”
The answer to this question is yes and no. There is a possibility to count files on Windows Servers that have an agent installed using this management pack: http://www.systemcentercentral.com/pack-catalog/file-system-management-pack-2/ but for shares located on non-Windows Servers, let’s say on a SAN for example I haven’t found a solution available.
Therefore I created my own management pack to monitor the file count, independent of the location of the file share (Windows Server or not).
In this post I describe how the management pack works. With the management pack you can count files with a specific extension (or no extension if everything should be counted) in a share (optionally also subfolders included).
There is also the ability to add a specific age zo the given scenario is possible: Count if there are more then 20 files in a share (subfolders included) that are older then 10 minutes.
First of all we need a seed discovery which is targeted to a registry key located on a SCOM agent monitored Windows Server.
The value in the registry is located under SOFTWARE\Filecount. The value is “CSV” and it should contain the path to a CSV file. The server will be discovered as a “File Count Watcher Node”
Next stop is the csv file itself, for every share to be monitored it should contain a line with a specific syntax shown in the screenshot below
Different parameters are added:
ID
Must be unique per share
Share
UNC path of the share
Extension
The extension of the files that needs to be counted, leave empty to count all files in the share
Count
How many files must be present for a critical state
Time
This is the time in minutes of the maximum file age of file count
Recurse
0 = No need to count files in subfolders
1 = Count also files in subfolders
When the info is filled in, SCOM will discover every line as a “File Count Share”. The properties are used to configure the monitoring.
A monitor is also defined based on the properties filled in the csv file, but it’s basically a powershell script with necessary parameters.
The file count is also gathered as a performance counter so it can be included in reporting or in a Squared Up dashboard for example.
The management pack is also configured to use a specific Run As account. This account needs rights on the shares: at least Read-only Share rights and Read-Only NTFS rights.
I’ve been able to help some customers already by using this management pack.
The first customer where I set this up is a big hospital in Belgium where they use this management pack to monitor shares which are used to store (and process) images and movies made during surgery.
The content should be processed from the network share and transferred somewhere else but sometimes the processing hangs and the share is getting full without anyone knowing. Since they have the management pack in place this hasn’t happened anymore.
Today I want to show you guys a nice example of how we can use custom scripts to visualise data coming from SCCM in Squared Up dashboards on top of a SCOM platform. And provide a strong integration between SCCM, SCOM & Squared Up.
I’m currently in the process of setting up a brand new SCOM 1801 environment at a customer and an SCCM CB environment at the same customer with the focus on servers and server patching. The customer wanted also an easy way to check the following parameters:
How many updates are pending on each server?
Is there a pending reboot due to the installation of Windows Updates?
When was the server last updated? And what Windows Updates have been applied then?
Are there any maintenance windows applied to a server?
Lots of questions to be answered! All this information can be found somewhere hidden in the SCCM reporting or we could create some custom reports containing that info. However, due to the fact I’m not a SQL reporting guru I wanted to see if we could show all this information in an easy way using Squared Up. As all the servers also have a SCOM agent installed we could get our information directly from there. And I have to say, it was really fun and easy to do!
I’ve used some custom powershell scripts which I added as SCOM tasks, these tasks are then added to a “Windows Updates” perspective I created for each Windows Computer object and this is the result:
All the components you see in the above perspective are based on powershell scripts put in SCOM tasks.
The “Pending Updates” counter is a custom performance counter reading information from WMI, this is the script:
# Any Arguments specified will be sent to the script as a single string. # If you need to send multiple values, delimit them with a space, semicolon or other separator and then use split. param([string]$Arguments)
$ScomAPI = New-Object -comObject “MOM.ScriptAPI”
# Collect your performance counter here. Note that in order to support the DW this script MUST only return a single # object/counter and those must be static values. You can return multiple instances just fine though.
On top of the perspective I’ve also added a button to install all pending updates directly from the dashboard without having to logon to the server and start the installation from Software Center. When clicking the button, Software Center kicks of the installation within minutes.
There is also an additional dashboard where they can review the servers with the most updates pending (top 20) per domain.
To finish things of I added some of the above information to the “Windows Server” perspective as well. In this way the customer can immediately see additional info (highlighted in red in the below screenshot):
How many pending updates?
When was the server last rebooted?
Is there a pending reboot due to Windows Updates?
If the server is a VM, what is the Hyper-V host the VM is running on?
Now they are using Squared Up as the go to tool for reviewing the health of the environment and for following-up on patching!
If you would like to accomplish the same or have any questions, please feel free to drop a comment below and I’ll be happy to share my scripts.
I’m currently busy at several customers setting up SCOM infrastructures. At one of those customers there were complaints by users that email messages were queued. When looking at the queues on the exchange servers they were actually HUGE.
Due to the fact Exchange was already monitored in SCOM I thought we would have seen this but that was not the case. In the Exchange Management pack there is NO monitor to check the actual queue length although there are rules which collect the queue lengths. I think this is not OK as it is really important to be notified when messages get queued in Exchange.
Therefore I created a custom Powershell monitor to check this. I created the monitor using the SquaredUp Powershell management pack, this management pack adds Powershell support everywhere throughout SCOM (where you would expect that by default :)) The management pack can be downloaded here: https://squaredup.com/content/management-packs/free-powershell-management-pack/
This is the powershell script itself:
# Any Arguments specified will be sent to the script as a single string. # If you need to send multiple values, delimit them with a space, semicolon or other separator and then use split. param([string]$Arguments)
# Example of use below, in this case return the length of the string passed in and we’ll set health state based on that. # Since the health state comparison is string based in this template we’ll need to create a state value and return it. # Ensure you return a unique value per health state (e.g. a service status), or a unique combination of values.
The monitor will become critical when the messagecount exceeds 500 messages, this can be increased or decreased if needed by changing “$queue -gt 500” accordingly.
This monitor has helped my customer to prevent this issue from happening again.
