Monitoring ESXi Syslogs with OpsMgr 2007 R2 (Part 1)

NOTE: Part 2 – Basic Reports was just published.

NOTE: This is a BETA POST! This post will be updated throughout the next few days to correct spelling mistakes and provide a lot more information about what’s going on in a learn-by-example sort of way. However, I can’t seem to resist the urge of publishing what I already have. Please let me know if you have any problems, and I’ll address them right away! This stuff is _hard_ at first!

There’s a few posts around the net on Syslog monitoring in Operations Manager 2007 ( SCOM ). These posts helped me understand the process (and OpsMgr) a lot better, but didn’t really go as far as I’d like. I needed to capture, process, and export syslog data from my ESXi hosts. Turns out this is extremely difficult to poke around and figure out, but easy to set up if you have a guide.

Some people are okay with the built-in syslog event collectors in OpsMgr, but they give you output like this. Notice how the OpsMgr’s event properties don’t really give you anything useful (Level: Success, Name: Win-Syslog, etc.)

I wanted more, and figured out how to make my events look like this:


Short-cut!

Getting this to work is a long process. If you want results without the learning curve, follow these steps. Otherwise, enter at your own risk!

  1. Complete steps 1-4 under “Preparation” below.
  2. Download and import my custom management pack (right-click, save-as): JPPacks.MonitoringSyslogs 1.0.0.9.
  3. Run these two PowerShell commands on the new VM that will be collecting syslogs:
    New-Item -Path HKLM:\SOFTWARE\JPPacks
    New-ItemProperty -Path HKLM:\SOFTWARE\JPPacks -Name SyslogEnabled -PropertyType DWord -Value 1
  4. “Configure ESXi” according to the screenshots below.
  5. Create an override for the new discovery “JPPacks – Discover Syslog Receivers” changing ‘frequency’ to “20” (seconds).
  6. You should see your VM show up under “Syslog Receiver States” within a minute or so.
  7. Delete that frequency override you made in step 5, or you’ll load down your OpsMgr server.
  8. Send a few test events from Kiwi Syslog Generator. By default only events severity 4 and under (Warning, Error, and Emergency) are logged, this can be changed in the collection rule (see below).

The rest of this post outlines how to create the custom management pack which allows you to parse the raw syslog data into nice, informative events. This includes DNS lookups and mapping <Severity> to <EventLevel> for the pretty Success\Warning\Error icons. Once you have the raw event data in PowerShell and all the modules working together, you can really do anything you want–create new properties, process syslogs differently depending on the source host\IP, etc.

Table of Contents

Preparation

First, prepare your environment:

  1. Deploy a new VM called Win-Syslogs. This system will receive the ESXi syslogs to take the load off of your OpsMgr RMS server.
  2. Install the OpsMgr agent on Win-Syslogs.
  3. Open port 514 in the firewall on Win-Syslogs to your ESXi hosts and your local workstation (for testing).
  4. Download and install Kiwi Syslog Generator on your local workstation (for testing).
  5. Download and install System Center Operations Manager 2007 R2 Authoring Resource Kit on your local workstation.
  6. Download and install XML Notepad 2007 on your local workstation.
  7. Download my Powershell event parsing script from my Chemistry website, and save it as Process-Syslogs-Rev1.ps1.

Configure ESXi

The next step is enabling ESXi Syslogging.

  1. Open your Hosts and Clusters view in vCenter (if you have it).
  2. Select the first host you want to collect from, then click the ‘Configuration’ tab, then ‘Advanced Settings’.
  3. Click ‘Syslog’ on the left pane, then enter the dns name or IP address of Win-Syslogs in the top box. Make sure you’re using Port 514, because OpsMgr will only listen on this port, then click OK.

Creating the Custom Management Pack

Getting Started

  1. First, open System Center Authoring Console and click the ‘New Management Pack’ button.
  2. Then, give your management pack a name.
  3. Next, enter a display name and description, then click Create. The display name will show when looking at the list of management packs in the OpsMgr Console’s Administration view.. The description text is as follows: This management pack contains a custom datasource which processes Syslog events with a PowerShell script, and maps parsed properties to generic OpsMgr events. It also contains a custom rule to collect and publish these events, and a view to look at them.

Creating the Data Source

  1. Next, click the “Type Library” view, then select ‘Module Types’ -> ‘Data Sources’. Right-click and choose “New” -> “Composite Data Source”.
  2. Name your new Data Source, then click “OK”.
  3. Enter a name for your composite data source, then a description. When finished, click the “Member Modules” tab. The name will show up when creating a custom view. The description text here is as follows: This data source recieves a DataItem from the built-in Syslog data source module, processes it with a powershell script using a probe action module, then maps custom parameters from a PowerShell Property Bag to a new Event’s EventData using the Generic Event Mapper condition detection module.
  4. Click “Add.”
  5. Un-Check “Condition Detection” and “Probe Action”, then enter “Syslog” in the “Look for:” text box. Select “System.ApplicationLog.Syslog.FilteredEventProvider”, then enter “SyslogDS” as the Module ID, then click OK.
  6. Click “Configure”.
  7. For Parameter name enter “Severity”, for Operator choose  “Less than or Equal To”, for Value enter “$Config/MinSeverity$”.
  8. The configuration should now say “Port 514”, and have the information filled in that you just entered. If so, click “OK”.
  9. Click ‘Add’ again.
  10. Un-Check “Condition Detection” and “Data Source”, then enter “Powershell” in the “Look for:” text box. Select “Microsoft.Windows.PowerShellPropertyBagProbe”, then enter “ParseScript” as the Module ID, then click OK.
  11. On the ParseScript configuration window, enter “Parse-SyslogEvents.ps1” as the script name, and “30” as the TimeoutSeconds value. Then, click “Edit”.
  12. Familiarize yourself with the default XML structure. You will be adding the script text between <ScriptBody> and </ScriptBody>. You will also be inserting <parameters> tags after </ScriptBody>.
  13. First, insert the Parameters tags as follows:
    <Parameters>
    <Parameter>
    <Name>xmlText</Name>
    <Value>$Data$</Value>
    </Parameter>
    </Parameters>
  14. Next, insert the PowerShell parsing script between the ScriptBody tags as follows:
  15. Save and close the XML file.
  16. Verify that the settings look correct, then click OK.
  17. Click, ‘Add’ again.
  18. Un-Check “Data Source” and “Probe Action”, then enter “Event” in the “Look for:” text box. Select “System.Event.GenericDataMapper”, then enter “Mapper” as the Module ID, then click OK.
  19. On the “Mapper” configuration page, click “Edit”.
  20. Familiarize yourself with the XML structure of the Mapper.
  21. Replace <EventOriginId> through <Params> with the following code:
    <EventOriginId>$Target/Id$</EventOriginId>
    <PublisherId>$MPElement$</PublisherId>
    <PublisherName>$MPElement$</PublisherName>
    <Channel>$Data/Property[@Name="opsmgr_Channel"]$</Channel>
    <LoggingComputer>$Data/Property[@Name="opsmgr_LoggingComputer"]$</LoggingComputer>
    <EventNumber>$Data/Property[@Name="opsmgr_EventNumber"]$</EventNumber>
    <EventCategory>0</EventCategory>
    <EventLevel>$Data/Property[@Name="opsmgr_EventLevel"]$</EventLevel>
    <UserName></UserName>
    <Description>$Data/Property[@Name="Message"]$</Description>
    <Params></Params>

  22. Close the XML file and click “Save”.
  23. Verify that the settings look correct, then click “OK”.
  24. Configure the “NextModule” fields so that SyslogDS leads to ParseScript leads to Mapper leads to Module Output.
  25. Click the “Configuration Schema” tab, then in the bottom frame click “Add”.
  26. Enter “MinSeverity” as the value, then click “OK”.
  27. Change “Type” to “Integer”, then click “Data Types”.
  28. Under the drop-down box, choose “System.Event.Data”, then click “OK”

Creating the Syslog Receiver Class

  1. Click “Service Model” to change views.
  2. Choose “Classes” from the left pane, then Right-Click the right pane and choose “New” -> “Custom Class”.
  3. Enter an ID for your custom class.
    ID: JPPacks.MonitorSyslogs.SyslogReceiver
  4. Enter a name and description for the custom class, then check the “Hosted” box, then Browse for a base class.
    Name: Syslog Receiver
    Description: Computer that receives syslog data.
  5. Click the “List” radio button, then choose “Microsoft.Windows.ComputerRole”, and click “OK”.
  6. Verify that your settings look correct, and click “OK”.

Creating a Discovery for your Class

  1. Click “Health Model” to change views.
  2. Click “Discoveries” in the left pane, then right-click the right pane and choose “New” -> “Registry (Filtered)”.
  3. Enter an ID, Name, and Description for your Discovery. Then, click “…” to browse for a target.
    ID: JPPacks.MonitorSyslogs.DiscoverSyslogReceivers
    Name: JPPacks – Discover Syslog Receivers
    Descriptoion: This discovery looks for the following registry DWORD value: “HKLM\SOFTWARE\JPPacks\SyslogEnabled”. If the value is present and equal to “1”, this discovery rule creates an instance of the ‘Syslog Receiver’ class referencing the discovered computer.
  4. Click the “list” radio button, then in the “Look for” text box enter “windows.o”. Click to choose “Microsoft.Windows.OperatingSystem”, and click “OK”.
  5. Verify that your settings look correct, and click “Next”.
  6. Change the schedule to run once per day. To get things started faster we can create an override inside OpsMgr console once the management pack is imported.
  7. On the “Computer” screen, keep the defaults and click “OK”.
  8. On the registry Probe Configuration page, click “Add”, then “Value”, then enter:
    Name: “SyslogEnabled”
    Path: SOFTWARE\JPPacks\SyslogEnabled
    Attribute Type: Int
  9. Confirm that the settings look correct and click “Next”.
  10. On the “Build Event Expression” page, click “Insert” then choose:
    Parameter Name: “Values/Syslog Enabled”
    Operator: “Equals”
    Value: 1
  11. On the “Discovery Mapper” page, under “Key Properties (Required)”, browse for a property then choose “(Host=Windows Computer)” -> “Principal Name”.
  12. Browse to the same property under “Non Key Properties (Optional)”, then click “Finish”.
  13. Verify that your settings look correct, than click “OK”.
  14. Right-Click your new discovery, and choose “Properties”.
  15. Click the “Configuration” tab, then scroll down to the bottom of the window. Make sure that the last 2 @Type names are set to value “Integer”, than click “OK”.

Reference the Data Warehouse Module

  1. Click “Health Model” to change views.
  2. Click “File” -> “Management Pack Properties”.
  3. Click the “References” tab, then click “Add Reference”.
  4. Browse to the Operations Manager 2007 server directory, and choose “Microsoft.SystemCenter.DataWarehouse.Library.mp”, then click “Open”.
  5. When asked to add the path to your reference paths, click “No”.
  6. Confirm that your settings look OK, then click “Close”.

Creating the Event Collection Rule

  1. First, select “Rules” on the left pane, then right-click the right pane and choose “New” -> “Custom Rule”.
  2. Give your rule an ID, then click OK. I chose “JPPacks.MonitorSyslogs.CollectSyslogs”.
  3. Enter a display name and description for your rule, then change the target to the custom class you created in previous steps. When finished click the “Modules” tab. I chose for following values:
    Name: “JPPacks – Collect Syslogs”
    Description: “This rule collects processed syslog events from the JPPacks custom datasource and publishes them.”
  4. On the “Modules” tab, under “Data Sources”, click “Create”.
  5. Choose the Custom Data Source created in previous steps, then name the Module ID “CustomSyslogDS”, then click “OK”. My Data Source has the ID “JPPacks.MonitorSyslogs.CustomSyslogDS”.
  6. Click “Edit” to configure the CystomSyslogDS module.
  7. Under “MinSeverity”, enter the minimum severity of the Syslog that you want to collect then click, “OK”. A value of “4” will collect all warnings and above (Warnings, Errors, Emergencies, etc.).
  8. On the “Modules” tab, under “Actions”, click “Create”.
  9. In the “Look for:” box, enter “event”, then choose “Microsoft.SystemCenter.CollectEvent”. Then, in the “Module ID” box enter “Collect”, then click “OK”.
  10. On the “Modules” tab, under “Actions”, click “Create” again.
  11. In the “Look for:” box, enter “event”, then choose “Microsoft.SystemCenter.DataWarehouse.PublishEventData”. Then, in the “Module ID” box enter “Publish”, then click “OK”.
  12. Confirm that your settings look correct, then click the “Options” tab.
  13. Uncheck “Confirm Delivery”, and change “Category” to “Event Collection”, then click “OK”.

Creating Views to See Events

  1. Click “Presentation” to switch views.
  2. Click “Views” on the left pane, then right-click the right pane and choose “New” -> “Folder”.
  3. Enter an ID for the view and click “Next”.
    ID: JPPacks.MonitorSyslogs.Views.JPPacksViews
  4. Right-click the right pane again, and choose “New” -> “State View”.
  5. Enter the following information:
    ID: JPPacks.MonitorSyslogs.Views.SyslogReceiversState
    Display Name: JPPacks – Syslog Receivers State
    Description: This view shows the names and states of all discovered Syslog Receivers.
    Target: choose ‘JPPacks.MonitorSyslogs.SyslogReceiver’
    Category: Choose ‘AvailabilityHealth’
  6. Right-click the right pane again, and choose “New” -> “Event View”.
  7. Enter the following information, then click “Finish”.
    ID: JPPacks.MonitorSyslogs.Views.SyslogEvents
    Display Name: JPPacks – Syslog Events
    Description:  This view shows all syslog events received by instances of the ‘Syslog Reveiver’ class.
    Target: choose ‘JPPacks.MonitorSyslogs.SyslogReceiver
  8. Right-click the Event View you created, then choose “Properties”.
  9. Click the “Configuration” tab, then click “Edit”.
  10. Familiarize yourself with the XML layout. We will be replacing the criteria tags.
  11. Replace the <Criteria> tags with the text provided by the file named “JPPacks.MonitorSyslogs.Views.SyslogEvents.txt”. If you followed the Preparation heading,  you already have this. Otherwise, download it here: JPPacks.MonitorSyslogs.Views.SyslogEvents.txt.
  12. Verify that your settings look correct, then click the “Folder” tab.
  13. Ensure that the only box checked is the folder you want this view to reside in. For me, that’s “JPPacks.MonitorySyslogs.Views.JPPacksViews”, then click “OK”.
  14. Right-click the State View created in step 4, and choose “Properties”.
  15. Click the “folder” tab, and ensure that the only box checked is the folder you want this view to reside in. For me, that’s “JPPacks.MonitorySyslogs.Views.JPPacksViews”, then click “OK”.
  16. Right-click the folder you created in step 2 and choose “Properties”.
  17. Enter the following information:
    Name: JPPacks Views
    Description: This folder contains all views related to the JPPacks Management Pack.

Creating a Health Monitor

  1. Click “Monitors” from the left pane, then expand “System.Health.EntityState”. Right-click “System.Health.AvailabilityState” and click “New” -> “Windows Services” -> “Basic Service Monitor”.
  2. Enter the following information then click “Next”.
    Element ID: JPPacks.MonitorSyslogs.Monitors.AvailabilityState.HealthService
    Display Name: JPPacks Monitors – Health Service
    Description: This monitor watches the health service on Syslog Receivers and reports its status.
    Target: choose JPPacks.MonitorSyslogs.SyslogReceiver
    Parent Monitor: choose System.HealthAvailabilityState
    Category: AvailabilityHealth
  3. For “Service Name” enter “HealthService”, then click “Finish”.
  4. Confirm that your settings look correct.

Congrats! Your management pack is complete! Now, follow the steps under “Short-Cut!” to get it working (except step 2 of course).

As always, check my Chemistry Wiki for semi-organized info on Operations Manager as I continue to learn it.

See Part 2 – Basic Reports to get this data exported.

References:

Advertisements

10 thoughts on “Monitoring ESXi Syslogs with OpsMgr 2007 R2 (Part 1)

  1. Pingback: OpsMgr 2007 Base Concepts and Helpful Links « windowsmasher

  2. Pingback: OpsMgr 2007 MP Authoring – Proxying Syslog Events « windowsmasher

  3. Pingback: Operations Manager 2007 vCenter Management Pack (pre-alpha) « windowsmasher

  4. Everything appears to be working, i see events in opsmgr but it doesnt seem to generate alerts even though I enabled alerts via override. any suggestion?

  5. In ESXi 5 the advanced options for syslog output have changed slightly. The way you enable the syslog via ESXi firewall has changed also. I am wondering if any of your powershell scripts ned to change b/c of this?

    Basically, I am in the middle of attempting to implement this in my SCOM env.

  6. Pingback: Monitoring ESXi Syslogs with OpsMgr 2007 R2 (Part 1) « Vinachip

  7. Pingback: OpsMgr 2007 Base Concepts and Helpful Links « Vinachip

  8. Great article! Congratulations!
    I tested this on a SCOM 2012 environment and managed to get it working when receiving syslog messages from a few apliances only (small wireless router, for example). When receiving syslog messages from ESXi and Slackware, I get the error below on the Win-Syslog server. The event is not processed and does not show up on SCOM. Did anyone try it successfully on a SCOM 2012? Thanks a lot!

    Log Name: Operations Manager
    Source: Health Service Modules
    Date: 7/16/2012 3:56:41 PM
    Event ID: 11061
    Task Category: None
    Level: Warning
    Keywords: Classic
    User: N/A
    Computer: WIN2K8SYSLOG.mydomain.local
    Description:
    Module was unable to perform parameter replacement on parameter ‘$Data/Property[@Name=”opsmgr_Channel”]$’

    Error: 0x80020005
    Details: Type mismatch.

  9. Pingback: Table of Contents | windowsmasher

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s