Running SMlets in Orchestrator as a user with privileges in SCSM

As described in SMlets on server without the SCSM console you can use the SMlets from codeplex on a server without the SCSM console. Particullary useful on an Orchestrator server used to automate different parts of Service Manager.

You can use your PowerShell scripts in a '.Net Activity'.

The question is if the Orchestrator service account has the permission to do anything in ServiceManager as the script default will run with this user account.

To come around this issue simply add Orchestrator variables containing the username and password of a service account with suitable privileges in Service Manager. In your .Net Activity you add the following lines of code to make all the cmdlets in the SMlets run under this account:

$RemoteUsername = '{RunAsUser_SM}'
$RemotePassword = {ConvertTo-SecureString 'RunAsPassword_SM}' -AsPlainText -Force
$SMCreds = New-Object System.Management.Automation.PSCredential($RemoteUsername, $RemotePassword) -ErrorAction:Stop
$PSdefaultParameterValues.Add("*SCSM*:Credential", $SMCreds)

The next obstacle might be your PowerShell ExecutionPolicy which will stop your script at importing the SMlets module. The ExecutionPolicy can be changed on the server by running PowerShell (x86) as an administrator and run the command

Set-ExecutionPolicy Unrestricted

This will be a global change on the server and might be against your companys policy. If you do not want to do this, you can add two more lines in your .net activity to bypass this

$var = Powershell -ExecutionPolicy Bypass {
  ....
  <your script here>
  ...
}

Your script will work, but you will loose the feedback from the script unless you pipe it back to $var

Cireson Self Service portal not working after hotfix 5.0.5

After upgrading a Cireson Self Service portal to hotfix 5.0.5 the site did not start as expected. So I logged in on the server and opened the the site to review the error there.

It said

Could not load file or assembly 'LicenseManagement.Client, Version=1.0.5882.21425, Culture=neutral, PublicKeyToken=98ba2176e2a9efbc' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

Turns out that the changes to the Cireson licensing has made some changes that requires a newer version of the LicenseManagement.Client.dll different from the version I had (1.0.0.8). This file is located under C:\inetpub\CiresonPortal\bin\LicenseManagement.Client.dll

I contacted Cireson support and they came with the solution quickly enough.

Download the updated installer from https://support.cireson.com/software/#/ reinstall and apply the hotfix again.

Seems there still is an issue with some NavigationNodes disappearing when you do a new install. So be sure to insert those again after your have re-installed to 5.0.1. and applying hotfix 5.0.5

My upgrade path before this hotfix was 5.0.0 > 5.0.2  > 5.0.5.
I think the dll was updated in 5.0.1, since that was the version it reported after a reinstall.

SMlets on server without the SCSM console

I use the SMlets from https://smlets.codeplex.com/ in my everyday work with ServiceManager. A very helpful set of cmdlets that originally requires to be installed on a system with the Service Manager Console installed. This is not something you always want to do. So here is the workaroud to avoid that.

Download and install SMlets from codeplex.

Copy the files Microsoft.EnterpriseManagement.Core.dll and Microsoft.EnterpriseManagement.ServiceManager.dll from c:\Program Files\Microsoft System Center\Service Manager 2010\SDK Binaries\ to your server.

Run Powershell as admin and run the following code to register the dlls

[System.Reflection.Assembly]::Load("System.EnterpriseServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a")       
$publish = New-Object System.EnterpriseServices.Internal.Publish
$publish.GacInstall("C:\sources\Microsoft.EnterpriseManagement.Core.dll")
$publish.GacInstall("C:\Sources\Microsoft.EnterpriseManagement.ServiceManager.dll")

Configure the module to run on a server without the console installed. Open C:\Program Files\Common Files\SMlets\Smlets.psml as administrator.

Comment out line 4, 5, 6 and 46
Add the following on line 47 to point to your SDK server:

$GLOBAL:smdefaultcomputer="scsmservername"

Note:
This will break your scripts that use the -ComputerName parameter in the SMlets. If you want to continue using this parameter if you manage several ServiceManager instances from one server, you can skip the last line addition.

Note2:
If you skip the last line, you can avoid using the -ComputerName parameter on all SMlets cmdlets by adding a default parameter at the beginning of your script like this:

$PSDefaultParameterValues.Add("*SCSM*:ComputerName","your.scsm.server")

SCSM Self Service Portal MaximumInstanceCount

Challenge:

You create a request offering in Service Manager which contains a Query Result prompt that returns a large result set. The Self Service Portal that ships with Service Manager (SMPortal) will only return 2000 values. If you are lucky you will find your value there, but otherwise you are out of luck.

Solution:

Extend the value that limits the result set

1. Open IIS Manager on your web server that contains SMPortal

2. Extend Sites > SCSMWebContentServer > ServiceHost

3. Double click on Application Settings

3. Create a new application setting by pressing Add in the task pane

4. Enter MaximumInstanceCount with your desired value

Please note that this will slow down/halt your silverlight application in the web browser until it is finished loading the result from the query. Consider changing your query or splitting the request offering into multiple offerings.

Orchestrator runbooks not syncing to SCSM

Today I copied one Orchestrator runbook from my test environment over to the production environment. Then I went to the Service Manager console and requested the Orchestrator connector to Synchronize, which it successfully did.

But I could not see the runbook showing up in the SCSM console. Neither could I find it on the Orchestrator web console.

Resolution:
1. Open SQL Management studio and connect to the SQL Server and instance hosting the Orchestrator Database

2.Type in the following SQL Query:

TRUNCATE TABLE
[Microsoft.SystemCenter.Orchestrator.Internal].AuthorizationCache

Refresh the Orchestrator web console, and the runbooks are there.

Resyncing the Orchestrator connector finally put the runbook in SCSM and I could proceed creating my runbook automation template

MonitoringHost.exe crashing after updating to SCSM 2012 SP1 UR2

Lately I have been troubleshooting a problem that occurred after installing Update Rollup 2 in a development environment.

My first indication of a problem was when I asked a colleague to take a look at a new request offering I had published. When he tried to log on to the portal he got access denied, something I guessed had to do him not being in the cmdb. I search for him under Configuration Items > Users, but he was not there. So I went ahead and checked where he is placed in Active Directory and found that he should be found by the AD connector. That was odd. But the connector had not run for a week, so I thought he might have been moved there recently and triggered a new synchronization. After a while it looked like it got stuck at 17%.
So I restarted the SCSM services, but that did not help. It did not help to restart the server either.

Time to troubleshoot

Windows Event Log
Service Manager logs a fair amount of information to the Windows Event Log. You will find the events under Application and Services Logs > Operations Manager

In my case I saw a pattern of events.

Source: HealthService (the System Center Management service)
Event id: 1103
Level: Warning
Summary: 155 rule(s)/monitor(s) failed and got unloaded, 155 of them reached the failure limit that prevents automatic reload. Management group "DEVELOPMENT". This is summary only event, please see other events with descriptions of unloaded rule(s)/monitor(s).

Source: HealthService
Event id: 4000
Level: Error
A monitoring host is unresponsive or has crashed.  The status code for the host failure was 2164195371.

And a lot of this. I never counted, but I guess it was 155.

Source: HealthService
Event id: 1206
Level: Information
Rule/Monitor "Microsoft.SystemCenter.CollectDiscoveryData", running for instance "Microsoft.ServiceManager.InternalDiscoveryCollectorTarget" with id:"{DB18B9A2-0117-0D4F-B484-C3060D1C31F0}" failed, got unloaded and reached the failure limit that prevents automatic reload. Management group "DEVELOPMENT".

This pattern reappeared each time i restarted the services/server.

Disabling the connector
Suspecting it might be the ad connector workflow timing out and causing trouble for the other workflows I went ahead and disabled the AD Connector. It still showed up as Running at 17%, even after restarting the console, the service manager services and even the server. The pattern in the event log was still there.

Troubleshooting the database / moving the watermark
I searched for event id 4000 and found references to MonitoringHost.exe. In my case it was not running for more than a couple of minutes before disappearing. This is an essential process. It is needed to run workflows and you will see one or more instances running on the Service Manager server under different workflow accounts. I did find a related blog post by Nathan Lasnoski SCSM A monitoring host is unresponsive or has crashed Error 4000 which pointed me to the database for troubleshooting. When running the sql query I found several workflows which was more than 4500 minutes behind. Many more than I would bother to disable in a test/dev environment. So I tried a quick fix from the System Center blog and move the watermark. See the blogpost Troubleshooting Workflow Performance and Delays

If the workflows are hopelessly behind and you want them to catch up immediately, you can update the State column on the CmdbInstanceSubscriptionState table by running this query:

DECLARE @MaxTransactionId Int, @RuleId uniqueidentifier
SET @RuleId = '6AA6B941-375B-E3AD-7FB647FC7B3E' --<-- set this to your rule id!!
SET @MaxTransactionID = (SELECT(MAX(EntityTransactionLogID) FROM EntityTransactionLog)

UPDATE CmdbInstanceSubscriptionState
SET State = @MaxTransactionId
WHERE RuleId = @RuleID

(Except I did not filter by RuleID. I did it for every rule)

The blog also states something to keep in mind:
Keep in mind that running this query will move the watermark forward.  Any transactions that would have triggered the subscription criteria and any resulting actions (notifications, workflows, etc) for those transactions that are being skipped over WILL NOT HAPPEN.  Be very careful using this update query.  Also, keep in mind that the EntityChangeLog table has a grooming routine that grooms rows from that table which are no longer needed.  One of the criteria for determining whether or not a row is needed is the position of the watermarks for the workflows.  We recommend slowly moving up the watermark so as not to trigger a massive grooming job that would slow down the overall system.  We also recommend moving the watermark during non-peak hours in order to minimize the impact of grooming on people using the system.

In my case this did not solve the problem. Workflows still fell "Minutes behind" quickly. I thought it might help since I also had imported quite a few management packs from the production environment, and that this might had been the issue. The issue with MonitoringHost.exe crashing was still there.

Tracing & Log files
Trying to dig deeper I went ahead and started doing tracing. StartSMTracing.cmd is a tool found in the Service Manager\Tools folder that lets you trace what is happening within Service Manager. It takes a couple of parameters that let you select trace level and trace area.  After recreating the issue you stop the trace with StopSMTrace.cmd. Now you will have an .etl file you can use the FormatTracing.cmd to convert it to a readable file.

This tracing gave me a lot of information. But I was not able to find the reason causing MonitoringHost.exe to crash.

The fix for me
Part 1:
Further searches and articles and led me to a hotfix for .net 3.5 "A .NET Framework 3.0-based WCF service may crash with a System.ServiceModel.CommunicationException exception if the service uses the netTcpBinding binding"

Thinking I might be a timeout issue, I hoped this would help me, so I installed this on my server and rebooted. But it still crashed after a little while as before.

Part 2:
Looking in the Service Manager directory i saw that there was no config file for MontioringHost.exe, it was renamed to .bak during the upgrade to UR2. I had previously tried to rename it back to .config, but it did not help at that point. But I thought I would give it a shoot and renamed it back once more.

Restarting the System Center Management service now proved successful. It started the MonitoringHost process(es) and they kept on running longer than they did before. I have also re-enabled the AD Connector and it has now completed successully.

It was a long way, and it took a bit of time, but I finally was able to solve the problem.

I should of course have updated the servers to R2, but we have not been able to do that yet. Hopefully this might help someone else struggling with similar problems.

/Michael

Troubleshooting SCSM

System Center Service Manager can be a bit cumbersome to troubleshoot if you don't know where to look. In this blog I have collected a few useful guides which may help you on your way.

Places to look

Windows Event Log:
Events can be found under the Operations Manager log.

Tracing:
SMTrace can be useful. See http://www.concurrency.com/blog/enable-tracing-in-service-manager/ for examples.

Workflows:
Take a look in the console under Administration > Workflows > Status