SCSM Self Service Portal MaximumInstanceCount

Challenge:

You create a request offering in Service Manager which contains a Query Result prompt that returns a large result set. The Self Service Portal that ships with Service Manager (SMPortal) will only return 2000 values. If you are lucky you will find your value there, but otherwise you are out of luck.

Solution:

Extend the value that limits the result set

1. Open IIS Manager on your web server that contains SMPortal

2. Extend Sites > SCSMWebContentServer > ServiceHost

3. Double click on Application Settings

3. Create a new application setting by pressing Add in the task pane

4. Enter MaximumInstanceCount with your desired value

Please note that this will slow down/halt your silverlight application in the web browser until it is finished loading the result from the query. Consider changing your query or splitting the request offering into multiple offerings.

Orchestrator runbooks not syncing to SCSM

Today I copied one Orchestrator runbook from my test environment over to the production environment. Then I went to the Service Manager console and requested the Orchestrator connector to Synchronize, which it successfully did.

But I could not see the runbook showing up in the SCSM console. Neither could I find it on the Orchestrator web console.

Resolution:
1. Open SQL Management studio and connect to the SQL Server and instance hosting the Orchestrator Database

2.Type in the following SQL Query:

TRUNCATE TABLE
[Microsoft.SystemCenter.Orchestrator.Internal].AuthorizationCache

Refresh the Orchestrator web console, and the runbooks are there.

Resyncing the Orchestrator connector finally put the runbook in SCSM and I could proceed creating my runbook automation template

MonitoringHost.exe crashing after updating to SCSM 2012 SP1 UR2

Lately I have been troubleshooting a problem that occurred after installing Update Rollup 2 in a development environment.

My first indication of a problem was when I asked a colleague to take a look at a new request offering I had published. When he tried to log on to the portal he got access denied, something I guessed had to do him not being in the cmdb. I search for him under Configuration Items > Users, but he was not there. So I went ahead and checked where he is placed in Active Directory and found that he should be found by the AD connector. That was odd. But the connector had not run for a week, so I thought he might have been moved there recently and triggered a new synchronization. After a while it looked like it got stuck at 17%.
So I restarted the SCSM services, but that did not help. It did not help to restart the server either.

Time to troubleshoot

Windows Event Log
Service Manager logs a fair amount of information to the Windows Event Log. You will find the events under Application and Services Logs > Operations Manager

In my case I saw a pattern of events.

Source: HealthService (the System Center Management service)
Event id: 1103
Level: Warning
Summary: 155 rule(s)/monitor(s) failed and got unloaded, 155 of them reached the failure limit that prevents automatic reload. Management group "DEVELOPMENT". This is summary only event, please see other events with descriptions of unloaded rule(s)/monitor(s).

Source: HealthService
Event id: 4000
Level: Error
A monitoring host is unresponsive or has crashed.  The status code for the host failure was 2164195371.

And a lot of this. I never counted, but I guess it was 155.

Source: HealthService
Event id: 1206
Level: Information
Rule/Monitor "Microsoft.SystemCenter.CollectDiscoveryData", running for instance "Microsoft.ServiceManager.InternalDiscoveryCollectorTarget" with id:"{DB18B9A2-0117-0D4F-B484-C3060D1C31F0}" failed, got unloaded and reached the failure limit that prevents automatic reload. Management group "DEVELOPMENT".

This pattern reappeared each time i restarted the services/server.

Disabling the connector
Suspecting it might be the ad connector workflow timing out and causing trouble for the other workflows I went ahead and disabled the AD Connector. It still showed up as Running at 17%, even after restarting the console, the service manager services and even the server. The pattern in the event log was still there.

Troubleshooting the database / moving the watermark
I searched for event id 4000 and found references to MonitoringHost.exe. In my case it was not running for more than a couple of minutes before disappearing. This is an essential process. It is needed to run workflows and you will see one or more instances running on the Service Manager server under different workflow accounts. I did find a related blog post by Nathan Lasnoski SCSM A monitoring host is unresponsive or has crashed Error 4000 which pointed me to the database for troubleshooting. When running the sql query I found several workflows which was more than 4500 minutes behind. Many more than I would bother to disable in a test/dev environment. So I tried a quick fix from the System Center blog and move the watermark. See the blogpost Troubleshooting Workflow Performance and Delays

If the workflows are hopelessly behind and you want them to catch up immediately, you can update the State column on the CmdbInstanceSubscriptionState table by running this query:

DECLARE @MaxTransactionId Int, @RuleId uniqueidentifier
SET @RuleId = '6AA6B941-375B-E3AD-7FB647FC7B3E' --<-- set this to your rule id!!
SET @MaxTransactionID = (SELECT(MAX(EntityTransactionLogID) FROM EntityTransactionLog)

UPDATE CmdbInstanceSubscriptionState
SET State = @MaxTransactionId
WHERE RuleId = @RuleID

(Except I did not filter by RuleID. I did it for every rule)

The blog also states something to keep in mind:
Keep in mind that running this query will move the watermark forward.  Any transactions that would have triggered the subscription criteria and any resulting actions (notifications, workflows, etc) for those transactions that are being skipped over WILL NOT HAPPEN.  Be very careful using this update query.  Also, keep in mind that the EntityChangeLog table has a grooming routine that grooms rows from that table which are no longer needed.  One of the criteria for determining whether or not a row is needed is the position of the watermarks for the workflows.  We recommend slowly moving up the watermark so as not to trigger a massive grooming job that would slow down the overall system.  We also recommend moving the watermark during non-peak hours in order to minimize the impact of grooming on people using the system.

In my case this did not solve the problem. Workflows still fell "Minutes behind" quickly. I thought it might help since I also had imported quite a few management packs from the production environment, and that this might had been the issue. The issue with MonitoringHost.exe crashing was still there.

Tracing & Log files
Trying to dig deeper I went ahead and started doing tracing. StartSMTracing.cmd is a tool found in the Service Manager\Tools folder that lets you trace what is happening within Service Manager. It takes a couple of parameters that let you select trace level and trace area.  After recreating the issue you stop the trace with StopSMTrace.cmd. Now you will have an .etl file you can use the FormatTracing.cmd to convert it to a readable file.

This tracing gave me a lot of information. But I was not able to find the reason causing MonitoringHost.exe to crash.

The fix for me
Part 1:
Further searches and articles and led me to a hotfix for .net 3.5 "A .NET Framework 3.0-based WCF service may crash with a System.ServiceModel.CommunicationException exception if the service uses the netTcpBinding binding"

Thinking I might be a timeout issue, I hoped this would help me, so I installed this on my server and rebooted. But it still crashed after a little while as before.

Part 2:
Looking in the Service Manager directory i saw that there was no config file for MontioringHost.exe, it was renamed to .bak during the upgrade to UR2. I had previously tried to rename it back to .config, but it did not help at that point. But I thought I would give it a shoot and renamed it back once more.

Restarting the System Center Management service now proved successful. It started the MonitoringHost process(es) and they kept on running longer than they did before. I have also re-enabled the AD Connector and it has now completed successully.

It was a long way, and it took a bit of time, but I finally was able to solve the problem.

I should of course have updated the servers to R2, but we have not been able to do that yet. Hopefully this might help someone else struggling with similar problems.

/Michael

Troubleshooting SCSM

System Center Service Manager can be a bit cumbersome to troubleshoot if you don't know where to look. In this blog I have collected a few useful guides which may help you on your way.

Places to look

Windows Event Log:
Events can be found under the Operations Manager log.

Tracing:
SMTrace can be useful. See http://www.concurrency.com/blog/enable-tracing-in-service-manager/ for examples.

Workflows:
Take a look in the console under Administration > Workflows > Status

Windows cannot find the Microsoft Software License Agreement

I ran into an issue when trying to install a Windows 8.1 eval client on my Client Hyper-V.

After booting up and pressing a key to start installation from CD/DVD I could select my regional settings, but I hit a wall when I press the "Install now" button:

My solution: Assign more than 512MB startup RAM.

According to System requirements for Windows 8.1
You need 1024 MB for a 32-bit and 2048 for 64-bit installation to run Windows 8.1

I first created a VM with only 512 MB as startup RAM and checked for "Enable dynamic memory". But I had to assign 1024 MB RAM as startup RAM to be able to kick off the setup.

Customizing Offline Servicing of Operating System Images

This blog describes how to change the drive used for Offline Servicing of operating system images in System Center 2012 Configuration Manager. Very useful if your C drive is a bit to small...

http://blogs.technet.com/b/configmgrteam/archive/2013/07/15/customizing-offline-servicing-of-operating-system-images.aspx

Run ConfigMgr cmdlets from Orchestrator

This week I was asked to create an Orchestrator runbook that would add a new computer object to ConfigMgr. At first glance I thought this would be straight forward task, since there is an Integration Pack (IP) for System Center 2012 R2 Configuration Manager.

But this IP does only contain a subset of the tasks you could wish for.

As you can see there are no activity "Import Computer Information" available in the IP from Microsoft.

Well. I'll just use the ConfigMgr cmdlet in a Run .Net Script activity. It supports powershell as a script language. So this should be managable.

The cmdlet is called Import-CMComputerInformation and is documentet on technet: http://technet.microsoft.com/en-us/library/jj821991(v=sc.20).aspx

Basically you can run

Import-CMComputerInformation -ComputerName "NewComputerName" -MacAddress "00:11:22:33:44:55"

But first you need to import the ConfigMgr module in powershell:

1:  Import-Module -Name "$(split-path $Env:SMS_ADMIN_UI_PATH)\ConfigurationManager.psd1"  

Cool. So you can think if we just have the ConfigMgr cmdlets available on the runbook server we should be ok. Not so fast. In Orchestrator powershell commands are run in a Run .Net Script activity. Unfortunately the Run .Net Activity only supports version 2 of Powershell. So this won't work on the Orchestrator Runbook Servers, at least not directly. You could of course invoke a script that start itself in Powershell version 3. But my solution is to create a PSSession against the ConfigMgr server and run the commands directly.

I have created a few global variables in the Orchestrator environment:
- {CM_ADMIN_USER} - a user with privileges in ConfigMgr.
- {CM_ADMIN_PW}     - the users password.
- {CM_SITE_SERVER}  - the site server we connect to.

These are the Powershell commands I execute in the Run .Net Script activity:

$username = "{CM_ADMIN_USER}"
$password = "{CM_ADMIN_PASSWORD}"
$cred = New-Object System.Management.Automation.PSCredential -ArgumentList @($username,(ConvertTo-SecureString -String $password -AsPlainText -Force))

$session = New-PSSession -ComputerName "{CM_SITE_SERVER}" -Credential $cred
$result = Invoke-Command -session $session -ScriptBlock {
    Import-Module -Name "$(split-path $Env:SMS_ADMIN_UI_PATH)\ConfigurationManager.psd1"
    $PSD = Get-PSDrive -PSProvider CMSite
    CD "$($PSD):"
    Import-CMComputerInformation -CollectionName "All Systems" -ComputerName "{New_Name}" -MacAddress "{MacAddress_from_Input}"

}
Remove-PSSession -Session $Session

Curly brackets is a placeholder for information in Orchestrator. It can be global variables or published data from previous actions. My runbook takes the Location the machine is placed, if it is a desktop or a laptop, and the MacAddress of the computer, queries the database for the last name with the given prefix based on this information, creates a NewName and imports the computer with the NewName and MacAddress.


EDIT:
Dexter Dhami gave me a tip about import-module pointed out by MVP David O Brien.
So I have replaced

Import-Module ($Env:SMS_ADMIN_UI_PATH.Substring(0,$Env:SMS_ADMIN_UI_PATH.Length-5) + "\ConfigurationManager.psd1")

with

Import-Module -Name "$(split-path $Env:SMS_ADMIN_UI_PATH)\ConfigurationManager.psd1"

Finding ConfigMgr SQL Server with Powershell

$sccmsrv=  "CM01" 

$site = (gwmi -ComputerName $sccmsrv -Namespace root\sms `
   -Class SMS_ProviderLocation).sitecode  
    
# enumerating SQL server name for the given SCCM site server  
$sccmCompquery = gwmi -q "Select distinct SiteSystem, Role, `
   SiteCode FROM SMS_SiteSystemSummarizer where role = `
  'SMS SQL Server' and siteCode = '$site' ORDER BY SiteCode" `
  -namespace "root\sms\site_$site" -ComputerName $sccmsrv  

[string]$tmpstr = `
   [regex]::Match($sccmCompquery.sitesystem, "\\\\w+\\$") 

$tmpstr = ($sccmCompquery.SiteSystem).SubString([regex]::Match( `
   $sccmCompquery.SiteSystem, "]\\").Index + 3)

$sccmSQLServer = $tmpstr.replace("\", "")

$sccmDB = "CM_$site"

Searching for smsts.log... again

When I troubleshoot operating system deployments that are failing I usually start looking in the smsts.log for obvious errors or clues for what might have gone wrong. But ever so often I keep looking for the correct path to the log file...

smsts.log moves around during the different phases of the task sequence as described in the table below

Task Sequence Phase	Location of smsts.log
WinPE before disk is formatted	X:\Windows\temp\SMSTSLog\
WinPE after disk is formatted	C:\_SMSTaskSequence\Logs\SMSTSLog\
Windows OS prior to installation of SCCM agent	C:\_SMSTaskSequence\Logs\SMSTSLog\
Windows OS after installation of SCCM agent	C:\Windows\System32\CCM\Logs\SMSTSLog\
Windows OS x64 after installation of SCCM agent	C:\Windows\SysWOW64\CCM\Logs\SMSTSLog\
Windows OS after task sequence has completed	C:\Windows\System32\CCM\Logs\
Windows OS x64 after Task Sequence has completed	C:\Windows\SysWOW64\CCM\Logs\

You can also create a step in you task sequence that will copy the logs to a network share if the deployment fails. More about that in a later post.