[Part-5]Private Cloud: Some common issues and troubleshooting tips   Leave a comment


This is the fifth and final post in the series of blog posts on how to build own Private Cloud using Windows Azure Pack (WAP).

 Here is a list of topics for the entire series:

 

1. PrivateCloud_Icon[Part-1]Private Cloud: Introduction and installation of pre-requisites

[Part-2]Private Cloud: Configure SCVMM and SPF

[Part-3]Private Cloud: Install and Configure WAP

[Part-4]Private Cloud: Demo of provisioning VM and SQL DB using Tenant portal.

[Part-5]Private Cloud: Some common issues and troubleshooting tips.

 

In this blog I will provide some troubleshooting tips e.g. where to look in case of any error and few scenarios.

When it comes to troubleshooting, usually, the following questions comes into mind:

  1. Log files: Where and what to look at?
  2. Troubleshooting example
  3. Common issues/errors

I am going to cover the above mentioned topics in this blog post.

1. Log files: Where and what to look at?

It is important to know that every activity in WAP will create a corresponding job in SCVMM. So, the first place to look for is the related history of the job in SCVMM.

  • Go to SCVMM console > Jobs to check the job history.

p5_1

  • If nothing is evident from the job history, likely the error occurred at a prior stage reaching to SCVMM. Look at the Event Viewer logs in this case.
  • Go to Event Viewer > Applications and Services Logs > Microsoft

Look under the folder/component possibly raising the error. e.g. WindowsAzurePack

 

p5_2

 

  • If you want to dig deeper, enable debug tools called Developer Tool. Developer Tool is under Settings within Internet Explorer (IE 10).

p5_3

 

  • You need to Refresh the page to find the tracing tool at the bottom in IE 10.

p5_4

 

Now, you can co-relate things with the other log files e.g.  HTTPLOG.

  • To see HTTPLOGs, go to Run prompt, type logfiles and open HTTPERR

p5_5

 

2. Troubleshooting example

Let us see one of the scenarios I faced and the troubleshooting steps to reoslve.

ISSUE:

I was getting following error when I was logging into Admin portal with a red icon.

p5_6

As a first troubleshooting step, I looked at the SCVMM job history, nothing obvious there. It was likely because error occurred while loading the page indicating it has occurred prior to VMM.

As a next step, I went to Event Viewer > Application and Services Logs > Microsoft > WindowsAzurePack > MgmtSvc-AdminAPI

p5_7

Not conclusive information in Event logs too.

Next step – I started the Developer Tools and looked into the Network Profile.

p5_8

As you see above, error is pretty clear – HTTPS Error 400 means Bad Request

Carefully reviewing event logs again, I got more clue. It seems to be related with user authentication token.

+++++++++++++++++++++++++++++++++++++++++++
<HttpContext>
    <User IsAuthenticated="true" Name="Manish\administrator" />
    <Request>
      <RawUrl>/Plan/List?ts=1423229252677</RawUrl>
      <UserHostAddress>10.52.18.171</UserHostAddress>
      <Headers>
        <Header Key="Cache-Control" Value="no-cache" />
        <Header Key="Connection" Value="Keep-Alive" />
        <Header Key="Content-Length" Value="56" />
        <Header Key="Content-Type" Value="application/json"
        <Header Key="x-ms-client-antiforgery-id" Value="ws3O12S2LNeg1msZCk1y0nO+LVQ8W9dOzX7n7RV0CFkwWoOnDx/xRrylvnxRsKeW2KL+hCIK0RkaT+ZAAFli+EdkqPiAzdQOLsGSpUnZh6BW8NO3SLTS/cg1Gd8IA0C9wXvEOeBzKEaeH1o1G5CpFw==" />
        <Header Key="x-ms-client-request-id" Value="63ef3416-4e64-450e-87b2-559a05fe851a-2015-02-06 13:27:32Z" />
        <Header Key="X-Requested-With" Value="XMLHttpRequest" />
        <Cookies>
          <Cookie Name="sub" Secure="false" Expires="0001-01-01T00:00:00Z" Domain="" Path="/" Value="Redacted (4 characters)" />
          <Cookie Name="AdminSiteFedAuth" Secure="false" Expires="0001-01-01T00:00:00Z" Domain="" Path="/" Value="Redacted (2000 characters)" />
          <Cookie Name="AdminSiteFedAuth1" Secure="false" Expires="0001-01-01T00:00:00Z" Domain="" Path="/" Value="Redacted (2000 characters)" />
          <Cookie Name="AdminSiteFedAuth2" Secure="false" Expires="0001-01-01T00:00:00Z" Domain="" Path="/" Value="Redacted (2000 characters)" />
          <Cookie Name="AdminSiteFedAuth3" Secure="false" Expires="0001-01-01T00:00:00Z" Domain="" Path="/" Value="Redacted (716 characters)" />
          <Cookie Name="__RequestVerificationToken_Admin" Secure="false" Expires="0001-01-01T00:00:00Z" Domain="" Path="/" Value="Redacted (128 characters)" /&gt
++++++++++++++++++++++++++++++++++++++++

HTTPLOG confirms this:

p5_9

It is now evident that their is a problem with the request being passed.Further research led me to this KB article which explains the root cause as authentication token length longer that allowed.

http://support.microsoft.com/kb/820129

I changed the user account to local Administrator and issue resolved.

So, looking at the logs mentioned in first section, you should be able to resolve most of the common issues.

3. Common issues/errors

1. You may get following error when you try to create a VM from WAP.

Failed to create virtual machine 'vmname'.
Failed to submit operation request.

Check the event log to find these errors:

 Log Name: Microsoft-WindowsAzurePack-MgmtSvc-TenantSite/Operational

 Source: Microsoft-WindowsAzurePack-MgmtSvc-TenantSite
 Date: 1/8/2015 3:16:51 AM
 Event ID: 232
 Task Category: (59)
 Level: Error
 Keywords: None
 User: IIS APPPOOL\MgmtSvc-TenantSite
 Computer: myvmserver.manish.com
 Description:
 Potential version mismatch between WAP and SPF, please verify both component's versions and if they're compatible, Exception: 'System.Data.Services.Client.DataServiceRequestException: An error occurred while processing this request. ---> System.Data.Services.Client.DataServiceClientException: <?xml version="1.0" encoding="utf-8"?><m:error xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"><m:code /><m:message xml:lang="en-US">An error occurred while processing this request.</m:message><m:innererror><m:message>The property 'DynamicMemoryMinimumMB' does not exist on type 'VMM.VirtualMachine'. Make sure to only use property names that are defined by the type.</m:message><m:type>Microsoft.Data.OData.ODataException</m:type><m:stacktrace> at Microsoft.Data.OData.ReaderValidationUtils.ValidateValuePropertyDefined(String propertyName, IEdmStructuredType owningStructuredType, ODataMessageReaderSettings messageReaderSettings, Boolean& ignoreProperty)

Resolution:

Follow this KB article if you are hitting the same issue.

http://support.microsoft.com/kb/2983453

If the KB article doesn’t help, it might be an issue with version mismatch. Please make sure all your component SCVMM, SPF, WAP are on the latest build.

2.  When you try to create a VM you may receive following error.

The server encountered an error while creating “Name of the VM”

 When you check the VMM log you will find errors mentioned below:

Error (12700)
VMM cannot complete the host operation on the myvmserver.manish.com server because of the error: The operation failed.
User “ Domain\manish” failed to create external configuration store at 'C:\ProgramData\Microsoft\Windows\Hyper-V\Test': A required privilege is not held by the client. (Virtual machine ID 0x80070522)
Unknown error (0x8000)
Recommended Action
Resolve the host issue and then try the operation again.

Cause:

By default the local User Right “Create Symbolic links” is assigned to the Adminstrators group. Once you install Hyper-V a extension is installed on this machine that allows Service SIDs to be used and the Service SID “Virtual Machines” is allowed to  “Create Symbolic links”

Resolution:

To get the necessary Service SID added, you need to edit the GptTmpl.inf that is associated with the GPO

Example with Administrators and added “Virtual Machines”
….
 [Privilege Rights]
 SeCreateSymbolicLinkPrivilege = *S-1-5-32-544, NT Virtual Machine\Virtual Machinesn the Sysvol directly. [You need to locate the correct GptTmp.inf by the GPO's Unique ID]

 

And, that concludes this series of Building Your Own Private Cloud using Windows Azure Pack.

Thank you all for following the series and our blog.

Stay tuned for some more interesting blogs.

 

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: