Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restore-AzWebAppSnapshot fails when large number of resources exist (sometimes) #26268

Open
danbuchal-mom opened this issue Oct 10, 2024 · 0 comments
Open

Restore-AzWebAppSnapshot fails when large number of resources exist (sometimes) #26268

danbuchal-mom opened this issue Oct 10, 2024 · 0 comments
Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. customer-reported needs-triage This is a new issue that needs to be triaged to the appropriate team.

Comments

@danbuchal-mom
Copy link

danbuchal-mom commented Oct 10, 2024 •
edited
Loading

Description

Restore-AzWebAppSnapshot can fail when a large number of resources exist in the subscription.
Restore-AzWebAppSnapshot calls the "resources" API which returns paginated results.

If the Web App of the Snapshot is not returned on the first "page" of results, the restore operation will fail with an InternalServerError

Note: a "filter" is passed to this API call, however, it appears that the Resources API is first retrieving all results, then only filtering using the first page.

I suspect the above as I have tested the API call using the Azure Resource Explorer (https://resources.azure.com/subscriptions/[subscription-guid-redacted]/resources)
We have 24 different apps and only have consistent issues restoring 5 of them with this call.

All 5 of the affected web apps do not appear on the first "page" of results in Azure Resource Explorer, while all working apps DO appear on the first page.

In addition, we have one web app that works some of the time.
Refreshing the results in Azure Resource Explorer shows that sometimes this app appears on the first page and sometimes it does not.

Note: In the debug output below you will see the call to the Resources API
https://management.azure.com/subscriptions/[redacted]/resources?$filter=resourceType eq 'Microsoft.Web/sites' and name eq '[redacted]'&api-version=2016-09-01

Observe it returns an empty array with a "nextLink"

Issue script & Debug output

$snapshot = (Get-AzWebAppSnapshot -Name $WebAppName -ResourceGroupName $ResourceGroupName)[0]

DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions/[redacted]/resourceGroups/[redacted]/providers/Microsoft.Web/sites/[redacted]/snapshots?api-version=2021-01-15

Headers:
x-ms-client-request-id        : 8b732e64-9590-447f-8ef0-a81b8b8f317c
Accept-Language               : en-US

Body:



DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
ETag                          : "1DB0EE97209AB0B"
Strict-Transport-Security     : max-age=31536000; includeSubDomains
x-ms-request-id               : 86bd8e45-aff8-4a31-8c31-059e0458e97e
X-AspNet-Version              : 4.0.30319
X-Powered-By                  : ASP.NET
x-ms-ratelimit-remaining-subscription-reads: 249
x-ms-ratelimit-remaining-subscription-global-reads: 3749
x-ms-correlation-request-id   : d27badb3-c7e0-474e-9c7e-44c996ba1411
x-ms-routing-request-id       : EASTUS:20241010T204514Z:d27badb3-c7e0-474e-9c7e-44c996ba1411
X-Content-Type-Options        : nosniff
X-Cache                       : CONFIG_NOCACHE
X-MSEdge-Ref                  : Ref A: E117A506F88B46559CD0664687396A12 Ref B: MNZ221060610029 Ref C: 2024-10-10T20:45:13Z
Date                          : Thu, 10 Oct 2024 20:45:14 GMT

Body:
{
  "value": [
    {
      "id": "/subscriptions/[redacted]/resourceGroups/[redacted]/providers/Microsoft.Web/sites/[redacted]/snapshots/2024-09-11T00_25_46_4226984",
      "name": "2024-09-11T00:25:46.4226984",
      "type": "Microsoft.Web/sites/snapshots",
      "properties": {
        "time": "2024-09-11T00:25:46.4226984"
      }
    },
    ...
  ]
}
Restore-AzWebAppSnapshot -Name $TargetWebAppName -ResourceGroupName $TargetResourceGroupName -InputObject $snapshot -Force -RecoverConfiguration -Debug

DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions/[redacted]/resources?$filter=resourceType eq 'Microsoft.Web/sites' and name eq '[redacted]'&api-version=2016-09-01

Headers:
x-ms-client-request-id        : bb36d532-556d-4e7a-90a4-f7f55f0bafad
Accept-Language               : en-US

Body:



DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
x-ms-ratelimit-remaining-subscription-reads: 249
x-ms-ratelimit-remaining-subscription-global-reads: 3749
x-ms-request-id               : 52ce3c1d-ac04-4795-bd9a-84a4e949f186
x-ms-correlation-request-id   : 52ce3c1d-ac04-4795-bd9a-84a4e949f186
x-ms-routing-request-id       : EASTUS:20241010T204621Z:52ce3c1d-ac04-4795-bd9a-84a4e949f186
Strict-Transport-Security     : max-age=31536000; includeSubDomains
X-Content-Type-Options        : nosniff
X-Cache                       : CONFIG_NOCACHE
X-MSEdge-Ref                  : Ref A: E4B43C8E16E345328AA4FB9B3127D5CB Ref B: MNZ221060618037 Ref C: 2024-10-10T20:46:21Z
Date                          : Thu, 10 Oct 2024 20:46:20 GMT

Body:
{
  "value": [],
  "nextLink": "https://management.azure.com/subscriptions/[redacted]/resources?%24filter=resourceType+eq+%27Microsoft.Web%2fsites%27+and+name+eq+%27[redacted]%27&api-version=2016-09-01&%24skiptoken=rZFNa4NAEED%2fi2cPm9VcAj20XWMNzoplzXb3JkbqR1TQFaOh%2f73bIC2BhID0OgNvHvPORp2elJ%2fXZWdszkbdtCpL0lq18bHXEyMdd738yBDj2zHG%2b9ErmpxWoSU4DHQCRXk00hEhwUrb584JsFCiEEhU0TogyRowDF6NngzT6Jr%2bMVvwENMpQcCelWRg0VeEAu6MPvscBHcUMAeBC3ZAwKYsQjM7jTvVd%2fiOsHTDUZJwLXmkaLU9Sg2FybkICyYUJTKnTOayEBYtDtUMffQHgWUGPFwBdpR2Wf24SlIO%2bg8ocDUWvxd6d5LEmWSxy65c76gK7NlAIksWiQrYSwm5Vq32lc%2b9AapIBS6sxJRYAckyIIf8wvwy%2fyoG9VtzPOiW8xDitkxbfe067ea%2fwprXWW9yl0Q156Q3gUuCmr85byKXxJwd7%2fCWhPwG"
}

DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
POST

Absolute Uri:
https://management.azure.com/subscriptions/[redacted]/resourceGroups/[redacted]/providers/Microsoft.Web/sites/[redacted]/restoreSnapshot?api-version=2021-01-15

Headers:
x-ms-client-request-id        : bb36d532-556d-4e7a-90a4-f7f55f0bafad
Accept-Language               : en-US

Body:
{
  "properties": {
    "snapshotTime": "2024-10-10T19:49:23",
    "recoverySource": {},
    "overwrite": true,
    "recoverConfiguration": true,
    "ignoreConflictingHostNames": true,
    "useDRSecondary": false
  }
}

...

DEBUG: ============================ HTTP RESPONSE ============================


Status Code:
InternalServerError

Headers:
Cache-Control                 : no-cache
Cache-Control                 : no-cache
Pragma                        : no-cache
Strict-Transport-Security     : max-age=31536000; includeSubDomains
X-AspNet-Version              : 4.0.30319
X-Powered-By                  : ASP.NET
x-ms-failure-cause            : service
x-ms-ratelimit-remaining-subscription-writes: 199
x-ms-ratelimit-remaining-subscription-global-writes: 2999
x-ms-request-id               : e65257ab-3bba-4119-8811-d379bf759616
x-ms-correlation-request-id   : e65257ab-3bba-4119-8811-d379bf759616
x-ms-routing-request-id       : EASTUS:20241010T204644Z:e65257ab-3bba-4119-8811-d379bf759616
X-Content-Type-Options        : nosniff
X-Cache                       : CONFIG_NOCACHE
X-MSEdge-Ref                  : Ref A: D5FB6060669A470A849BE4CBC9E966BF Ref B: MNZ221060610047 Ref C: 2024-10-10T20:46:44Z
Date                          : Thu, 10 Oct 2024 20:46:43 GMT

Body:
{
  "Message": "An error has occurred."
}

Environment data

Name                           Value
----                           -----
PSVersion                      7.2.23
PSEdition                      Core
GitCommitId                    7.2.23
OS                             Microsoft Windows 10.0.22621
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     3.0.2                 Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault…}
Script     5.1.0                 Az.KeyVault                         {Add-AzKeyVaultCertificate, Add-AzKeyVaultCertificateContact, Add-AzKeyVaultKey, Add-AzKe… 
Script     6.13.0                Az.Resources                        {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment…}   
Script     1.2.2                 Az.TrafficManager                   {Add-AzTrafficManagerCustomHeaderToEndpoint, Add-AzTrafficManagerCustomHeaderToProfile, A… 
Script     3.1.2                 Az.Websites                         {Add-AzWebAppAccessRestrictionRule, Add-AzWebAppTrafficRouting, Edit-AzWebAppBackupConfig…

Error output

Message        : Operation returned an invalid status code 'InternalServerError'
StackTrace     :    at Microsoft.Azure.Management.WebSites.WebAppsOperations.BeginRestoreSnapshotWithHttpMessagesAsync(String resourceGroupName, String name,   
                 SnapshotRestoreRequest restoreRequest, Dictionary`2 customHeaders, CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.WebSites.WebAppsOperations.RestoreSnapshotWithHttpMessagesAsync(String resourceGroupName, String name,        
                 SnapshotRestoreRequest restoreRequest, Dictionary`2 customHeaders, CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.WebSites.WebAppsOperationsExtensions.RestoreSnapshotAsync(IWebAppsOperations operations, String
                 resourceGroupName, String name, SnapshotRestoreRequest restoreRequest, CancellationToken cancellationToken)
                    at Microsoft.Azure.Management.WebSites.WebAppsOperationsExtensions.RestoreSnapshot(IWebAppsOperations operations, String
                 resourceGroupName, String name, SnapshotRestoreRequest restoreRequest)
                    at Microsoft.Azure.Commands.WebApps.Utilities.WebsitesClient.RestoreSnapshot(String resourceGroupName, String webSiteName, String
                 slotName, SnapshotRestoreRequest restoreReq)
                    at Microsoft.Azure.Commands.WebApps.Cmdlets.BackupRestore.RestoreAzureWebAppSnapshot.<>c__DisplayClass17_0.<ExecuteCmdlet>b__0()
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ConfirmAction(Boolean force, String continueMessage, String
                 processMessage, String target, Action action, Func`1 useShouldContinue)
                    at Microsoft.Azure.Commands.ResourceManager.Common.AzureRMCmdlet.ConfirmAction(Boolean force, String continueMessage, String
                 processMessage, String target, Action action)
                    at Microsoft.Azure.Commands.WebApps.Cmdlets.BackupRestore.RestoreAzureWebAppSnapshot.ExecuteCmdlet()
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.<>c__3`1.<ExecuteSynchronouslyOrAsJob>b__3_0(T c)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet, Action`1 executor)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
Exception      : Microsoft.Azure.Management.WebSites.Models.DefaultErrorResponseException
InvocationInfo : {Restore-AzWebAppSnapshot}
Line           : Restore-AzWebAppSnapshot -Name [redacted] -ResourceGroupName [redacted] -InputObject $snapshot -Force
                 -RecoverConfiguration -Debug
@danbuchal-mom danbuchal-mom added bug This issue requires a change to an existing behavior in the product in order to be resolved. needs-triage This is a new issue that needs to be triaged to the appropriate team. labels Oct 10, 2024
@microsoft-github-policy-service microsoft-github-policy-service bot added customer-reported needs-triage This is a new issue that needs to be triaged to the appropriate team. and removed needs-triage This is a new issue that needs to be triaged to the appropriate team. labels Oct 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. customer-reported needs-triage This is a new issue that needs to be triaged to the appropriate team.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@danbuchal-mom