Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Az.Storage version 6.1.1 cmdlets performance issue while using Federated Credential service connection. #25377

Closed
shpengmsft opened this issue Jun 25, 2024 · 3 comments
Closed

Az.Storage version 6.1.1 cmdlets performance issue while using Federated Credential service connection. #25377

shpengmsft opened this issue Jun 25, 2024 · 3 comments
Assignees
@blueww
Labels
question The issue doesn't require a change to the product in order to be resolved. Most issues start as that

Comments

@shpengmsft
Copy link
Member

Description

This issue happened with Azure DevOps release pipeline.

This pipeline is using AzurePowershell@5 tasks to upload/update blobs on specific storage account. (These blobs are Python wheels or indexes.)

When the pipeline is using a service connection with Federated Credentials.

In the first 3~4 minutes, all these cmdlets (Set.AzContext, Set.AzCurrentStorageAccount) work fine. After that, we observed one of these cmdlets' executions time will be longer than 2.5 hours.

Detailed repro steps. Please contact me via internal email.

Issue script & Debug output

2024-06-20T23:13:35.7192921Z ##[section]Starting: Deploy packages/index
...
2024-06-20T23:14:16.8909540Z     Get-Blobs:: Getting blobs from storage '<***** Storage account name *****>', container 'wheels' with prefix '<***** Blob prefix *****>' and suffix '.html'
2024-06-20T23:14:16.8909932Z     Get-Blobs::Get-AzContext
2024-06-20T23:14:16.8910231Z     Get-Blobs:: Set-AzCurrentStorageAccount took 0.5183503 seconds
2024-06-20T23:15:49.0161572Z     Get-Blobs:: Get-AzStorageBlob took 92.72106 seconds
2024-06-20T23:15:49.0299267Z     Get-Blobs:: Found 165 blobs (excluding snapshots)
2024-06-20T23:15:49.0326070Z Creating backups for current target index
2024-06-20T23:15:49.0417778Z     Create-Backup:: Back-up container is not specified. Skipping 165 blobs
2024-06-20T23:15:49.0492424Z     Create-Snapshots:: Snapshots are not enabled. Skipping 165 blobs
2024-06-20T23:15:49.0524583Z 0
2024-06-20T23:15:49.0528154Z 1
2024-06-20T23:15:49.0570142Z Updating index
2024-06-20T23:15:49.0761010Z     Create-Index:: Creating index for <***** Storage account name *****>.azureedge.net endpoint
2024-06-20T23:15:49.0837207Z         Create-Index:: Looking for blobs in a location with id 'arm|<***** Storage account name *****>|<***** Storage account name *****>|<***** Subscription Name *****>|repo|<***** Blob prefix *****>'
2024-06-20T23:15:49.0862404Z             Get-Blobs:: Getting blobs from storage '<***** Storage account name *****>', container 'repo' with prefix '<***** Blob prefix *****>' and suffix '.whl'
2024-06-20T23:15:49.0890365Z             Get-Blobs::Get-AzContext
2024-06-20T23:15:49.6931297Z             Get-Blobs:: Set-AzCurrentStorageAccount took 0.5625513 seconds
2024-06-20T23:16:47.7566855Z             Get-Blobs:: Get-AzStorageBlob took 58.0590726 seconds
2024-06-20T23:16:47.7598845Z             Get-Blobs:: Found 10146 blobs (excluding snapshots)
2024-06-20T23:16:47.8496193Z         Create-Index:: New-AzStorageBlobSASToken generated SAS token for '<***** .whl ****>'
2024-06-20T23:16:47.8496193Z         Create-Index:: New-AzStorageBlobSASToken generated SAS token for '<***** .whl ****>'
2024-06-20T23:16:47.8496193Z         Create-Index:: New-AzStorageBlobSASToken generated SAS token for '<***** .whl ****>'
...
2024-06-20T23:22:47.3379612Z         Create-Index:: Looking for blobs in a location with id 'arm|<***** Storage account name *****>|<***** Storage account name *****>|<***** Subscription Name *****>|wheels|<***** Blob prefix *****>'
2024-06-20T23:22:47.3405929Z             Get-Blobs:: Getting blobs from storage '<***** Storage account name *****>', container 'wheels' with prefix '<***** Blob prefix *****>' and suffix '.whl'
2024-06-20T23:22:47.3435561Z             Get-Blobs::Get-AzContext
2024-06-21T01:47:55.7849622Z             Get-Blobs:: Set-AzCurrentStorageAccount took 8708.3301906 seconds
2024-06-21T01:49:24.1209139Z             Get-Blobs:: Get-AzStorageBlob took 88.3414451 seconds

Environment data

N/a please contact me via internal email.

Module versions

N/a please contact me via internal email.

Error output

N/a please contact me via internal email.
@shpengmsft shpengmsft added bug This issue requires a change to an existing behavior in the product in order to be resolved. needs-triage This is a new issue that needs to be triaged to the appropriate team. labels Jun 25, 2024
@microsoft-github-policy-service microsoft-github-policy-service bot removed the needs-triage This is a new issue that needs to be triaged to the appropriate team. label Jun 25, 2024
@blueww
Copy link
Member

blueww commented Jun 26, 2024 •
edited
Loading

Hi @shpengmsft

How you get above log? (It looks not Azure Powershell cmdlets log.)
Would you please share more information the issue template asked, so we can continue follow up the issue?

I will write a mail to you, you can share the info in mail.

BTW, the Azure DevOps pipeline AzurePowershell@5 task not owned by our team. You might need contact Azure Devops to get support on the AzurePowershell@5 task.

@blueww blueww self-assigned this Jun 26, 2024
@blueww blueww added question The issue doesn't require a change to the product in order to be resolved. Most issues start as that and removed bug This issue requires a change to an existing behavior in the product in order to be resolved. labels Jun 26, 2024
@isra-fel
Copy link
Member

It's a complicated problem that involves in Azure Devops pipeline, Federated Identity, possible token cache persistance and MSAL. I'll update this issue once we've figure out the root cause.

@shpengmsft
Copy link
Member Author

It's a complicated problem that involves in Azure Devops pipeline, Federated Identity, possible token cache persistance and MSAL. I'll update this issue once we've figure out the root cause.

I believe so. Thanks for reaching out via internal email. Let's use that to continue the investigation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@blueww blueww

Labels
question The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@blueww @isra-fel @shpengmsft