Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

azcopy thinks file is directory when sas token has slashes #873

Closed
astrahan87 opened this issue Feb 20, 2020 · 27 comments · Fixed by #1412
Closed

azcopy thinks file is directory when sas token has slashes #873

astrahan87 opened this issue Feb 20, 2020 · 27 comments · Fixed by #1412

Comments

@astrahan87
Copy link

astrahan87 commented Feb 20, 2020

Which version of the AzCopy was used?

AzCopy 10.3.4

Which platform are you using? (ex: Windows, Mac, Linux)

Linux
(Running in a powershell script on a Jenkins node)

What command did you run?

$sourceUri = az storage blob generate-sas ... --full-uri
$targetUri = az storage blob generate-sas ... --full-uri
azcopy copy $sourceUri $targetUri --overwrite true

What problem was encountered?

Got the error:failed to perform copy command due to error: cannot use directory as source without --recursive or a trailing wildcard (/*)

The files being copied are .vhd files and definitely not directories.
The primary problem occurred when changing from using the Az powershell module to the az cli to generate the SAS URIs. e.g.:

New-AzStorageBlobSASToken ... -FullUri
# Changed to:
az storage blob generate-sas ... --full-uri

The issue seems to be that the az cli does not properly escape the / character. e.g.:

  • az cli output: https://(account).blob.core.windows.net/(container)/(filename).vhd?...&sig=.../.../...
  • Az powershell module output: https://(account).blob.core.windows.net/(container)/(filename).vhd?...&sig=...%2F...%2F...

This is probably also a bug for the az cli, but I think azcopy should update its parsing to better handle the SAS tokens. The current parsing should not mistake a file for a directory because of URI parameters.

How can we reproduce the problem in the simplest way?

Use the az cli to generate two file Uris+SAS and see if they have an unescaped / in the SAS token.

Have you found a mitigation/solution?

Switching back to the AZ powershell module works.
Manually escaping the slashes prior to the azcopy call should also work.

@JohnRusk
Copy link
Member

@adreed-msft Can you take a look, and add to backlog as appropriate?

@adreed-msft
Copy link
Member

Sure thing, though I have my suspicions this is actually fixed after the text processing errors PR. I'll check both sides of that.

@Plasma
Copy link

Plasma commented Feb 20, 2020

Myself and others have been getting this error also, see MicrosoftDocs/azure-docs#41969 (comment)

@astrahan87
Copy link
Author

I've created an issue on the az-cli side to fix the encoding: Azure/azure-cli#12291

@Guilhermeslucas
Copy link

Im having the same issue when generating a sas token with the ip information as well, trying to copy from a blob to fileshare. Anyone has fixed that?

@JohnRusk
Copy link
Member

Depending on the result of @adreed-msft 's check, above, it may be fixed in the upcoming release 10.4. (No date announced)

@skmkzyk
Copy link

skmkzyk commented Apr 6, 2020

As far as I tested, SAS token with '/' will work fine when I hit the URL in the browser.
So I think it's the issue of the azcopy itself, not az-cli.

The SignedVersion from az-cli is sv=2018-03-28, and same from Powershell is sv=2019-02-02.
PowerShell generated vaule sv=2019-02-02 is same as Azure Portal generated.
Would you please investigate how azcopy handles URL parameters?

@JohnRusk
Copy link
Member

JohnRusk commented Apr 6, 2020

Given that there's a workaround (to escape the slashes) we won't look at this right now. But we'll leave this issue open to remind us to look at it later.

@inju
Copy link

inju commented Jul 23, 2020

Not solved in AzcopyVersion 10.5.0

Trying to fetch a ZIP file from Azure storage on MacOS (tried zsh and pwsh). File URL copied from portal and do get failure: URL is treated as directory

Note, the SAStoken has "/" escaped with %2F, does not help. GET request shows in URL still "restype=directory"

@patridge
Copy link

Is there a workaround for this? I saw mention of escaping the slashes, but that doesn't seem to work.

I just confirmed the issue happens in 10.5.1, causing an error on this Microsoft Learn module. When the tokens are retrieved in the portal there, the slashes are already escaped.

@astrahan87
Copy link
Author

@patridge To workaround it, I've been using the powershell module instead of az cli:

$sourceUri = New-AzStorageBlobSASToken ... -FullUri
$targetUri = New-AzStorageBlobSASToken ... -FullUri
azcopy cp $sourceUri $targetUri --overwrite true

@connorcarolan
Copy link

Version 10.6.0
Hit this same error message when copying files that contained a plus symbol in their path (not related to the SAS token). My workaround was to encode the file path, converting + to %2B:

folder+one/foldertwo/file.txt
folder%2Bone/foldertwo/file.txt

@arroyc
Copy link
Member

arroyc commented Jan 20, 2021

AzCopy 10.8.0 ... it was working well for us ... Since last week we are seeing this error

echo
+ echo 'Blob '\''dotnet-5.0.102.tar.gz'\'' does not exist in Prod storage container '\''dotnet'\''. Copying it...'
+ /tmp/azcopy-tool/azcopy copy 'https://oryxsdksdev.blob.core.windows.net/dotnet/dotnet-5.0.102.tar.gz?***' 'https://oryxsdksprod.blob.core.windows.net/dotnet/dotnet-5.0.102.tar.gz?***'

Blob 'dotnet-5.0.102.tar.gz' does not exist in Prod storage container 'dotnet'. Copying it...
INFO: Scanning...

failed to perform copy command due to error: cannot use directory as source without --recursive or a trailing wildcard (/*)



We are trying to copy a blob from one storage account to another with SAS token. Is there any workaround?

@arroyc
Copy link
Member

arroyc commented Jan 21, 2021

any update on this?

@jackfoxy
Copy link

jackfoxy commented Feb 13, 2021

AzCopy 10.8.0 not working for me either

failed to perform copy command due to error: cannot use directory as source without --recursive or a trailing wildcard (/*)

there are not slashes in my SAS token.

Trying to copy files from one storage account's blobs to another storage account's file share.

@nakulkar-msft
Copy link
Member

@jackfoxy This error usually means azcopy could not list the properties of a certain path on the source account - invalid path or incorrect/insufficient permissions. Can you verify from azcopy log file if it indeed was the case?

@jackfoxy
Copy link

@nakulkar-msft thanks for reaching out.
azcopy is being inconsistent about writing logs to user/.azcopy.
The most recent log timestamps appears to be from earlier tests that failed for other reasons.

I ran the test just now, which wrote the following to stout:

INFO: Scanning...

failed to perform copy command due to error: cannot use directory as source without --recursive or a trailing wildcard (/*)

but did not write a log file with today's timestamp. (And none of my last tests from yesterday have a matching timestamp. I know the TS is in UTC.)

Other odd things about the logs: it records delete statements for the dest file, even though there was no file to delete, all the messages are for the dest paths, no messages at all about the source path.

@jackfoxy
Copy link

and the log version is AzcopyVersion 10.7.0, even though I am running 10.8.0

@jackfoxy
Copy link

The only messages about properties look like this:

2021/02/12 23:17:59 Parallelize getting file properties (file.Stat): false (Based on AZCOPY_PARALLEL_STAT_FILES environment variable)
2021/02/12 23:17:59 INFO: [P#0-T#6] https://bdaasprodsouthcentralus.file.core.windows.net/locsim-data/tachcorp/fits/cfrfb202081998/final?se=2021-03-14t23%3A17%3A58z&sig=-REDACTED-&sp=rwdl&sr=s&sv=2018-03-28 (folder properties) Queuing folder, to be deleted after it's children are deleted Dst: fits/cfrfb202081998/final

@qianz2
Copy link

qianz2 commented Feb 18, 2021

Please take this issue into priority

@arroyc
Copy link
Member

arroyc commented Feb 19, 2021

I'm still getting this issue .. any update on this?

@arroyc
Copy link
Member

arroyc commented Feb 25, 2021

any update on this? I'm still seeing this issue .. this is blocking our weekly release. What is the resolution? Seems like this issue is there since a month and moving from one version to another version. We regenerated the token multiple times with keeping the start date in past, but no way we could make it work.

+ devStorage='https://oryxsdksdev.blob.core.windows.net/dotnet/defaultVersion.txt?***'
+ prodStorage='https://oryxsdksprod.blob.core.windows.net/dotnet/defaultVersion.txt?***'
+ /tmp/azcopy-tool/azcopy cp 'https://oryxsdksdev.blob.core.windows.net/dotnet/defaultVersion.txt?***' 'https://oryxsdksprod.blob.core.windows.net/dotnet/defaultVersion.txt?***'

Blob 'dotnet-5.0.102.tar.gz' already exists in Prod storage container 'dotnet'. Skipping copying it...

Blob 'defaultVersion.txt' does not exist in Prod storage container 'dotnet'. Copying it...
INFO: Scanning...

failed to perform copy command due to error: cannot use directory as source without --recursive or a trailing wildcard (/*)

As you can see we are trying to copy a file and not a directory, but somehow it always detect this as directory

@zezha-msft @JohnRusk

@patridge
Copy link

Not sure if others are experiencing the same problem I encountered when they got this message, but my error may not have been related to the token containing the slash as much as I was trying to operate on several objects in a container, for which my SAS token was not configured to allow. I was able to resolve this issue by adding Object to the Allowed resource types when I generated a SAS (shared access signature).

@st0le
Copy link

st0le commented Sep 2, 2021

@patridge comment helped resolved this for me. The error message was not helpful. Perhaps the error should guide the user to validate their token.

@adreed-msft
Copy link
Member

There's still a PR open related to this issue, but in 10.12.1, /s in SAS tokens work as expected.

@ziggy90127
Copy link

This fixed my problem. Had to select "object" when generating the SAS token.

@gapra-msft
Copy link
Member

The above PR has been merged to improve error reporting when determining if a source is a directory. Closing this issue as a result. If you are still experiencing any issues, please reopen the issue or create a new one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.