Fix decompression in web cmdlets #17955

iSazonov · 2022-08-24T07:33:41Z

PR Summary

Follow doesn't work for original cmdlet:

Invoke-WebRequest https://httpbin.org/deflate

I don't know why our tests are passed.

I guess there was something changed .Net from early versions.

The fix removes old workaround and turn on embedded automatic decompression in HttpClient handler.

PR Context

PR Checklist

PaulHigin · 2022-08-24T15:53:04Z

test/powershell/Modules/Microsoft.PowerShell.Utility/WebCmdlets.Tests.ps1


        # Validate response content
-        $headers.'Content-Encoding'[0] | Should -BeExactly $dataEncoding
+        # If Content is compressed we cannot read it as JSON.


I don't understand this comment. Do we need to convert content from json similar to above at line 601?

I think this comment is saying that if the content wasn't decompressed correctly then it wouldn't be valid JSON so the test would fail? If that is case, perhaps the comment should make that more clear.

@SteveL-MSFT What about:

Suggested change

# If Content is compressed we cannot read it as JSON.

# If Content is still compressed we cannot read it as JSON.

The thing is, there is no convertto-json call following this comment, so the comment doesn't seem to make sense.

Comment updated.

PaulHigin · 2022-08-24T15:53:54Z

@iSazonov

Overall, these changes look Ok to me. But I don't know what prompted this PR. Did the tests begin failing at some point? Or is this just code clean up to remove the current workaround and instead use the built-in automatic decompression setting?

iSazonov · 2022-08-24T18:01:21Z

@PaulHigin The Invoke-WebRequest https://httpbin.org/deflate did not work before the PR but our tests for deflate was passed.
And yes, .Net Runtime remove encoding headers after automatic decompression. I was wonder but it makes sense - if there was follow processing it shouldn't see any compression header since content is already not compressed.
I checked this with Wireshark:

Request
Response

Then the cmdlets return the response without Content-Encoding: deflate.

Second issue was with tests. I tried to keep them as near to original as possible. But I started to check with httpbin.org. The site send another response than our WebListener. httpbin.org response has a flag for used decompression method, WebListener haven't. I don't know how enhance WebListener. So I had to change tests.

I don't know why our tests passed before the PR for deflate. I did not investigate either a root is in WebListener or in old workaround.

ghost · 2022-09-06T14:01:22Z

This pull request has been automatically marked as Review Needed because it has been there has not been any activity for 7 days.
Maintainer, please provide feedback and/or mark it as Waiting on Author

ghost · 2022-09-14T14:01:27Z

This pull request has been automatically marked as Review Needed because it has been there has not been any activity for 7 days.
Maintainer, please provide feedback and/or mark it as Waiting on Author

src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/StreamHelper.cs

test/powershell/Modules/Microsoft.PowerShell.Utility/WebCmdlets.Tests.ps1

daxian-dbw · 2022-09-26T23:31:13Z

test/powershell/Modules/Microsoft.PowerShell.Utility/WebCmdlets.Tests.ps1


        # Validate response content
-        $headers.'Content-Encoding'[0] | Should -BeExactly $dataEncoding
+        # If Content is compressed we cannot read it as JSON.


The thing is, there is no convertto-json call following this comment, so the comment doesn't seem to make sense.

daxian-dbw

LGTM, with one suggestion.

test/powershell/Modules/Microsoft.PowerShell.Utility/WebCmdlets.Tests.ps1

…s.Tests.ps1 Co-authored-by: Dongbo Wang <dongbow@microsoft.com>

pull-request-quantifier-deprecated · 2022-09-28T03:34:23Z

This PR has 22 quantified lines of changes. In general, a change size of upto 200 lines is ideal for the best PR experience!

Quantification details

Label      : Extra Small
Size       : +8 -14
Percentile : 8.8%

Total files changed: 3

Change summary by file extension:
.cs : +1 -7
.ps1 : +7 -7

Change counts above are quantified counts, based on the PullRequestQuantifier customizations.

Why proper sizing of changes matters

Optimal pull request sizes drive a better predictable PR flow as they strike a
balance between between PR complexity and PR review overhead. PRs within the
optimal size (typical small, or medium sized PRs) mean:

Fast and predictable releases to production:
- Optimal size changes are more likely to be reviewed faster with fewer
  iterations.
- Similarity in low PR complexity drives similar review times.
Review quality is likely higher as complexity is lower:
- Bugs are more likely to be detected.
- Code inconsistencies are more likely to be detected.
Knowledge sharing is improved within the participants:
- Small portions can be assimilated better.
Better engineering practices are exercised:
- Solving big problems by dividing them in well contained, smaller problems.
- Exercising separation of concerns within the code changes.

What can I do to optimize my changes

Use the PullRequestQuantifier to quantify your PR accurately
- Create a context profile for your repo using the context generator
- Exclude files that are not necessary to be reviewed or do not increase the review complexity. Example: Autogenerated code, docs, project IDE setting files, binaries, etc. Check out the Excluded section from your prquantifier.yaml context profile.
- Understand your typical change complexity, drive towards the desired complexity by adjusting the label mapping in your prquantifier.yaml context profile.
- Only use the labels that matter to you, see context specification to customize your prquantifier.yaml context profile.
Change your engineering behaviors
- For PRs that fall outside of the desired spectrum, review the details and check if:
  - Your PR could be split in smaller, self-contained PRs instead
  - Your PR only solves one particular issue. (For example, don't refactor and code new features in the same PR).

How to interpret the change counts in git diff output

One line was added: +1 -0
One line was deleted: +0 -1
One line was modified: +1 -1 (git diff doesn't know about modified, it will
interpret that line like one addition plus one deletion)
Change percentiles: Change characteristics (addition, deletion, modification)
of this PR in relation to all other PRs within the repository.

Was this comment helpful? 👍 :ok_hand: :thumbsdown: (Email)
Customize PullRequestQuantifier for this repository.

daxian-dbw · 2022-10-03T22:30:57Z

@PaulHigin Can you please take another look?

CarloToso · 2022-10-04T12:20:40Z

This PR also fixes
Invoke-WebRequest https://httpbin.org/brotli

God-damnit-all · 2022-12-23T22:52:50Z

Is this going to make it into 7.4?

iSazonov · 2022-12-24T13:06:48Z

It is already in v7.4.0-preview.1

God-damnit-all · 2022-12-25T00:07:42Z

It is already in v7.4.0-preview.1

Oh, cool. Normally a bot responds to the PR saying if it made it in or whatever.

Fix decompression in web cmdlets

c17df7f

iSazonov added the CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log label Aug 24, 2022

iSazonov assigned PaulHigin Aug 24, 2022

iSazonov requested a review from PaulHigin as a code owner August 24, 2022 07:33

pull-request-quantifier-deprecated bot added the Extra Small label Aug 24, 2022

iSazonov requested a review from SteveL-MSFT August 24, 2022 07:33

iSazonov added 5 commits August 24, 2022 15:10

Fix tests

7852223

Fix tests 2

3ec5f92

Fix tests 3

8d3a69c

Fix tests 4

e7097d7

Fix tests 5

4700f54

PaulHigin suggested changes Aug 24, 2022

View reviewed changes

ghost added the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label Aug 24, 2022

ghost removed the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label Aug 24, 2022

ghost added the Review - Needed The PR is being reviewed label Sep 6, 2022

ghost removed the Review - Needed The PR is being reviewed label Sep 7, 2022

ghost added the Review - Needed The PR is being reviewed label Sep 14, 2022

daxian-dbw requested changes Sep 26, 2022

View reviewed changes

Feedback

e7917ad

daxian-dbw approved these changes Sep 27, 2022

View reviewed changes

test/powershell/Modules/Microsoft.PowerShell.Utility/WebCmdlets.Tests.ps1 Outdated Show resolved Hide resolved

Update test/powershell/Modules/Microsoft.PowerShell.Utility/WebCmdlet…

0849d44

…s.Tests.ps1 Co-authored-by: Dongbo Wang <dongbow@microsoft.com>

PaulHigin approved these changes Oct 3, 2022

View reviewed changes

iSazonov mentioned this pull request Oct 4, 2022

Fix WebCmdlets decompression of Deflate and Brotli encoding #18218

Closed

22 tasks

daxian-dbw assigned daxian-dbw and unassigned PaulHigin Oct 4, 2022

daxian-dbw merged commit 3a506b9 into PowerShell:master Oct 4, 2022

iSazonov deleted the web--stream-deflate branch October 4, 2022 17:11

iSazonov mentioned this pull request Oct 4, 2022

Add support for brotli compression to Web Cmdeltes #4947

Closed

CarloToso mentioned this pull request Nov 8, 2022

Web cmdlets: support compression #3178

Closed

CarloToso mentioned this pull request Jan 8, 2023

Create test to check if WebCmdlets decompress brotli-encoded data #18905

Merged

o-l-a-v mentioned this pull request Nov 8, 2023

Add Brotli decoding as a new feature of 7.4 MicrosoftDocs/PowerShell-Docs#10616

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix decompression in web cmdlets #17955

Fix decompression in web cmdlets #17955

iSazonov commented Aug 24, 2022 •

edited

Loading

PaulHigin Aug 24, 2022

SteveL-MSFT Aug 29, 2022

iSazonov Aug 30, 2022

daxian-dbw Sep 26, 2022

iSazonov Sep 27, 2022

PaulHigin commented Aug 24, 2022 •

edited

Loading

iSazonov commented Aug 24, 2022 •

edited

Loading

ghost commented Sep 6, 2022

ghost commented Sep 14, 2022

daxian-dbw Sep 26, 2022

daxian-dbw left a comment

pull-request-quantifier-deprecated bot commented Sep 28, 2022

What can I do to optimize my changes

How to interpret the change counts in git diff output

daxian-dbw commented Oct 3, 2022

CarloToso commented Oct 4, 2022 •

edited

Loading

God-damnit-all commented Dec 23, 2022

iSazonov commented Dec 24, 2022

God-damnit-all commented Dec 25, 2022

	# If Content is compressed we cannot read it as JSON.
	# If Content is still compressed we cannot read it as JSON.

Fix decompression in web cmdlets #17955

Fix decompression in web cmdlets #17955

Conversation

iSazonov commented Aug 24, 2022 • edited Loading

PR Summary

PR Context

PR Checklist

PaulHigin Aug 24, 2022

Choose a reason for hiding this comment

SteveL-MSFT Aug 29, 2022

Choose a reason for hiding this comment

iSazonov Aug 30, 2022

Choose a reason for hiding this comment

daxian-dbw Sep 26, 2022

Choose a reason for hiding this comment

iSazonov Sep 27, 2022

Choose a reason for hiding this comment

PaulHigin commented Aug 24, 2022 • edited Loading

iSazonov commented Aug 24, 2022 • edited Loading

ghost commented Sep 6, 2022

ghost commented Sep 14, 2022

daxian-dbw Sep 26, 2022

Choose a reason for hiding this comment

daxian-dbw left a comment

Choose a reason for hiding this comment

pull-request-quantifier-deprecated bot commented Sep 28, 2022

What can I do to optimize my changes

How to interpret the change counts in git diff output

daxian-dbw commented Oct 3, 2022

CarloToso commented Oct 4, 2022 • edited Loading

God-damnit-all commented Dec 23, 2022

iSazonov commented Dec 24, 2022

God-damnit-all commented Dec 25, 2022

iSazonov commented Aug 24, 2022 •

edited

Loading

PaulHigin commented Aug 24, 2022 •

edited

Loading

iSazonov commented Aug 24, 2022 •

edited

Loading

CarloToso commented Oct 4, 2022 •

edited

Loading