Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ForEach-Object -Parallel Pipeline and other breakdowns #13588

Closed
bobbytreed opened this issue Sep 6, 2020 · 8 comments
Closed

ForEach-Object -Parallel Pipeline and other breakdowns #13588

bobbytreed opened this issue Sep 6, 2020 · 8 comments
Labels
Issue-Question ideally support can be provided via other mechanisms, but sometimes folks do open an issue to get a Resolution-No Activity Issue has had no activity for 6 months or more WG-Cmdlets-Core cmdlets in the Microsoft.PowerShell.Core module

Comments

@bobbytreed
Copy link

So, I know you are working on improving ForEach-Object -Parallel, but I wanted to offer what I have seen so far.

Within a ForEach-Object -Parallel, I load a script containing functions and class declarations. At some point within the processing, the pipeline breaks down first

something as simple as @() | Where-Object hangs indefinitely, while @().Where({}) executes fine. Further into the processing, even calling a static function of a defined PS class hangs indefinitely.

So far I've been working through the script trying to remove all Write-Progress calls (they intermittently causes hangs as well), Any Pipeline Where or ForEach calls, etc. But its odd now that even though breakpointing the script shows that the PS Class I created does exist, calling a static function hangs.

I know it would be impossible to provide enough context to help identify the issue, but I'm very confident that its not a breakdown in my code.

In addition to any insight you have, is there any way to see more about whats going on?

Thank you

I'm using the latest version of PS 7.
@bobbytreed bobbytreed added the Issue-Question ideally support can be provided via other mechanisms, but sometimes folks do open an issue to get a label Sep 6, 2020
@SeeminglyScience
Copy link
Collaborator

something as simple as @() | Where-Object hangs indefinitely, while @().Where({}) executes fine.

Can you post the exact snippet you're using? I can't reproduce it.

Further into the processing, even calling a static function of a defined PS class hangs indefinitely.

Yeah classes aren't thread safe and are likely to cause deadlocks or state corruption. See #12801

@iSazonov iSazonov added the WG-Cmdlets-Core cmdlets in the Microsoft.PowerShell.Core module label Sep 7, 2020
@bobbytreed
Copy link
Author

I'll try to post something in the next day or so that replicates my logic.

Classes not being thread safe should not matter.

Each parallel execution dot sources the script containing the class definition. The code is written such that every parallel execution should not interact with another.

@bobbytreed
Copy link
Author

Place in class.ps1

    class InnerClass {
        [string]$Value;
    }

    class OuterClass
    {
        [InnerClass[]]$Values;

        [void] TestWhere ([string]$filter) {
            Write-Host "TestWhere"
            $this.Values | Where-Object {
                Write-Host "`tTestWhere: {`$_.Value -match $filter}: $($_ -match $filter)"
                $_.Value -match $filter
            }
        }

        [void] TestForEach ([string]$filter) {
            Write-Host "TestForEach"
            $this.Values | ForEach-Object {
                Write-Host "`tTestForEach: $($_.Value)"
            }
        }

        [void] TestForEachMethod ([string]$filter) {
            Write-Host "TestForEachMethod"
            $this.Values.ForEach({
                Write-Host "`tTestForEachMethod: $($_.Value)"
            })
        }

        [string] TestStaticMethod([string]$filter) {
            Write-Host "TestStaticMethod"
            return [OuterClass]::ReturnString($filter)
        }

        static [string] ReturnString([string]$string) {
            Write-Host "`tTestStaticMethod: $string"
            return $string + " append"
        }
    }

    Function Test-Loop {
        [CmdletBinding()]
        param (
            [Parameter()]
            [OuterClass]
            $OuterClass,

            [Parameter()]
            [int]$Value,

            [Parameter(Mandatory)]
            [ValidateSet("Where", "ForEach", "Static")]
            [string]$Type
        )

        process {
            $OuterClass.TestForEachMethod($Value)
            switch ($type) {
                "Static" {
                    $output = $OuterClass.TestStaticMethod($value)
                }

                "Where" {
                    $OuterClass.TestWhere($Value)
                }

                "ForEach" {
                    $OuterClass.TestForEach($Value)
                }
            }
        }
    }

Place in testscript.ps1

param(
    [Parameter(Mandatory)]
    [ValidateSet("Where", "ForEach", "Static")]
    [string]$Type,

    [Parameter()]
    [string]$ClassScriptLocation = "C:\users\robreed\documents\powershell\temp\class.ps1"
)

. $ClassScriptLocation
$passScriptLocation = $ClassScriptLocation
$passType = $Type

$Class = [OuterClass]::new()

$class.Values = 1..5 | Foreach-object {
    $tmp = [InnerClass]::new()
    $tmp.Value = $_
    $tmp
}

1..2 | Foreach-object -Parallel {
    . $using:passScriptLocation
    Test-Loop -OuterClass $using:class -Value $_ -Type $using:passType
}

Call Testscript.ps1 -Type ForEach

Output:

TestForEachMethod
        TestForEachMethod: 1
        TestForEachMethod: 2
        TestForEachMethod: 3
        TestForEachMethod: 4
        TestForEachMethod: 5
TestForEachMethod
        TestForEachMethod: 1
        TestForEachMethod: 2
        TestForEachMethod: 3
        TestForEachMethod: 4
        TestForEachMethod: 5
TestForEach

Call Testscript.ps1 -Type Static

Output:

TestForEachMethod
        TestForEachMethod: 1
        TestForEachMethod: 2
        TestForEachMethod: 3
        TestForEachMethod: 4
        TestForEachMethod: 5
TestForEachMethod
        TestForEachMethod: 1
        TestForEachMethod: 2
        TestForEachMethod: 3
        TestForEachMethod: 4
        TestForEachMethod: 5
TestStatic

Call Testscript.ps1 -Type Where

Output:

TestForEachMethod
        TestForEachMethod: 1
        TestForEachMethod: 2
        TestForEachMethod: 3
        TestForEachMethod: 4
        TestForEachMethod: 5
TestForEachMethod
        TestForEachMethod: 1
        TestForEachMethod: 2
        TestForEachMethod: 3
        TestForEachMethod: 4
        TestForEachMethod: 5
TestWhere

Static, Where, and ForEach alll fail and foreachmethod works.

@bobbytreed
Copy link
Author

And by "fail" I mean they sit there forever without doing anything, no error or indication of error

@SeeminglyScience
Copy link
Collaborator

SeeminglyScience commented Sep 8, 2020 •
edited
Loading

Classes not being thread safe should not matter.

Each parallel execution dot sources the script containing the class definition. The code is written such that every parallel execution should not interact with another.

Yeah, it's still the same generated type though. It's cached based on the AST which will be the same in all runspaces. You can see it by running this:

$classFile = 'temp:\myClassFile.ps1'
try {
    $null = New-Item -ItemType File -Path $classFile -Value 'class Testing { }'
    $hashCodes = 0..100 | ForEach-Object -Parallel {
        . $using:classFile
        [Testing].GetHashCode()
    }

    $hashCodes | Group-Object
} finally {
    Remove-Item $classFile -ErrorAction Ignore
}

You should get something like this:

Count Name                      Group
----- ----                      -----
  101 56517414                  {56517414, 56517414, 56517414, 56517414…}

Though you might get a couple different on the first run if you hit a race condition, it should be mostly the same generated type.

@microsoft-github-policy-service Microsoft GitHub Policy Service
Copy link
Contributor

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

1 similar comment
@microsoft-github-policy-service Microsoft GitHub Policy Service
Copy link
Contributor

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

@microsoft-github-policy-service microsoft-github-policy-service bot added the Resolution-No Activity Issue has had no activity for 6 months or more label Nov 16, 2023
@microsoft-github-policy-service Microsoft GitHub Policy Service
Copy link
Contributor

This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Issue-Question ideally support can be provided via other mechanisms, but sometimes folks do open an issue to get a Resolution-No Activity Issue has had no activity for 6 months or more WG-Cmdlets-Core cmdlets in the Microsoft.PowerShell.Core module
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bobbytreed @iSazonov @SeeminglyScience