Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[test] DragDrop tests are resulting in dotnet.exe process crash #8598

Open
Tracked by #8607
dreddy-work opened this issue Feb 9, 2023 · 5 comments
Open
Tracked by #8607

[test] DragDrop tests are resulting in dotnet.exe process crash #8598

dreddy-work opened this issue Feb 9, 2023 · 5 comments
Labels
test-bug Problem in test source code (most likely)
Milestone
Future

Comments

@dreddy-work
Copy link
Member

Test

PictureBox_SetData_DoDragDrop_RichTextBox_ReturnsExptected_Async and possibly others in this file.

Issue description

Although these crashes are not frequent, they happen often enough to impede the CI build process. As these crashes occur in dotnet.exe, our tests are unable to collect dumps or provide detailed logs or call stacks.

I manually setup lab machine and locally debugged to collect dumps. Below are the call stack:

 	System.Private.CoreLib.dll!System.ModuleHandle.ResolveMethodHandle(int methodToken, System.RuntimeTypeHandle[] typeInstantiationContext, System.RuntimeTypeHandle[] methodInstantiationContext)	Unknown
 	System.Private.CoreLib.dll!System.Reflection.CustomAttribute.FilterCustomAttributeRecord(System.Reflection.MetadataToken caCtorToken, System.Reflection.MetadataImport scope, System.Reflection.RuntimeModule decoratedModule, System.Reflection.MetadataToken decoratedToken, System.RuntimeType attributeFilterType, bool mustBeInheritable, ref System.RuntimeType.ListBuilder<object> derivedAttributes, out System.RuntimeType attributeType, out System.IRuntimeMethodInfo ctorWithParameters, out bool isVarArg)	Unknown
 	System.Private.CoreLib.dll!System.Reflection.CustomAttribute.AddCustomAttributes(ref System.RuntimeType.ListBuilder<object> attributes, System.Reflection.RuntimeModule decoratedModule, int decoratedMetadataToken, System.RuntimeType attributeFilterType, bool mustBeInheritable, System.RuntimeType.ListBuilder<object> derivedAttributes)	Unknown
 	System.Private.CoreLib.dll!System.Reflection.CustomAttribute.GetCustomAttributes(System.Reflection.RuntimeModule decoratedModule, int decoratedMetadataToken, int pcaCount, System.RuntimeType attributeFilterType)	Unknown
 	System.Private.CoreLib.dll!System.Reflection.CustomAttribute.GetCustomAttributes(System.Reflection.RuntimeMethodInfo method, System.RuntimeType caType, bool inherit)	Unknown
 	System.Private.CoreLib.dll!System.Attribute.GetCustomAttributes(System.Reflection.MemberInfo element, System.Type attributeType, bool inherit)	Unknown
 	System.Private.CoreLib.dll!System.Diagnostics.StackTrace.TryResolveStateMachineMethod(ref System.Reflection.MethodBase method, out System.Type declaringType)	Unknown
 	System.Private.CoreLib.dll!System.Diagnostics.StackTrace.ToString(System.Diagnostics.StackTrace.TraceFormat traceFormat, System.Text.StringBuilder sb)	Unknown
 	System.Private.CoreLib.dll!System.Diagnostics.StackTrace.ToString()	Unknown
 	System.Windows.Forms.Primitives.dll!System.DisposalTracking.Tracker.Tracker()	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.GdiCache.ScreenGraphicsScope.ScreenGraphicsScope(ref System.Windows.Forms.ScreenDcCache.ScreenDcScope scope)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.GdiCache.GetScreenDCGraphics()	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.ProfessionalColorTable.InitCommonColors(ref System.Collections.Generic.Dictionary<System.Windows.Forms.ProfessionalColorTable.KnownColors, System.Drawing.Color> rgbTable)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.ProfessionalColorTable.InitSystemColors(ref System.Collections.Generic.Dictionary<System.Windows.Forms.ProfessionalColorTable.KnownColors, System.Drawing.Color> rgbTable)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.ProfessionalColorTable.InitThemedColors(ref System.Collections.Generic.Dictionary<System.Windows.Forms.ProfessionalColorTable.KnownColors, System.Drawing.Color> rgbTable)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.ProfessionalColorTable.ColorTable.get()	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.ProfessionalColorTable.FromKnownColor(System.Windows.Forms.ProfessionalColorTable.KnownColors color)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.ProfessionalColorTable.ToolStripBorder.get()	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.ToolStripProfessionalRenderer.OnRenderToolStripBorder(System.Windows.Forms.ToolStripRenderEventArgs e)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.ToolStripRenderer.DrawToolStripBorder(System.Windows.Forms.ToolStripRenderEventArgs e)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.ToolStrip.OnPaint(System.Windows.Forms.PaintEventArgs e)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.Control.PaintWithErrorHandling(System.Windows.Forms.PaintEventArgs e, short layer)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.Control.WmPaint(ref System.Windows.Forms.Message m)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.Control.WndProc(ref System.Windows.Forms.Message m)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.ScrollableControl.WndProc(ref System.Windows.Forms.Message m)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.ToolStrip.WndProc(ref System.Windows.Forms.Message m)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.Control.ControlNativeWindow.OnMessage(ref System.Windows.Forms.Message m)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.Control.ControlNativeWindow.WndProc(ref System.Windows.Forms.Message m)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.NativeWindow.Callback(Windows.Win32.Foundation.HWND hWnd, Interop.User32.WM msg, Windows.Win32.Foundation.WPARAM wparam, Windows.Win32.Foundation.LPARAM lparam)	Unknown
 	[Native to Managed Transition]	
 	user32.dll!UserCallWinProcCheckWow()	Unknown
 	user32.dll!DispatchClientMessage()	Unknown
 	user32.dll!__fnDWORD�()	Unknown
 	ntdll.dll!KiUserCallbackDispatcherContinue�()	Unknown
 	win32u.dll!NtUserDispatchMessage�()	Unknown
 	user32.dll!DispatchMessageWorker()	Unknown
 	[Managed to Native Transition]	
 	System.Windows.Forms.dll!System.Windows.Forms.Application.ComponentManager.Interop.Mso.IMsoComponentManager.FPushMessageLoop(nuint dwComponentID, Interop.Mso.msoloop uReason, void* pvLoopData)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Interop.Mso.msoloop reason, System.Windows.Forms.ApplicationContext context)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Interop.Mso.msoloop reason, System.Windows.Forms.ApplicationContext context)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.Application.RunDialog(System.Windows.Forms.Form form)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.Form.ShowDialog(System.Windows.Forms.IWin32Window owner)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.Form.ShowDialog()	Unknown
 	System.Windows.Forms.UI.IntegrationTests.dll!System.Windows.Forms.UITests.ControlTestBase.RunFormWithoutControlAsync<System.Windows.Forms.UITests.DragDropTests.DragImageDropDescriptionForm>(System.Func<System.Windows.Forms.UITests.DragDropTests.DragImageDropDescriptionForm> createForm, System.Func<System.Windows.Forms.UITests.DragDropTests.DragImageDropDescriptionForm, System.Threading.Tasks.Task> testDriverAsync) Line 291	C#
>	System.Windows.Forms.UI.IntegrationTests.dll!System.Windows.Forms.UITests.DragDropTests.PictureBox_SetData_DoDragDrop_RichTextBox_ReturnsExptected_Async() Line 486	C#
 	[Native to Managed Transition]	
 	[Managed to Native Transition]	
 	xunit.execution.dotnet.dll!Xunit.Sdk.TestInvoker<System.__Canon>.CallTestMethod(object testClassInstance) Line 150	C#

Collected dump for reference is copied: here
IInstructions to get a lab machine to investigate: here
@dreddy-work dreddy-work added the untriaged The team needs to look at this issue in the next triage label Feb 9, 2023
@dreddy-work dreddy-work changed the title DragDrop tests are resulting in dotnet.exe process crash [test] DragDrop tests are resulting in dotnet.exe process crash Feb 9, 2023
@dreddy-work dreddy-work added the Priority:0 Work that we can't release without label Feb 9, 2023
dreddy-work added a commit that referenced this issue Feb 9, 2023
@dreddy-work
Skip or fix flaky tests
27d942c 
Related to:

#8602
#8601
#8598
@dreddy-work dreddy-work mentioned this issue Feb 9, 2023
Merged
dreddy-work added a commit that referenced this issue Feb 9, 2023
@dreddy-work
Skip or fix flaky tests blocking CI (#8603)
4d2a8ad 
* Skip or fix flaky tests

Related to:

#8602
#8601
#8598

* Skipping test related to #6610
@dreddy-work dreddy-work mentioned this issue Feb 9, 2023
Closed
54 tasks
@dreddy-work
Copy link
Member Author

We see few other crashes that are not associated with specific tests. call stack:

Fatal error. 0xC0000005
   at Windows.Win32.PInvoke.DeleteObject(Windows.Win32.Graphics.Gdi.HGDIOBJ)
   at Windows.Win32.PInvoke+CreateBrushScope.Dispose()
   at System.Windows.Forms.Control.PaintBackColor(System.Windows.Forms.PaintEventArgs, System.Drawing.Rectangle, System.Drawing.Color)
   at System.Windows.Forms.Control.PaintBackground(System.Windows.Forms.PaintEventArgs, System.Drawing.Rectangle, System.Drawing.Color, System.Drawing.Point)
   at System.Windows.Forms.Control.PaintBackground(System.Windows.Forms.PaintEventArgs, System.Drawing.Rectangle)
   at System.Windows.Forms.Control.OnPaintBackground(System.Windows.Forms.PaintEventArgs)
   at System.Windows.Forms.ScrollableControl.OnPaintBackground(System.Windows.Forms.PaintEventArgs)
   at System.Windows.Forms.Tests.ScrollableControlTests+SubScrollableControl.OnPaintBackground(System.Windows.Forms.PaintEventArgs)
   at System.Windows.Forms.Tests.ScrollableControlTests.ScrollableControl_OnPaintBackground_InvokeWithParent_CallsPaint(System.Windows.Forms.Control, Boolean, Boolean, Boolean, System.Drawing.Color, System.Drawing.Image, System.Windows.Forms.ImageLayout, Int32)
   at DynamicClass.InvokeStub_ScrollableControlTests.ScrollableControl_OnPaintBackground_InvokeWithParent_CallsPaint(System.Object, System.Object, IntPtr*)
   at System.Reflection.MethodInvoker.Invoke(System.Object, IntPtr*, System.Reflection.BindingFlags)
   at System.Reflection.RuntimeMethodInfo.InvokeWithManyArguments(System.Reflection.RuntimeMethodInfo, Int32, System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
   at System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
   at System.Reflection.MethodBase.Invoke(System.Object, System.Object[])
   at Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].CallTestMethod(System.Object)
   at Xunit.Sdk.UITestInvoker+<>c__DisplayClass2_0+<<RunAsync>b__2>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Xunit.Sdk.UITestInvoker+<>c__DisplayClass2_0+<<RunAsync>b__2>d, Xunit.StaFact, Version=1.0.0.0, Culture=neutral, PublicKeyToken=593f35978b459a4b]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Xunit.Sdk.UITestInvoker+<>c__DisplayClass2_0+<<RunAsync>b__2>d, Xunit.StaFact, Version=1.0.0.0, Culture=neutral, PublicKeyToken=593f35978b459a4b]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Xunit.Sdk.UITestInvoker+<>c__DisplayClass2_0+<<RunAsync>b__2>d, Xunit.StaFact, Version=1.0.0.0, Culture=neutral, PublicKeyToken=593f35978b459a4b]].MoveNext()
   at Xunit.Sdk.Utilities+SyncContextAwaiter+<>c.<OnCompleted>b__5_0(System.Object)
   at DynamicClass.InvokeStub_SendOrPostCallback.Invoke(System.Object, System.Object, IntPtr*)
   at System.Reflection.MethodInvoker.Invoke(System.Object, IntPtr*, System.Reflection.BindingFlags)
   at System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
   at System.Delegate.DynamicInvokeImpl(System.Object[])
   at System.Delegate.DynamicInvoke(System.Object[])
   at System.Windows.Forms.Control.InvokeMarshaledCallbackDo(ThreadMethodEntry)
   at System.Windows.Forms.Control.InvokeMarshaledCallbackHelper(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Windows.Forms.Control.InvokeMarshaledCallback(ThreadMethodEntry)
   at System.Windows.Forms.Control.InvokeMarshaledCallbacks()
   at System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
   at System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
   at System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
   at System.Windows.Forms.NativeWindow.Callback(Windows.Win32.Foundation.HWND, WM, Windows.Win32.Foundation.WPARAM, Windows.Win32.Foundation.LPARAM)
   at Windows.Win32.PInvoke.DispatchMessage(Windows.Win32.UI.WindowsAndMessaging.MSG*)
   at System.Windows.Forms.Application+ComponentManager.Interop.Mso.IMsoComponentManager.FPushMessageLoop(UIntPtr, msoloop, Void*)
   at System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(msoloop, System.Windows.Forms.ApplicationContext)
   at System.Windows.Forms.Application+ThreadContext.RunMessageLoop(msoloop, System.Windows.Forms.ApplicationContext)
   at Xunit.Sdk.WinFormsSynchronizationContextAdapter.PumpTill(System.Threading.SynchronizationContext, System.Threading.Tasks.Task)
   at Xunit.Sdk.ThreadRental+<>c__DisplayClass11_0.<CreateAsync>b__0()
   at System.Threading.Thread.StartCallback()

@dreddy-work dreddy-work mentioned this issue Feb 17, 2023
Merged
@merriemcgaw merriemcgaw removed the untriaged The team needs to look at this issue in the next triage label Feb 21, 2023
@merriemcgaw merriemcgaw added this to the 8.0 Preview2 milestone Feb 21, 2023
@willibrandon
Copy link
Contributor

Interesting!

@willibrandon
Copy link
Contributor

I'm taking a look to see if I can reproduce it. Please let me know if there is anything I can do to help.

@willibrandon
Copy link
Contributor

I've been running the UIIntegrationTests today using the approach described in Running and debugging crashed tests.

.\start-code.cmd
.\restore.cmd
pushd .\src\System.Windows.Forms\tests\IntegrationTests\UIIntegrationTests
dotnet test

I've done about 100 UIIntegration runs today with no failures or crashes on my machine.

Passed!  - Failed:     0, Passed:   175, Skipped:     8, Total:   183, Duration: 39 s - System.Windows.Forms.UI.IntegrationTests.dll (net8.0)

@dreddy-work
Copy link
Member Author

@willibrandon, it was hard to repro this locally. We have seen this on build agents multiple times and captured the state. We will keep this open and add call stack as we see(if) them. We may have to draw some conclusions form call stack or dumps.

@dreddy-work dreddy-work modified the milestones: 8.0 Preview7, 8.0 RC1 Jul 18, 2023
@LeafShi1 LeafShi1 modified the milestones: 8.0 RC1, 9.0 Preview1 Aug 15, 2023
@JeremyKuhne JeremyKuhne removed this from the 9.0 Preview1 milestone Aug 15, 2023
@JeremyKuhne JeremyKuhne added this to the .NET 9.0 milestone Aug 15, 2023
@elachlan elachlan added the test-bug Problem in test source code (most likely) label Nov 3, 2023
@Tanya-Solyanik Tanya-Solyanik added Priority:0 Work that we can't release without and removed Priority:0 Work that we can't release without labels Nov 16, 2023
@JeremyKuhne JeremyKuhne removed the Priority:0 Work that we can't release without label Jul 24, 2024
@JeremyKuhne JeremyKuhne modified the milestones: .NET 9.0, .NET 10.0, Future Jul 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
test-bug Problem in test source code (most likely)
Projects
None yet
Milestone
Future
Development

No branches or pull requests
7 participants
@elachlan @willibrandon @JeremyKuhne @Tanya-Solyanik @merriemcgaw @dreddy-work @LeafShi1