Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unity Mousewheel crash. How do I implement scroll pages in HoloLens 2 Unity apps? #3705

Open
PurpleJoy opened this issue Aug 14, 2023 · 10 comments
Assignees
Labels
bug Something isn't working tracked We are tracking this work internally.

Comments

@PurpleJoy
Copy link

PurpleJoy commented Aug 14, 2023

I went through the tutorial and downloaded the examples.

Following the example, I implemented OnPointerDown() and OnPointerUp().

But found that can only achieve click operations and can not scroll the web page.

I want to make it easy for users to scroll and navigate the web using gestures. How do I do that?

(I tried to implement MouseWheel operation in OnPointerDragged(), but couldn't find any instructions or examples, and every time I tried to send a MouseEvent myself, Unity crashed.)

Here is part of my code :

bool isMouseDown = false;
Vector3 startPos;

public void OnPointerDown(MixedRealityPointerEventData eventData)
{
    TranslateToWebViewMouseEvent(eventData, WebViewMouseEventData.EventType.MouseDown);
    UnityEngine.Ray leftHandRay;
    bool isGetSuccess = InputRayUtils.TryGetHandRay(Microsoft.MixedReality.Toolkit.Utilities.Handedness.Right, out leftHandRay);
    if (isGetSuccess)
    {
        startPos = leftHandRay.origin;
    }
    isMouseDown = true;
}

public void OnPointerUp(MixedRealityPointerEventData eventData)
{
    TranslateToWebViewMouseEvent(eventData, WebViewMouseEventData.EventType.MouseUp);
    isMouseDown = false;
}

private void TranslateToWebViewMouseEvent(MixedRealityPointerEventData eventData, WebViewMouseEventData.EventType evenType)
{
    var hitCoord = NormalizeWorldPoint(eventData.Pointer.Result.Details.Point);

    hitCoord.x *= _webView.Width;
    hitCoord.y *= _webView.Height;

    var mouseEventsWebView = _webView as IWithMouseEvents;
    WebViewMouseEventData mouseEvent = new WebViewMouseEventData
    {
        X = (int)hitCoord.x,
        Y = (int)hitCoord.y,
        Device = WebViewMouseEventData.DeviceType.Pointer,
        Type = evenType,
        Button = WebViewMouseEventData.MouseButton.ButtonLeft,
        TertiaryAxisDeviceType = WebViewMouseEventData.TertiaryAxisDevice.PointingDevice
    };

    mouseEventsWebView.MouseEvent(mouseEvent);
}

private Vector2 NormalizeWorldPoint(Vector3 worldPoint)
{
    // Convert the world point to our control's local space.
    Vector3 localPoint = transform.InverseTransformPoint(worldPoint);

    var boundsSize = Collider.sharedMesh.bounds.size;
    var boundsExtents = Collider.sharedMesh.bounds.max;

    // Adjust the point to be based on a 0,0 origin.
    var uvTouchPoint = new Vector2((localPoint.x + boundsExtents.x), -1.0f * (localPoint.y - boundsExtents.y));

    // Normalize the point so it can be mapped to the WebView's texture.
    var normalizedPoint = new Vector2(uvTouchPoint.x / boundsSize.x, uvTouchPoint.y / boundsSize.y);

    return normalizedPoint;
}

public void OnPointerDragged(MixedRealityPointerEventData eventData) 
{
    if (isMouseDown)
    {
        UnityEngine.Ray leftHandRay;
        bool isGetSuccess = InputRayUtils.TryGetHandRay(Microsoft.MixedReality.Toolkit.Utilities.Handedness.Right, out leftHandRay);
        if (isGetSuccess)
        {
            var distance = Vector3.Distance(startPos, leftHandRay.origin);
            if (distance > 0.01)
            {
                var mouseEventsWebView = _webView as IWithMouseEvents;
                WebViewMouseEventData mouseEvent = new WebViewMouseEventData
                {
                    WheelX = 0,
                    WheelY = distance,
                    Device = WebViewMouseEventData.DeviceType.Pointer,
                    Type = WebViewMouseEventData.EventType.MouseWheel,
                    Button = WebViewMouseEventData.MouseButton.ButtonMiddle,
                    TertiaryAxisDeviceType = WebViewMouseEventData.TertiaryAxisDevice.PointingDevice
                };

                mouseEventsWebView.MouseEvent(mouseEvent);
            }
        }
    }
}

AB#46192317

@champnic
Copy link
Member

@michaelfarnsworth could you please take a look?

@RubenGarcia
Copy link

RubenGarcia commented Aug 17, 2023

I'm using Webview2 0.17.1.pre3 and I don't have Device in

        Device = WebViewMouseEventData.DeviceType.Pointer,

What version are you using?
I don't see Device at
https://learn.microsoft.com/en-us/windows/mixed-reality/develop/advanced-concepts/webview2-plugin-mouse-event-data-class

@michaelfarnsworth
Copy link
Collaborator

@RubenGarcia that's odd. There should be no published version without that enum defined. It's under the Microsoft.MixedReality.WebView namespace. Are you just trying to build the GettingStarted sample app?

@PurpleJoy
Copy link
Author

PurpleJoy commented Aug 18, 2023

@RubenGarcia I'm using version 0.17.1-Pre.5. This is the example I mentioned earlier: https://github.com/MicrosoftEdge/WebView2Samples/tree/main/GettingStartedGuides/HoloLens2_GettingStarted. In this example, WebViewBrowser.cs uses Device.

@RubenGarcia
Copy link

I had issues with Pre.5; I'll test it again and report.

@RubenGarcia
Copy link

I confirm that removing the Device line works on Pre.3, and clicks are correctly received and processed by the webpage.

@RubenGarcia
Copy link

I confirm that using your code to create MouseWheel events crashes Unity, both in OnPointerDragged and on Update().
However, using a MouseWheel event mapped to a keypress seems to work.
I suspect that calling a MouseWheel event too often (before the previous one finishes executing) is causing the issue.

@PurpleJoy
Copy link
Author

I confirm that using your code to create MouseWheel events crashes Unity, both in OnPointerDragged and on Update(). However, using a MouseWheel event mapped to a keypress seems to work. I suspect that calling a MouseWheel event too often (before the previous one finishes executing) is causing the issue.

I tried to create MouseWheel events in OnPointerDown(), Unity still crash. How did you make it work?
Here is my code:

public void OnPointerDown(MixedRealityPointerEventData eventData)
{
    var mouseEventsWebView = _webView as IWithMouseEvents;
    WebViewMouseEventData mouseEvent = new WebViewMouseEventData
    {
        WheelX = 0,
        WheelY = -0.5f,
        Device = WebViewMouseEventData.DeviceType.Pointer,
        Type = WebViewMouseEventData.EventType.MouseWheel,
        Button = WebViewMouseEventData.MouseButton.ButtonMiddle,
        TertiaryAxisDeviceType = WebViewMouseEventData.TertiaryAxisDevice.PointingDevice
    };
    mouseEventsWebView.MouseEvent(mouseEvent);
}

@champnic champnic changed the title How do I implement scroll pages in HoloLens 2 Unity apps? Unity Mousewheel crash. How do I implement scroll pages in HoloLens 2 Unity apps? Aug 25, 2023
@champnic champnic added bug Something isn't working tracked We are tracking this work internally. and removed question labels Aug 25, 2023
@champnic
Copy link
Member

I've converted this to a bug for the crash and added it to our backlog for the Hololens team to investigate.

@ArDevKarl
Copy link

Any insights about this issue? I'm running into the same problem

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working tracked We are tracking this work internally.
Projects
None yet
Development

No branches or pull requests

5 participants