Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Language worker "User" logs are ignored after execution is complete #8222

Open
ejizba opened this issue Mar 10, 2022 · 5 comments
Open

Language worker "User" logs are ignored after execution is complete #8222

ejizba opened this issue Mar 10, 2022 · 5 comments

Comments

@ejizba
Copy link
Contributor

ejizba commented Mar 10, 2022

Back when @yojagad added a LogCategory for language workers to pass either System or User in this PR to the node worker, @pragnagopa said

With in the execution context, we should not be logging system logs

However, some of the logs in question are like this:

Error: 'done' has already been called. Please check your script for extraneous calls to 'done'.

Which actually happens after execution completes. As of today, those logs are silently ignored by the host here (nothing in app insights, nothing in the console, etc.). Which leads me to a few questions:

  • What exactly is the difference between System and User logs? (We might want to document in the protobuf definition)
  • How should language workers categorize logs that are related to an execution, but after an execution?
  • Should the host really be ignoring these logs? These logs could be coming directly from the user, and perhaps they don't realize they log after execution completes because they never see the logs.

cc @alrod

@pragnagopa
Copy link
Member

What exactly is the difference between System and User logs? (We might want to document in the protobuf definition)

User logs - Cx have access to these logs via App Insights - are super set for Function Execution logs + Host / Extension logs - Any logs coming from Host are considered as system logs and can be logged to Kusto. User logs cannot be logged to Kusto as they are considered PII

How should language workers categorize logs that are related to an execution, but after an execution?

Logs between invocation request and invocation complete should be considered user logs- logs after execution completes are orphaned logs
https://github.com/Azure/azure-functions-language-worker-protobuf/blob/6635670b136d565f2a196ce1a64fee1ea5c9c1c9/src/proto/FunctionRpc.proto#L59-L62

Should the host really be ignoring these logs? These logs could be coming directly from the user, and perhaps they don't realize they log after execution completes because they never see the logs.

Host should not log from orphaned executions - but you bring up a good point specifically for errors - as of now we do not have a good solution for this.
Tagging @fabiocav @liliankasem for design discussion on how to address this.

@ejizba
Copy link
Contributor Author

ejizba commented Mar 10, 2022

Logs between invocation request and invocation complete should be considered user logs

If the language worker is crafting the message itself, can't we consider it system logs regardless of when it happens? There's no risk of PII as long as it's a simple hard-coded string, which is why I assume it's okay for the host to write system logs during this time. Having those worker logs in our telemetry would be very helpful for supportability of language workers.

@liliankasem
Copy link
Member

liliankasem commented Mar 11, 2022

ich actually happens after execution completes

If the error is coming from the worker itself, I would consider it a system log, especially if it doesn't include the user's actual function logs (which is the PII part). i.e. like Eric suggested, the worker is seeing some sort of error and is crafting a log for it

@fabiocav
Copy link
Member

This item requires design and we need to gather requirements for worker logs not associated with an invocation.

Adding to the backlog for the design work.

@fabiocav
Copy link
Member

@AnatoliB it would be good to sync on this to make sure we're not duplicating anything you're already planning for this area.

@fabiocav fabiocav removed this from the Triaged milestone Jun 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants