Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inconsistent memory req/lim between single-userserver and user-placeholder #3539

Open
dev-dsp opened this issue Oct 3, 2024 · 5 comments
Open

Inconsistent memory req/lim between single-userserver and user-placeholder #3539

dev-dsp opened this issue Oct 3, 2024 · 5 comments
Labels
bug

Comments

@dev-dsp
Copy link

dev-dsp commented Oct 3, 2024 •
edited
Loading

Bug description

My configuration

singleuser:
  memory:
    limit: 1800M
    guarantee: 1800M
scheduling:
  userPlaceholder:
    enabled: true
    replicas: 4

Results in

$ kubectl describe pod jupyter-...
Containers:
  notebook:
    Limits:
      cpu:     2
      memory:  1887436800
    Requests:
      cpu:     500m
      memory:  1887436800


$ kubectl describe pod user-placeholder-...
Containers:
  pause:
    Limits:
      cpu:     2
      memory:  1800M
    Requests:
      cpu:        500m
      memory:     1800M

So 1887436800 bytes obviously is 1800Mi(B), which is 1887,4368 M(B), so not what I've requested.
1800M is a correct notation for MB, and should result in 1800000000 bytes (~1717 MiB)

For my HW this means that VM can fit 3 user-servers and 4 user-placeholders 🤷

Configuration

Z2JH 3.2.1
@dev-dsp dev-dsp added the bug label Oct 3, 2024
@dev-dsp
Copy link
Author

dev-dsp commented Oct 3, 2024

Actually it's even stranger now that I see the calculated value in user-server fields, while it would make more sense to find it in those of user-placeholder.
Somehow user-placeholder has correct user-server values from my values.yaml, while user-server has something calculated.

@dev-dsp
Copy link
Author

dev-dsp commented Oct 3, 2024

I've tried replacing requests/limits with 1717Mi, now I receive

    traitlets.traitlets.TraitError: 1717Mi is not a valid memory specification. Must be an int or a string with suffix K, M, G, T

And BackOff-ing hub 🤷

@consideRatio
Copy link
Member

Hehe sorry for the trouble with this, a workaround is to specify 123456... bytes directly.

The issue is that JupyterHub's k8s unaware config is used AND k8s native specification, and the translation cause issues. In the jupyterhub software, or traitlets config system software, the M means Mi, and in the k8s specification it means M and not Mi.

So, to avoid mismatch, use the lowest possible specification - the number of bytes as a plain integer.

@dev-dsp
Copy link
Author

dev-dsp commented Oct 3, 2024

Yep, just fixed with bytes. But I believe this should be fixed in chart somehow?..

@consideRatio
Copy link
Member

Yes! A fix is most welcome btw

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@consideRatio @dev-dsp