Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error reports when loading model checkpoint #80

Open
JackeyLee007 opened this issue Jun 30, 2024 · 6 comments
Open

Error reports when loading model checkpoint #80

JackeyLee007 opened this issue Jun 30, 2024 · 6 comments

Comments

@JackeyLee007
Copy link

When the following code executes,

import timesfm
import os
import numpy as np


tfm = timesfm.TimesFm(
    context_len=64,
    horizon_len=32,
    input_patch_len=32,
    output_patch_len=128,
    num_layers=20,
    model_dims=1280,
    backend='cpu',
)
tfm.load_from_checkpoint(repo_id="google/timesfm-1.0-200m")

forecast_input = [
    np.sin(np.linspace(0, 20, 100)),
    np.sin(np.linspace(0, 20, 200)),
    np.sin(np.linspace(0, 20, 400)),
]
frequency_input = [0, 1, 2]

print('before predicate')
point_forecast, experimental_quantile_forecast = tfm.forecast(
    forecast_input,
    freq=frequency_input,
)

print('after predicate')
print(point_forecast)

It reports the following errors

Restoring checkpoint from /root/.cache/huggingface/hub/models--google--timesfm-1.0-200m/snapshots/8775f7531211ac864b739fe776b0b255c277e2be/checkpoints.
WARNING:absl:No registered CheckpointArgs found for handler type: <class 'paxml.checkpoints.FlaxCheckpointHandler'>
WARNING:absl:Configured `CheckpointManager` using deprecated legacy API. Please follow the instructions at https://orbax.readthedocs.io/en/latest/api_refactor.html to migrate by May 1st, 2024.
WARNING:absl:train_state_unpadded_shape_dtype_struct is not provided. We assume `train_state` is unpadded.
Killed
(timfm_env02) root@instance:~/opensrc# /root/miniconda3/envs/timfm_env02/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
@king571834
Copy link

I have the same problem.

@JackeyLee007
Copy link
Author

@siriuz42 Take a look at this issue. Would you please help us out?

@siriuz42
Copy link
Collaborator

siriuz42 commented Jul 1, 2024

My current guess is that this is a RAM issue - I've seen this issue once when I was loading the model on a 8G RAM machine, and resolved it by going up to 16G.

@king571834
Copy link

king571834 commented Jul 2, 2024 •
edited
Loading

我目前的猜測是這是一個 RAM 問題 - 當我在 8G RAM 機器上加載模型時，我曾經遇到過這個問題，並通過升級到 16G 來解決它。

But my RAM is 48G also has the same problem

@JackeyLee007
Copy link
Author

My current guess is that this is a RAM issue - I've seen this issue once when I was loading the model on a 8G RAM machine, and resolved it by going up to 16G.

@siriuz42 Thanks. It solved my problem. I raised my RAM to 64G, then it works well so far.

@siriuz42
Copy link
Collaborator

siriuz42 commented Jul 2, 2024

@king571834 Could you paste your error message / stack trace if there's any?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@JackeyLee007 @siriuz42 @king571834