Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task ressources is collected from machine ressources #520

Open
ghostvfx-jimmy opened this issue Jul 30, 2021 · 8 comments
Open

Task ressources is collected from machine ressources #520

ghostvfx-jimmy opened this issue Jul 30, 2021 · 8 comments

Comments

@ghostvfx-jimmy
Copy link
Contributor

ghostvfx-jimmy commented Jul 30, 2021

Task ressources are collected from machine ressources and not actual task ressources. When running multiple tasks at the same time it gives an incorrect picture.

I would suggest that eg. 'res_mem_peak_mb' is renamed 'host_res_mem_peak_mb' to start with before a proper solution to collect task ressources is in place.

@timurhai
Copy link
Member

timurhai commented Aug 2, 2021

Hi!
May be it is not enough correct, but was designed suggesting that resources are important on a "heavy" tasks, and such "heavy" tasks are usually running single on the host, just one task at the same host at the same time.
And if somebody needs resources per task, parser::processResources() function can be overloaded.

Yes, we can rename res_mem_peak_mb in host_res_mem_peak_mb.
Also res_cpu_agv can be renamed in host_res_cpu_agv.
May be someone will use both and res_mem_peak_mb and host_res_mem_peak_mb.

@lithorus
Copy link
Member

lithorus commented Aug 2, 2021

It makes total sense and it has it uses. But IMO it would be best to have both host and task ressources.

We're running several tasks per host quite alot. Mostly for nuke and houdini. When running multiple tasks it's essential to now how much the use per task to not overload the ressources on the host.

@lithorus
Copy link
Member

lithorus commented Aug 2, 2021

I see that it's only in the python part. Will create a merge request with the changes..

@lithorus
Copy link
Member

lithorus commented Aug 2, 2021

Or would it be better to collect the process stats in C? I've previously created a process stats collector, but was depending on a seperate python library (psutil).

@timurhai
Copy link
Member

timurhai commented Aug 2, 2021

I think that better to do it in C. But it can be much harder than in Python. So I do not know, not a simple question.

@lithorus
Copy link
Member

lithorus commented Aug 2, 2021

So rename it now, so it doesn't break too much and make ready for task resources?

@lithorus
Copy link
Member

lithorus commented Aug 2, 2021

btw. psutil should work in all OS'es.

@timurhai
Copy link
Member

timurhai commented Aug 2, 2021

Yes, multi-os is very good.
Yes, we can rename it right now. It is a new feature, and i did not think that someone already wrote much in-house scripts depending on that names.
Just AfWatch tasks window can recognize that names just to display some more human readable labels - i can fix it quickly.

May be better put import psutil in some try block, to make afrender works w/o this module as before.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants