-
|
Hi all, Our PaaS provider recently started charging for network egress fees. I have noticed that Django Q2 seems to use quite a lot of network bandwidth even when "idle". In this instance, Q is running some very low-intensity daily tasks which send email reminders via a web API. Here's a graph of network activity over one week (each datapoint represents a one-hour period): That's accumulated to something like ~36GB so far during November. Our config looks like this: Q_CLUSTER = {
'name': 'project_name',
'label': 'Tasks',
'catch_up': False,
'orm': 'default',
'timeout': 2700,
'retry': 3600,
}Is this normal for something like Django Q? Given our very light usage, would we benefit from setting a much higher |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 4 replies
-
|
Do you think that's all from django-q2? Because that sounds very high. Django-q2 does poll the database quite often for new tasks. You can indeed change the |
Beta Was this translation helpful? Give feedback.
-
|
Thanks. I've upped the This particular process just runs I've just pushed another change which will reduce the |
Beta Was this translation helpful? Give feedback.
-
|
You could also try to set the guard_cycle if you want to reduce memory use/cpu use. It should also help with bandwidth usage as it saves the current state of the cluster every time it runs through a cycle (which is currently every 0.5 seconds). Default is 0.5 seconds and should be between 0 and 60 seconds. |
Beta Was this translation helpful? Give feedback.
-
|
Increasing the And our two daily scheduled tasks ran without issue this morning. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot for reporting back. I wasn't aware that django-q(2) is so heavy on bandwidth calls. 36GB is absolutely insane. The status saving on the guard cycle is in most cases excessive, especially with very few schedules/tasks. I will rewrite that to first check (compared to a local object) if we actually need to update the status. With that, bandwidth would likely get even lower for you as it would only need to update 2 times a day for your daily scheduled tasks, instead of every 20 seconds now. |
Beta Was this translation helpful? Give feedback.
Thanks a lot for reporting back. I wasn't aware that django-q(2) is so heavy on bandwidth calls. 36GB is absolutely insane.
The status saving on the guard cycle is in most cases excessive, especially with very few schedules/tasks. I will rewrite that to first check (compared to a local object) if we actually need to update the status. With that, bandwidth would likely get even lower for you as it would only need to update 2 times a day for your daily scheduled tasks, instead of every 20 seconds now.