You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
(optional) I have confirmed this bug exists on the master branch of pandas.
Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.
Code Sample, a copy-pastable example
# Your code hereimportpandasaspd# Generate sample dfdf=pd.DataFrame({'column1': range(600), 'group': 5*['l'+str(i) foriinrange(120)]})
# sort by group for easy/efficient joining of new columns to dfdf=df.sort_values('group',kind='mergesort').reset_index(drop=True)
# timing of groupby rolling count, sum and mean%timeitdf['mean']=df.groupby('group').rolling(3,min_periods=1)['column1'].mean().values%timeitdf['sum']=df.groupby('group').rolling(3,min_periods=1)['column1'].sum().values%timeitdf['count']=df.groupby('group').rolling(3,min_periods=1)['column1'].count().values### Output6.14ms ± 812µsperloop (mean ± std. dev. of7runs, 100loopseach)
5.61ms ± 179µsperloop (mean ± std. dev. of7runs, 100loopseach)
76.1ms ± 4.78msperloop (mean ± std. dev. of7runs, 10loopseach)
Problem description
I am running a groupby rolling count, sum & mean using Pandas v1.1.0 and I notice that the rolling count is considerably slower than the rolling mean & sum. This seems counter intuitive as we can derive the count from the mean and sum and save time.
Expected Output
Expecting more efficient computation of groupby rolling count
I suspect @nrcjea001 was in fact looking for a size operator instead. I noticed it exists for groupby but not for RollingGroupBy. That might be a simple operation to add and could come in handy. The equivalent rollgrpby.apply(len) or rollgrpby.apply(lambda g: g.shape[0]) works but is 2x slower than rollgrpby.sum() for example. I presume it could be established very efficiently without creating groups, just looking at the indices differences (end - start) of the rolling windows.
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
(optional) I have confirmed this bug exists on the master branch of pandas.
Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.
Code Sample, a copy-pastable example
Problem description
I am running a groupby rolling count, sum & mean using Pandas v1.1.0 and I notice that the rolling count is considerably slower than the rolling mean & sum. This seems counter intuitive as we can derive the count from the mean and sum and save time.
Expected Output
Expecting more efficient computation of groupby rolling count
Output of
pd.show_versions()
commit : d9fff27
python : 3.8.5.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.18362
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 11, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252
pandas : 1.1.0
numpy : 1.18.5
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.1
setuptools : 49.2.1.post20200802
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.17.0
pandas_datareader: None
bs4 : 4.9.1
bottleneck : None
fsspec : 0.8.0
fastparquet : None
gcsfs : None
matplotlib : 3.3.0
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 1.0.0
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.0
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : 0.48.0
The text was updated successfully, but these errors were encountered: