Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: cython version of groupby.cummax throws error on datetimes #15561

Closed
adbull opened this issue Mar 3, 2017 · 4 comments · Fixed by #15569
Closed

BUG: cython version of groupby.cummax throws error on datetimes #15561

adbull opened this issue Mar 3, 2017 · 4 comments · Fixed by #15569
Labels
Bug Dtype Conversions Unexpected or buggy dtype conversions Groupby
Milestone
0.20.0

Comments

@adbull
Copy link
Contributor

adbull commented Mar 3, 2017 •
edited
Loading

Code Sample, a copy-pastable example if possible

>>> import pandas as pd
>>> x = pd.DataFrame(dict(a=[1], b=pd.to_datetime(['2001'])))
>>> x.groupby('a').b.cummax()

Traceback (most recent call last):
  File "<ipython-input-9-316257648d5f>", line 1, in <module>
    x.groupby('a').b.cummax()
  File "~/anaconda3/lib/python3.5/site-packages/pandas/core/groupby.py", line 1454, in cummax
    return self._cython_transform('cummax', **kwargs)
  File "~/anaconda3/lib/python3.5/site-packages/pandas/core/groupby.py", line 806, in _cython_transform
    raise DataError('No numeric types to aggregate')
DataError: No numeric types to aggregate

Problem description

The current github version of pandas has cython implementations of groupby.cummin and groupby.cummax, which throw an error if called on datetime columns. (See #15048, 0fe491d.)

Expected Output

0   2001-01-01
Name: b, dtype: datetime64[ns]

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.8-100.fc24.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: C
LANG: C
LOCALE: None.None

pandas: 0.19.0+531.g04e1168
pytest: 3.0.5
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.11.3
scipy: 0.18.1
xarray: 0.9.1
IPython: 4.2.0
sphinx: 1.5.1
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: 1.2.0
tables: 3.3.0
numexpr: 2.6.2
feather: None
matplotlib: 2.0.0
openpyxl: 2.4.1
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.7.2
bs4: 4.5.3
html5lib: 0.999
sqlalchemy: 1.1.5
pymysql: None
psycopg2: None
jinja2: 2.9.4
s3fs: None
pandas_gbq: None
pandas_datareader: None
@chris-b1
Copy link
Contributor

chris-b1 commented Mar 3, 2017

I think this may be as easy as adding a numeric_only=False to this function call. (of course with tests)

pandas/pandas/core/groupby.py

Line 1454 in 04e1168

return self._cython_transform('cummax', **kwargs)

@chris-b1 chris-b1 added this to the Next Major Release milestone Mar 3, 2017
@jreback
Copy link
Contributor

jreback commented Mar 3, 2017

This is correct and not a bug. datetimes are not numeric types. This has to be specifically enabled.
this was covered by #15054

In [29]: x = pd.DataFrame(dict(a=[1], b=pd.to_datetime(['2001'])))
    ...: >>> x.groupby('a').b.cummax(numeric_only=False)
    ...: 
    ...: 
Out[29]: 
0   2001-01-01
Name: b, dtype: datetime64[ns]

@jreback jreback closed this as completed Mar 3, 2017
@chris-b1
Copy link
Contributor

chris-b1 commented Mar 3, 2017

@jreback - maybe the API is set, but isn't this inconsistent?

In [23]: df = pd.DataFrame(dict(a=[1], b=pd.to_datetime(['2001'])))

In [24]: df['b'].max()
Out[24]: Timestamp('2001-01-01 00:00:00')

In [25]: df['b'].cummax()
Out[25]: 
0   2001-01-01
Name: b, dtype: datetime64[ns]

In [26]: df.groupby('a')['b'].max()
Out[26]: 
a
1   2001-01-01
Name: b, dtype: datetime64[ns]

In [27]: df.groupby('a')['b'].cummax()
Out[27]: 
DataError: No numeric types to aggregate

@jreback
Copy link
Contributor

jreback commented Mar 3, 2017

@chris-b1 hmm, ahh we are defaulting numeric_only=False for .max

yes that should be done for cummax/cummin.

@jreback jreback reopened this Mar 3, 2017
@jreback jreback added Difficulty Novice Dtype Conversions Unexpected or buggy dtype conversions labels Mar 3, 2017
mroeschke added a commit to mroeschke/pandas that referenced this issue Mar 4, 2017
@mroeschke
BUG: Groupby.cummin/max DataError on datetimes (pandas-dev#15561)
5e5d70d
@mroeschke mroeschke mentioned this issue Mar 4, 2017
Merged
3 tasks
@jreback jreback modified the milestones: 0.20.0, Next Major Release Mar 4, 2017
TomAugspurger pushed a commit that referenced this issue Mar 5, 2017
@mroeschke @TomAugspurger
BUG: Groupby.cummin/max DataError on datetimes (#15561) (#15569)
f5b7bcb
AnkurDedania pushed a commit to AnkurDedania/pandas that referenced this issue Mar 21, 2017
@mroeschke @AnkurDedania
BUG: Groupby.cummin/max DataError on datetimes (pandas-dev#15561) (pa…
ad24d7f 
…ndas-dev#15569)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Dtype Conversions Unexpected or buggy dtype conversions Groupby
Projects
None yet
Milestone
0.20.0
Development

Successfully merging a pull request may close this issue.

BUG: Groupby.cummin/max DataError on datetimes (#15561) mroeschke/pandas
3 participants
@jreback @chris-b1 @adbull