Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DEPR: groupby with as_index=False doesn't add grouper as column when passing a Series as group key #49519

Closed
jorisvandenbossche opened this issue Nov 4, 2022 · 4 comments · Fixed by #52333
Closed

DEPR: groupby with as_index=False doesn't add grouper as column when passing a Series as group key #49519

jorisvandenbossche opened this issue Nov 4, 2022 · 4 comments · Fixed by #52333
Assignees
@rhshadrach
Labels
API - Consistency Internal Consistency of API/Behavior Deprecate Functionality to remove in pandas Groupby

Comments

@jorisvandenbossche
Copy link
Member

I don't know if this is strictly speaking a but (the documentation doesn't really specify it), but I would have expected a different result:

In [28]: df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})

In [29]: key = pd.Series(['foo', 'foo', 'bar'], name="key")

In [30]: df.groupby(key).sum()
Out[30]: 
     a  b
key      
bar  3  6
foo  3  9

In [31]: df.groupby(key, as_index=False).sum()
Out[31]: 
   a  b
0  3  6
1  3  9

(running with current main, 2.0.0.dev0+532.gdec9be2a5c)

When your key is a column in the DataFrame, specifying as_index=False will move the group key values to a column in the resulting dataframe:

In [34]: df["key"] = key

In [35]: df.groupby("key", as_index=False).sum()
Out[35]: 
   key  a  b
0  bar  3  6
1  foo  3  9

I would have expected the same behaviour if you pass the key as a Series.

cc @rhshadrach

This is related to #49450 (comment), where I encountered this behaviour for the case where the Series is actually a column of the original DataFrame (when changing this Series to be a view, and not an identical object).
@jorisvandenbossche
Copy link
Member Author

jorisvandenbossche commented Nov 4, 2022 •
edited
Loading

Hmm, this seems to be a duplicate of #40591, which is mentioning the user guide (I only looked in the reference docs, where this is indeed no mentioned)

@rhshadrach
Copy link
Member

rhshadrach commented Nov 4, 2022 •
edited
Loading

Assuming that the implementation/performance isn't a barrier here (I can't imagine why it would be), +1 on always including the groupers when as_index=False regardless if they are "in-axis" or not. Also, while #40591 may have preceded this, I would recommend closing that is as a duplicate of this as this one is more concise.

@rhshadrach rhshadrach added the API - Consistency Internal Consistency of API/Behavior label Nov 4, 2022
This was referenced Nov 5, 2022
Open
Closed
@rhshadrach rhshadrach mentioned this issue Dec 23, 2022
Closed
@rhshadrach
Copy link
Member

I've confirmed the implementation is rather straightforward here, but the deprecation less so. It is easy to reliably emit a warning the behavior will change when more groupers will be included in the future, however there is no way for users to adopt future behavior early, preserve old behavior when the change is made, nor silence the warning.

I think we also can't call this purely a bug fix, it does appear in the documentation:

Passing as_index=False will return the groups that you are aggregating over, if they are named columns.

I wonder if this is a case to make a breaking change without deprecation. A few opinionated thoughts:

  • It's hard to imagine preferring only certain types of groupers be included in the result and not others
  • My experience is that most groupers are in-axis

@rhshadrach
Copy link
Member

Actually, we can instruct users to use in-axis groupers in a deprecation warning. I'm in favor of deprecating this properly.

@rhshadrach rhshadrach self-assigned this Jan 5, 2023
@rhshadrach rhshadrach mentioned this issue Jan 11, 2023
Open
@rhshadrach rhshadrach mentioned this issue Feb 16, 2023
Merged
5 tasks
@rhshadrach rhshadrach added the Deprecate Functionality to remove in pandas label Mar 31, 2023
@rhshadrach rhshadrach changed the title BUG?: groupby with as_index=False doesn't add grouper as column when passing a Series as group key DEPR: groupby with as_index=False doesn't add grouper as column when passing a Series as group key Mar 31, 2023
@rhshadrach rhshadrach mentioned this issue Mar 31, 2023
Merged
5 tasks
@rhshadrach rhshadrach mentioned this issue Jun 19, 2023
Closed
3 tasks
@rhshadrach rhshadrach mentioned this issue Jan 26, 2024
Open
3 tasks
@rhshadrach rhshadrach mentioned this issue Mar 6, 2024
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rhshadrach rhshadrach

Labels
API - Consistency Internal Consistency of API/Behavior Deprecate Functionality to remove in pandas Groupby
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

DEPR: groupby with as_index=False not including out-of-axis groupings rhshadrach/pandas
2 participants
@jorisvandenbossche @rhshadrach