Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: DataFrame.select_dtypes returns reference to original DataFrame #48090

Closed
2 of 3 tasks
villebro opened this issue Aug 15, 2022 · 4 comments · Fixed by #48176
Closed
2 of 3 tasks

BUG: DataFrame.select_dtypes returns reference to original DataFrame #48090

villebro opened this issue Aug 15, 2022 · 4 comments · Fixed by #48176
Labels
Regression Functionality that used to work in a prior pandas version
Milestone
1.4.4

Comments

@villebro
Copy link

villebro commented Aug 15, 2022 •
edited
Loading

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

from decimal import Decimal
import pandas as pd


df = pd.DataFrame({"a": [1, 2, None, 4], "b": [1.5, None, 2.5, 3.5]})
selected_df = df.select_dtypes(include=["number", Decimal])
selected_df.fillna(0, inplace=True)

df.to_markdown()

Issue Description

On Pandas 1.3.4 and previously, mutating the DataFrame returned by select_dtypes didn't change the original DataFrame. However, on 1.4.3, it appears mutating the selected DataFrame changes the original one. I suspect this is the PR that has caused the change in behavior: #42611

On 1.4.3, the original DataFrame is mutated:

a b
0 1 1.5
1 2 0
2 0 2.5
3 4 3.5

Expected Behavior

on 1.3.4, the original DataFrame is unchanged:

a b
0 1 1.5
1 2 nan
2 nan 2.5
3 4 3.5

Installed Versions

INSTALLED VERSIONS ------------------ commit : e8093ba python : 3.8.13.final.0 python-bits : 64 OS : Darwin OS-release : 21.6.0 Version : Darwin Kernel Version 21.6.0: Sat Jun 18 17:07:22 PDT 2022; root:xnu-8020.140.41~1/RELEASE_ARM64_T6000 machine : arm64 processor : arm byteorder : little LC_ALL : None LANG : None LOCALE : None.UTF-8

pandas : 1.4.3
numpy : 1.23.2
pytz : 2022.2.1
dateutil : 2.8.2
setuptools : 63.2.0
pip : 22.2.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.4.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
markupsafe : 2.1.1
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : 0.8.10
xarray : None
xlrd : None
xlwt : None
zstandard : None
@villebro villebro added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Aug 15, 2022
@villebro villebro mentioned this issue Aug 15, 2022
Merged
9 tasks
@jorisvandenbossche jorisvandenbossche added Regression Functionality that used to work in a prior pandas version and removed Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Aug 18, 2022
@jorisvandenbossche jorisvandenbossche added this to the 1.4.4 milestone Aug 18, 2022
@jorisvandenbossche jorisvandenbossche mentioned this issue Aug 20, 2022
Merged
5 tasks
@jorisvandenbossche
Copy link
Member

@villebro thanks for the report! That's indeed an unintended change in behaviour in 1.4 (and #42611 seems indeed the likely cause). I opened a PR to fix this: #48176

@villebro
Copy link
Author

@jorisvandenbossche thanks for looking into this issue and fixing it! FWIW, PR reviewed and approved 👍

simonjayhawkins added a commit to simonjayhawkins/pandas that referenced this issue Aug 23, 2022
@simonjayhawkins
code sample for pandas-dev#48090
da88b15
@simonjayhawkins
Copy link
Member

That's indeed an unintended change in behaviour in 1.4 (and #42611 seems indeed the likely cause)

can confirm, from bisect, first bad commit: [37ea2cd] PERF: select_dtypes (#42611)

@EugeneTorap EugeneTorap mentioned this issue Sep 1, 2022
Merged
9 tasks
@pickfire
Copy link

pickfire commented Apr 3, 2023 •
edited
Loading

I noticed this broke our code in 1.5 since we are depending on the select_dtypes to return a reference to the original data frame in 1.4 as a feature, so that we can do .select_dtypes(exclude=['int8']).fillna('', inplace=True), I tried finding this but can't seemed to find it in the list of API breaking change.

https://pandas.pydata.org/pandas-docs/stable/whatsnew/v1.5.0.html#backwards-incompatible-api-changes

Can we also put there?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Regression Functionality that used to work in a prior pandas version
Projects
None yet
Milestone
1.4.4
Development

Successfully merging a pull request may close this issue.

REGR: ensure DataFrame.select_dtypes() returns a copy jorisvandenbossche/pandas
4 participants
@jorisvandenbossche @pickfire @simonjayhawkins @villebro