Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: multi-index df fillna with value not working after 1.4.3 #47649

Closed
3 tasks done
Yikun opened this issue Jul 9, 2022 · 4 comments · Fixed by #47774
Closed
3 tasks done

BUG: multi-index df fillna with value not working after 1.4.3 #47649

Yikun opened this issue Jul 9, 2022 · 4 comments · Fixed by #47774
Labels
Bug Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate MultiIndex Regression Functionality that used to work in a prior pandas version
Milestone
1.4.4

Comments

@Yikun
Copy link
Contributor

Yikun commented Jul 9, 2022 •
edited
Loading

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np
pdf = pd.DataFrame(
    {
        ("x", "a"): [np.nan, 2, 3, 4, np.nan, 6],
        ("x", "b"): [1, 2, np.nan, 4, np.nan, np.nan],
        ("y", "c"): [1, 2, 3, 4, np.nan, np.nan],
    },
    index=np.random.rand(6),
)

>>> pdf.fillna({"x": -1})
           x        y
           a   b    c
0.563210 NaN NaN  1.0
0.612561 NaN NaN  2.0
0.141892 NaN NaN  3.0
0.851834 NaN NaN  4.0
0.702158 NaN NaN  NaN
0.142645 NaN NaN  NaN
>>> pdf.fillna({"x": -1, ("x", "b"): -2})
           x         y
           a    b    c
0.563210 NaN -2.0  1.0
0.612561 NaN -2.0  2.0
0.141892 NaN -2.0  3.0
0.851834 NaN -2.0  4.0
0.702158 NaN -2.0  NaN
0.142645 NaN -2.0  NaN
>>> pdf.fillna({("x", "b"): -2, "x": -1})
           x        y
           a   b    c
0.563210 NaN NaN  1.0
0.612561 NaN NaN  2.0
0.141892 NaN NaN  3.0
0.851834 NaN NaN  4.0
0.702158 NaN NaN  NaN
0.142645 NaN NaN  NaN

Issue Description

multi-index df fillna with value not working after 1.4.3

Expected Behavior

# With pandas 1.4.2:
>>> pdf.fillna({"x": -1})
            x         y
            a    b    c
0.572182 -1.0  1.0  1.0
0.819731  2.0  2.0  2.0
0.202910  3.0 -1.0  3.0
0.371307  4.0  4.0  4.0
0.038513 -1.0 -1.0  NaN
0.025520  6.0 -1.0  NaN
>>> pdf.fillna({"x": -1, ("x", "b"): -2})
            x         y
            a    b    c
0.572182 -1.0  1.0  1.0
0.819731  2.0  2.0  2.0
0.202910  3.0 -1.0  3.0
0.371307  4.0  4.0  4.0
0.038513 -1.0 -1.0  NaN
0.025520  6.0 -1.0  NaN
>>> pdf.fillna({("x", "b"): -2, "x": -1})
            x         y
            a    b    c
0.572182 -1.0  1.0  1.0
0.819731  2.0  2.0  2.0
0.202910  3.0 -2.0  3.0
0.371307  4.0  4.0  4.0
0.038513 -1.0 -2.0  NaN
0.025520  6.0 -2.0  NaN

Installed Versions

pd.show_versions()
/Users/yikun/venv/lib/python3.9/site-packages/_distutils_hack/init.py:30: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")

INSTALLED VERSIONS

commit : e8093ba
python : 3.9.12.final.0
python-bits : 64
OS : Darwin
OS-release : 21.5.0
Version : Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:37 PDT 2022; root:xnu-8020.121.3~4/RELEASE_ARM64_T6000
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : zh_CN.UTF-8
LOCALE : zh_CN.UTF-8

pandas : 1.4.3
numpy : 1.22.4
pytz : 2021.3
dateutil : 2.8.2
setuptools : 60.5.0
pip : 22.0.4
Cython : 0.29.28
pytest : 7.1.0
hypothesis : None
sphinx : 3.0.4
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : 8.1.1
pandas_datareader: None
bs4 : 4.10.0
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
markupsafe : 2.0.1
matplotlib : 3.2.2
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 7.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.8.0
snappy : None
sqlalchemy : 1.4.32
tables : None
tabulate : 0.8.9
xarray : None
xlrd : None
xlwt : None
zstandard : None
@Yikun Yikun added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jul 9, 2022
@Yikun
Copy link
Contributor Author

Yikun commented Jul 9, 2022

First bad commit: 3364f9a #47327

cc @phofl

@phofl
Copy link
Member

phofl commented Jul 9, 2022

Yeah this is unfortunate, but directly related to the other issues referencing the loc problems when setting with a MultiIndex

@phofl phofl added Regression Functionality that used to work in a prior pandas version MultiIndex and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Jul 9, 2022
@phofl phofl added this to the 1.4.4 milestone Jul 9, 2022
@xr-chen
Copy link
Contributor

xr-chen commented Jul 13, 2022 •
edited
Loading 

import numpy as np
import pandas as pd

df = pd.DataFrame(
    {
        ("x", "a"): np.arange(6),
        ("x", "b"): np.arange(6),
        ("y", "c"): np.arange(6),
    },
)
>>>print(df)

   x     y
   a  b  c
0  0  0  0
1  1  1  1
2  2  2  2
3  3  3  3
4  4  4  4
5  5  5  5

df.loc[:, "x"] = df["x"]
>>>print(df)
    x      y
    a   b  c
0 NaN NaN  0
1 NaN NaN  1
2 NaN NaN  2
3 NaN NaN  3
4 NaN NaN  4
5 NaN NaN  5

The same thing happens when trying to set values with a multiIndex even the data frame doesn't contain any NAN

@xr-chen xr-chen mentioned this issue Jul 17, 2022
Merged
5 tasks
@simonjayhawkins simonjayhawkins added the Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate label Jul 20, 2022
@simonjayhawkins
Copy link
Member

simonjayhawkins commented Jul 20, 2022 •
edited
Loading

Yeah this is unfortunate, but directly related to the other issues referencing the loc problems when setting with a MultiIndex

not tried but it seems that there are some inconsistencies noted in #45751 (comment) that we might be able to use to our advantage to fix the regression?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate MultiIndex Regression Functionality that used to work in a prior pandas version
Projects
None yet
Milestone
1.4.4
Development

Successfully merging a pull request may close this issue.

BUG: Fix fillna on multi indexed Dataframe doesn't work xr-chen/pandas
4 participants
@Yikun @simonjayhawkins @xr-chen @phofl