-
-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BUG: using NamedTuples with .loc works only sometimes #48124
Comments
Interestingly this seems to work fine until version 1.3.5. I wouldn't mind taking a better look into what changed between 1.3.5 and 1.4.0 to cause this behaviour if no one has any immediate ideas. |
Thank you very much for investigating this!
…On Thu, 18 Aug 2022, 23:16 James Moro, ***@***.***> wrote:
Interestingly this seems to work fine until version 1.3.5. I wouldn't mind
taking a better look into what changed between 1.3.5 and 1.4.0 if no one
has any immediate ideas.
—
Reply to this email directly, view it on GitHub
<#48124 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACSHBO5WMLXM4ODFDTW3UNTVZ2R3HANCNFSM56ZQLRLA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
@AKuederle df.loc[("A", ["a", "b"]), :] and df.loc[indexer_tuple(first="A", second=["a", "b"]), :] will start to diverge here: pandas/pandas/core/indexing.py Line 1291 in 30efa5f
This method explicitly checks if pandas/pandas/core/indexers/utils.py Lines 67 to 80 in 30efa5f
This causes the former to move to "nested tuple slicing", while the latter moves to "straight lookup". However, straight lookup fails, as So, my question is, why are NamedTuples not allowed to be used in anything other than straight lookups? Perhaps it's simply an oversight, or perhaps there's some reason that I can't think of. |
In version v1.3.5, the following line still exists just before the aforementioned pandas/pandas/core/indexing.py Line 1138 in 66e3805
This line would convert Indexer(first='A', second=['a', 'b']) to ('A', ['a', 'b']) . From that point onwards, both test cases I mentioned in my last comment will be the same, and both work correctly.
This method was deprecated in #42476, and this line was simply removed, which caused the bug. |
A similar bug exists with import pandas as pd
import numpy as np
# A df with two level MultiIndex
df = pd.DataFrame(np.arange(120).reshape(6, -1), index=pd.MultiIndex.from_product([["A", "B"], ["a", "b", "c"]], names=["first", "second"]))
# Indexing with a normal tuple works as expected:
# normal subset
print(df.iloc[1, 2]) # <- Works
# complicated subset
print(df.iloc[(1, [2, 3])]) # <- Works
# Now the same with a named tuple
from collections import namedtuple
indexer_tuple = namedtuple("Indexer", df.index.names)
print(df.iloc[tuple(indexer_tuple(first=1, second=[2, 3]))]) # <- Works
print(df.iloc[indexer_tuple(first=1, second=[2, 3])]) # <- DOES NOT WORK!
This again originates from a There might be more cases where |
Converting is not an option, NamedTuples are valid Index Elements. Hence this could be a label based lookup. is the iloc case also a regression? |
You're right, initial conversion would not work then. I believe the |
Let’s open a different issue then with a broader discussion. I am a bit hesitant here, since I am not a fan of value dependent behavior. the loc case worked accidentally on the 1.3.x branch |
Indeed. I'll open one |
Iloc is probably easy, because it’s always positional, loc and regular getitem is tricky |
Pandas version checks
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
Issue Description
When using loc with a namedtuple, only certain types of indexing work. That is surprising as I assumed that namedtuples (as they are a tuple subtype) would always just add a like a tuple.
The reason, why I used namedtuples in the first place was to make
loc
with multiple index levels easier to read. Therefore, it would be great, if namedtuples would work equivalently to tuples inloc
Expected Behavior
namedtuples should behave like normal tuples, as they are tuple sub-classes
Installed Versions
INSTALLED VERSIONS
commit : e8093ba
python : 3.8.12.final.0
python-bits : 64
OS : Linux
OS-release : 5.19.1-2-MANJARO
Version : #1 SMP PREEMPT_DYNAMIC Thu Aug 11 19:05:47 UTC 2022
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.4.3
numpy : 1.23.2
pytz : 2022.2.1
dateutil : 2.8.2
setuptools : 62.6.0
pip : 22.1.2
Cython : None
pytest : 7.1.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.4.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
markupsafe : 2.1.1
matplotlib : 3.5.3
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.9.0
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
The text was updated successfully, but these errors were encountered: