Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: pd.DataFrame.to_hdf does not support multiindex frames with string[pyarrow_numpy] #55088

Closed
3 tasks done
nvasilevv opened this issue Sep 11, 2023 · 4 comments · Fixed by #56194
Closed
3 tasks done

BUG: pd.DataFrame.to_hdf does not support multiindex frames with string[pyarrow_numpy] #55088

nvasilevv opened this issue Sep 11, 2023 · 4 comments · Fixed by #56194
Labels
Bug IO HDF5 read_hdf, HDFStore
Milestone
2.1.4

Comments

@nvasilevv
Copy link

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np
data = pd.DataFrame(
    np.random.randint(0, 50, size=(15, 4)), columns=list('ABCD')
).set_index(['A', 'B'])

# This action will be successful as it works on numpy objects
data.to_hdf('test.hdf', key='test', format='table')

# If we force the usage of pyarrow string, the same will fail

pd.options.future.infer_string = True

data_arrow = pd.DataFrame(
    np.random.randint(0, 50, size=(15, 4)), columns=list('ABCD')
).set_index(['A', 'B'])

data_arrow.to_hdf('test_arrow.hdf', key='test', format='table') # This will raise ValueError("cannot have non-object label DataIndexableCol")

Issue Description

Looks like using string[pyarrow_numpy] as dtype and to_hdf with a multiindex frame throws a value error. The error comes from pytables.py and more specifically the check below.

pandas/pandas/io/pytables.py

Lines 2647 to 2650 in 723feb9

def validate_names(self) -> None:
if not is_object_dtype(Index(self.values).dtype):
# TODO: should the message here be more specifically non-str?
raise ValueError("cannot have non-object label DataIndexableCol")

The is_object_dtype always checks against np.object_.

pandas/pandas/core/dtypes/common.py

Lines 137 to 165 in 723feb9

def is_object_dtype(arr_or_dtype) -> bool:
"""
Check whether an array-like or dtype is of the object dtype.
Parameters
----------
arr_or_dtype : array-like or dtype
The array-like or dtype to check.
Returns
-------
boolean
Whether or not the array-like or dtype is of the object dtype.
Examples
--------
>>> from pandas.api.types import is_object_dtype
>>> is_object_dtype(object)
True
>>> is_object_dtype(int)
False
>>> is_object_dtype(np.array([], dtype=object))
True
>>> is_object_dtype(np.array([], dtype=int))
False
>>> is_object_dtype([1, 2, 3])
False
"""
return _is_dtype_type(arr_or_dtype, classes(np.object_))

Expected Behavior

I would expect that is_object_dtype allows for string[pyrrow_numpy] (if and only if downstream behavior is supported by that dtype). If that action is not supported with string[pyarrow_numpy] I would expected a more informative error.

Installed Versions

INSTALLED VERSIONS

commit : 723feb9
python : 3.9.13.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19044
machine : AMD64
processor : Intel64 Family 6 Model 165 Stepping 2, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : English_United States.1252

pandas : 2.2.0dev0+208.g723feb984e
numpy : 1.22.4
pytz : 2022.7.1
dateutil : 2.8.2
setuptools : 68.0.0
pip : 23.2.1
Cython : 0.29.36
pytest : 6.2.5
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : None
pandas_datareader : None
bs4 : None
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : 0.56.4
numexpr : 2.8.5
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 10.0.1
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
sqlalchemy : None
tables : 3.7.0
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None
@nvasilevv nvasilevv added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 11, 2023
@phofl
Copy link
Member

phofl commented Sep 11, 2023

Thx for the report. Your assumption is correct, this should work

@phofl phofl added this to the 2.1.1 milestone Sep 11, 2023
@phofl phofl added IO HDF5 read_hdf, HDFStore and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 11, 2023
@nvasilevv
Copy link
Author

I think that I could try to propose a PR fixing this. It would be my first one in this project, so if you think that the topic is too hard to approach, I am fine with waiting for the solution.

@phofl
Copy link
Member

phofl commented Sep 11, 2023

I'd appreciate the help if you have time. The first thing I would do is figuring out where "pyarrow[string]" is treated differently than the new dtype.

@nvasilevv
Copy link
Author

After undertaking a short investigation, I believe that the different treatment in that case comes from here

https://github.com/pandas-dev/pandas/blob/705d4312cf1d94ef2497bcd8091e0eabd1085f4a/pandas/_libs/lib.pyx#L2700C12-L2706

Essentially, this part always uses pyarrow_numpy as a string type if pd.options.future.infer_string = True. However, if that is not set and instead pd.options.mode.string_storage = 'pyarrow' nothing changes.

I see that this part was changed here - 1ecfa67 and was released in v2.1.0. Previously, the array was not casted at this point.

@lithomas1 lithomas1 modified the milestones: 2.1.1, 2.1.2 Sep 21, 2023
@lithomas1 lithomas1 modified the milestones: 2.1.2, 2.1.3 Oct 26, 2023
@jorisvandenbossche jorisvandenbossche modified the milestones: 2.1.3, 2.1.4 Nov 13, 2023
@phofl phofl mentioned this issue Nov 26, 2023
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug IO HDF5 read_hdf, HDFStore
Projects
None yet
Milestone
2.1.4
Development

Successfully merging a pull request may close this issue.

BUG: hdf can't deal with ea dtype columns phofl/pandas
4 participants
@jorisvandenbossche @lithomas1 @phofl @nvasilevv