Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: interchange protocol with nullable pyarrow datatypes a non-null validity provides nonsense results #57664

Closed
3 tasks done
MarcoGorelli opened this issue Feb 28, 2024 · 3 comments · Fixed by #57764
Closed
3 tasks done

BUG: interchange protocol with nullable pyarrow datatypes a non-null validity provides nonsense results #57664

MarcoGorelli opened this issue Feb 28, 2024 · 3 comments · Fixed by #57764
Labels
Bug Interchange Dataframe Interchange Protocol

Comments

@MarcoGorelli
Copy link
Member

MarcoGorelli commented Feb 28, 2024 •
edited
Loading

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pyarrow.interchange as pai
import pandas as pd
import polars as pl

data = pd.DataFrame(
    {
        "b1": pd.Series([1, 2, None], dtype='Int64[pyarrow]'),
    }
)
print(pd.api.interchange.from_dataframe(data.__dataframe__()))
print(pai.from_dataframe(data))
print(pl.from_dataframe(data))

Issue Description

Result:

                    b1
0  4607182418800017408
1  4611686018427387904
2  9221120237041090560
pyarrow.Table
b1: int64
----
b1: [[4607182418800017408,4611686018427387904,9221120237041090560]]
shape: (3, 1)
┌─────────────────────┐
│ b1                  │
│ ---                 │
│ i64                 │
╞═════════════════════╡
│ 4607182418800017408 │
│ 4611686018427387904 │
│ 9221120237041090560 │
└─────────────────────┘

Expected Behavior

[1, 2, None]

Installed Versions

INSTALLED VERSIONS

commit : 92a52e2
python : 3.10.12.final.0
python-bits : 64
OS : Linux
OS-release : 5.15.133.1-microsoft-standard-WSL2
Version : #1 SMP Thu Oct 5 21:02:42 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 3.0.0.dev0+355.g92a52e2315
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.8.2
setuptools : 59.6.0
pip : 24.0
Cython : 3.0.8
pytest : 8.0.0
hypothesis : 6.98.6
sphinx : 7.2.6
blosc : None
feather : None
xlsxwriter : 3.1.9
lxml.etree : 5.1.0
html5lib : 1.1
pymysql : 1.4.6
psycopg2 : 2.9.9
jinja2 : 3.1.3
IPython : 8.21.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
bottleneck : 1.3.7
fastparquet : 2024.2.0
fsspec : 2024.2.0
gcsfs : 2024.2.0
matplotlib : 3.8.3
numba : 0.59.0
numexpr : 2.9.0
odfpy : None
openpyxl : 3.1.2
pyarrow : 15.0.0
pyreadstat : 1.2.6
python-calamine : None
pyxlsb : 1.0.10
s3fs : 2024.2.0
scipy : 1.12.0
sqlalchemy : 2.0.27
tables : 3.9.2
tabulate : 0.9.0
xarray : 2024.1.1
xlrd : 2.0.1
zstandard : 0.22.0
tzdata : 2024.1
qtpy : 2.4.1
pyqt5 : None
@MarcoGorelli MarcoGorelli added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Feb 28, 2024
@MarcoGorelli MarcoGorelli changed the title BUG: interchange protocol with nullable datatypes a non-null validity provides nonsense results BUG: interchange protocol with nullable pyarrow datatypes a non-null validity provides nonsense results Feb 28, 2024
@MarcoGorelli
Copy link
Member Author

same as #57643, but for pyarrow nullable

@MarcoGorelli MarcoGorelli added Interchange Dataframe Interchange Protocol and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Feb 28, 2024
@WillAyd
Copy link
Member

WillAyd commented Feb 28, 2024

Oof...I think this is a lifecycle issue similar to data-apis/dataframe-api#351

Looking at the pandas implementation, we are doing:

        elif self.dtype[0] in (
            DtypeKind.INT,
            DtypeKind.UINT,
            DtypeKind.FLOAT,
            DtypeKind.BOOL,
        ):
            np_arr = self._col.to_numpy()
            buffer = PandasBuffer(np_arr, allow_copy=self._allow_copy)
            dtype = self.dtype

For non-null arrow arrays this "works" by pure coincidence. The self._col.to_numpy() call is zero-copy from Arrow -> NumPy, and creates a view that does not own the data. When that NumPy array goes out of scope the memory address it stored is still valid as long as the original PyArrow array is:

By contrast, when there are null values present it is no longer a zero-copy exchange, so when buffer and np_arr get garbage collected the heap storage np_arr pointed to originally is free'd. Someone somewhere else still has a buffer that points to this location, but it is garbage.

I think this is a rather large deficiency in the protocol

@WillAyd
Copy link
Member

WillAyd commented Feb 28, 2024

As you have already mentioned self._col.to_numpy() is also not really viable if it performs any kind of conversions, which in this case it does. There are any number of workarounds to that problem although I think we are best served in the long term if we stop trying to fit everything through a numpy lens and create the buffer managing objects we need at a lower level

@MarcoGorelli MarcoGorelli mentioned this issue Mar 7, 2024
Merged
@MarcoGorelli MarcoGorelli mentioned this issue Jul 26, 2024
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Interchange Dataframe Interchange Protocol
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

BUG: PyArrow dtypes were not supported in the interchange protocol MarcoGorelli/pandas
2 participants
@WillAyd @MarcoGorelli