-
-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BUG: interchange protocol with nullable pyarrow datatypes a non-null validity provides nonsense results #57664
Comments
same as #57643, but for pyarrow nullable |
Oof...I think this is a lifecycle issue similar to data-apis/dataframe-api#351 Looking at the pandas implementation, we are doing: elif self.dtype[0] in (
DtypeKind.INT,
DtypeKind.UINT,
DtypeKind.FLOAT,
DtypeKind.BOOL,
):
np_arr = self._col.to_numpy()
buffer = PandasBuffer(np_arr, allow_copy=self._allow_copy)
dtype = self.dtype For non-null arrow arrays this "works" by pure coincidence. The By contrast, when there are null values present it is no longer a zero-copy exchange, so when I think this is a rather large deficiency in the protocol |
As you have already mentioned |
Pandas version checks
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
Issue Description
Result:
Expected Behavior
[1, 2, None]
Installed Versions
INSTALLED VERSIONS
commit : 92a52e2
python : 3.10.12.final.0
python-bits : 64
OS : Linux
OS-release : 5.15.133.1-microsoft-standard-WSL2
Version : #1 SMP Thu Oct 5 21:02:42 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8
pandas : 3.0.0.dev0+355.g92a52e2315
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.8.2
setuptools : 59.6.0
pip : 24.0
Cython : 3.0.8
pytest : 8.0.0
hypothesis : 6.98.6
sphinx : 7.2.6
blosc : None
feather : None
xlsxwriter : 3.1.9
lxml.etree : 5.1.0
html5lib : 1.1
pymysql : 1.4.6
psycopg2 : 2.9.9
jinja2 : 3.1.3
IPython : 8.21.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
bottleneck : 1.3.7
fastparquet : 2024.2.0
fsspec : 2024.2.0
gcsfs : 2024.2.0
matplotlib : 3.8.3
numba : 0.59.0
numexpr : 2.9.0
odfpy : None
openpyxl : 3.1.2
pyarrow : 15.0.0
pyreadstat : 1.2.6
python-calamine : None
pyxlsb : 1.0.10
s3fs : 2024.2.0
scipy : 1.12.0
sqlalchemy : 2.0.27
tables : 3.9.2
tabulate : 0.9.0
xarray : 2024.1.1
xlrd : 2.0.1
zstandard : 0.22.0
tzdata : 2024.1
qtpy : 2.4.1
pyqt5 : None
The text was updated successfully, but these errors were encountered: