-
-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BUG: .str.split()
fails with pyarrow strings
#56271
Comments
Just to clarify, you would expect this to split on |
For Python strings, calling
|
Reproducible with only pyarrow: import pyarrow as pa
test = pa.array(['foo t', 'bar', 'baz'], type=pa.string())
pa.compute.split_pattern(test, pattern=None) Pyarrow needs a string as pattern https://arrow.apache.org/docs/python/generated/pyarrow.compute.split_pattern.html. How does pandas normally handles such cases? Especially if in the pandas documentation is stated:
I'm new around here and happy to contribute if there is anything to do here! |
I would lean towards patching the arrow behaviour in pandas and to specify a proper dtype |
if pat == None:
return ArrowExtensionArray(pa.array([x.as_py().split(pat, n) for x in self._pa_array])) What do you think about something like that (here: https://github.com/pandas-dev/pandas/blob/main/pandas/core/arrays/arrow/array.py#L2352)? |
Pandas version checks
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
Issue Description
In Pandas 1, you could split with no parameters (like python does with
'matt h'.split()
. This doesn't work with pyarrow types.It gives a
TypeError
:Expected Behavior
I expect Pandas 1 behavior.
Installed Versions
INSTALLED VERSIONS
commit : e86ed37
python : 3.10.13.final.0
python-bits : 64
OS : Darwin
OS-release : 21.6.0
Version : Darwin Kernel Version 21.6.0: Wed Aug 10 14:28:23 PDT 2022; root:xnu-8020.141.5~2/RELEASE_ARM64_T6000
machine : arm64
processor : arm
byteorder : little
LC_ALL : en_US.UTF-8
LANG : None
LOCALE : en_US.UTF-8
pandas : 2.1.1
numpy : 1.23.5
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.6.1
pip : 23.2.1
Cython : 3.0.4
pytest : 7.2.0
hypothesis : 6.81.2
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.1.2
lxml.etree : 4.9.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.8.0
pandas_datareader : None
bs4 : 4.11.1
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : 2023.3.0
gcsfs : None
matplotlib : 3.6.2
numba : 0.56.4
numexpr : 2.8.4
odfpy : None
openpyxl : 3.0.10
pandas_gbq : None
pyarrow : 13.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.0
sqlalchemy : 2.0.21
tables : None
tabulate : 0.9.0
xarray : None
xlrd : 2.0.1
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None
The text was updated successfully, but these errors were encountered: