Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: Series with dtype='timestamp[ns][pyarrow]' Raises ArrowTypeError with String Dates #56266

Closed
2 of 3 tasks
mattharrison opened this issue Nov 30, 2023 · 1 comment · Fixed by #56294
Closed
2 of 3 tasks

BUG: Series with dtype='timestamp[ns][pyarrow]' Raises ArrowTypeError with String Dates #56266

mattharrison opened this issue Nov 30, 2023 · 1 comment · Fixed by #56294
Labels
Arrow pyarrow functionality Bug Dtype Conversions Unexpected or buggy dtype conversions

Comments

@mattharrison
Copy link

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

# Sample data: string representations of dates
string_dates = ['2020-01-01 04:30:00', '2020-01-02 00:00:00', '2020-01-03 00:00:00']

# Attempt to directly convert to 'timestamp[ns][pyarrow]' dtype (expected to raise an error)
try:
    series = pd.Series(string_dates, dtype='timestamp[ns][pyarrow]')
    print(series)
except Exception as e:
    print("Error:", str(e))

# Workaround: Convert to Pandas 1.x date first, then to 'timestamp[ns][pyarrow]'
workaround_series = pd.Series(pd.to_datetime(string_dates)).astype('timestamp[ns][pyarrow]')
print(workaround_series)

Issue Description

In Pandas 2.1 attempting to directly convert a series of string representations of dates to a timestamp[ns][pyarrow] dtype results in an ArrowTypeError. This behavior is unexpected as it should ideally handle the conversion from string dates directly without requiring an intermediate conversion to Pandas 1.x date format.

Expected Behavior

The series of string dates should be converted to timestamp[ns][pyarrow] dtype without errors.

Installed Versions

INSTALLED VERSIONS

commit : e86ed37
python : 3.10.13.final.0
python-bits : 64
OS : Darwin
OS-release : 21.6.0
Version : Darwin Kernel Version 21.6.0: Wed Aug 10 14:28:23 PDT 2022; root:xnu-8020.141.5~2/RELEASE_ARM64_T6000
machine : arm64
processor : arm
byteorder : little
LC_ALL : en_US.UTF-8
LANG : None
LOCALE : en_US.UTF-8

pandas : 2.1.1
numpy : 1.23.5
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.6.1
pip : 23.2.1
Cython : 3.0.4
pytest : 7.2.0
hypothesis : 6.81.2
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.1.2
lxml.etree : 4.9.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.8.0
pandas_datareader : None
bs4 : 4.11.1
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : 2023.3.0
gcsfs : None
matplotlib : 3.6.2
numba : 0.56.4
numexpr : 2.8.4
odfpy : None
openpyxl : 3.0.10
pandas_gbq : None
pyarrow : 13.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.0
sqlalchemy : 2.0.21
tables : None
tabulate : 0.9.0
xarray : None
xlrd : 2.0.1
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None
@mattharrison mattharrison added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Nov 30, 2023
@mroeschke
Copy link
Member

Looks like the core issue is that pyarrow.array doesn't have any native date parsing functionality when told to cast strings to dates, but pyarrow.array.cast does

(Pdb) value
['2020-01-01 04:30:00', '2020-01-02 00:00:00', '2020-01-03 00:00:00']
(Pdb) pa_type
TimestampType(timestamp[ns])
(Pdb) pa.array(value, type=pa_type)
*** pyarrow.lib.ArrowTypeError: object of type <class 'str'> cannot be converted to int
(Pdb) pa.array(value, from_pandas=True).cast(pa_type)
<pyarrow.lib.TimestampArray object at 0x1453b0040>
[
  2020-01-01 04:30:00.000000000,
  2020-01-02 00:00:00.000000000,
  2020-01-03 00:00:00.000000000
]

Looks like we try to use cast internally, so the ArrowTypeError needs to be caught to fallback

@mroeschke mroeschke added Dtype Conversions Unexpected or buggy dtype conversions Arrow pyarrow functionality and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Nov 30, 2023
@mroeschke mroeschke mentioned this issue Dec 2, 2023
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Arrow pyarrow functionality Bug Dtype Conversions Unexpected or buggy dtype conversions
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

BUG: Series(strings, dtype=ArrowDtype[timestamp]) raising mroeschke/pandas
2 participants
@mattharrison @mroeschke