BUG: pd.read_pickle not backwards compatible from v2.1.0 to v1.3.4 #55137

elerys · 2023-09-14T15:26:35Z

Pandas version checks

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

# in a pandas 1.3.4 environment

import pandas as pd
import numpy as np

df = pd.DataFrame(np.ones((100, 4)))
df.to_pickle("old_pandas.pkl")



# in pandas 2.1.0 environment
import pandas as pd

df = pd.read_pickle("old_pandas.pkl")

Issue Description

The read_pickle method raises an error when attempting to read a file serialized with to_pickle in pandas 1.3.4. No error is raised when using 2.0. to read_pickle, or if the file is serialized with version >=1.4.

The documentation states the method should be backwards compatible to pandas 0.20.3 provided the object was serialized with to_pickle (as in this case).

Traceback (most recent call last):
  File "/home/me/envs/new_pandas/lib/python3.10/site-packages/pandas/io/pickle.py", line 206, in read_pickle
    return pickle.load(handles.handle)
  File "/home/me/envs/new_pandas/lib/python3.10/site-packages/pandas/core/internals/blocks.py", line 2400, in new_block
    return klass(values, ndim=ndim, placement=placement, refs=refs)
TypeError: Argument 'placement' has incorrect type (expected pandas._libs.internals.BlockPlacement, got slice)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/me/envs/new_pandas/lib/python3.10/site-packages/pandas/compat/pickle_compat.py", line 35, in load_reduce
    stack[-1] = func(*args)
  File "/home/me/envs/new_pandas/lib/python3.10/site-packages/pandas/core/internals/blocks.py", line 2400, in new_block
    return klass(values, ndim=ndim, placement=placement, refs=refs)
TypeError: Argument 'placement' has incorrect type (expected pandas._libs.internals.BlockPlacement, got slice)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/me/envs/new_pandas/lib/python3.10/site-packages/pandas/io/pickle.py", line 211, in read_pickle
    return pc.load(handles.handle, encoding=None)
  File "/home/me/envs/new_pandas/lib/python3.10/site-packages/pandas/compat/pickle_compat.py", line 225, in load
    return up.load()
  File "/usr/lib/python3.10/pickle.py", line 1213, in load
    dispatch[key[0]](self)
  File "/home/me/envs/new_pandas/lib/python3.10/site-packages/pandas/compat/pickle_compat.py", line 55, in load_reduce
    elif args and issubclass(args[0], PeriodArray):
TypeError: issubclass() arg 1 must be a class

Expected Behavior

Method should be backwards compatible, as per the documentation.

Installed Versions

INSTALLED VERSIONS

commit : f00efd0
python : 3.10.12.final.0
python-bits : 64
OS : Linux
OS-release : 5.15.0-78-generic
Version : #85-Ubuntu SMP Fri Jul 7 15:25:09 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8

pandas : 2.2.0dev0+225.gf00efd0344
numpy : 1.25.2
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 59.6.0
pip : 22.0.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader : None
bs4 : None
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

INSTALLED VERSIONS ------------------ commit : 945c9ed python : 3.10.12.final.0 python-bits : 64 OS : Linux OS-release : 5.15.0-78-generic Version : #85-Ubuntu SMP Fri Jul 7 15:25:09 UTC 2023 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_GB.UTF-8 LOCALE : en_GB.UTF-8

pandas : 1.3.4
numpy : 1.25.2
pytz : 2023.3.post1
dateutil : 2.8.2
pip : 22.0.2
setuptools : 59.6.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

The text was updated successfully, but these errors were encountered:

aterrel · 2023-09-15T06:51:42Z

Support seems to be removed in #49316

Need to decide if we want to revert back or update the documentation

aterrel · 2023-09-16T14:02:42Z

Investigating this more, pandas 0.20.3 still works for this example but not pandas 1.3.4.

Will add both examples in PR.

aterrel · 2023-09-17T12:49:09Z

Okay bisected and I see the bug with 1.3.4 starting at 15fd7d7

* PERF: new_block * fix arraymanager build

timvdlee · 2023-11-08T09:20:21Z

Is there an update on this issue? I am encountering the same bug

PennyWieser · 2023-11-08T15:39:53Z

Same bug here.

PapaNoyel · 2023-11-09T10:57:56Z

Encountered it today.

v0lle85 · 2023-11-17T20:38:35Z

I'm affected, too.
Thanks to the comment by tomvannuenen in this issue I tried downgrading pandas to version 2.0.3, which in my case does the trick for now.
(Most likely I could have known by looking up the commit that broke it mentioned by aterrel, but anyway...)

patticat · 2023-11-21T14:51:11Z

Same problem. 2.0.3 is the latest version that "works" for me... I've been checking this issue for weeks hoping for the best

aterrel · 2023-11-30T16:08:20Z

@mroeschke

DanielFPerez · 2023-12-01T13:00:53Z

Facing the same issue

ferrannoguera · 2023-12-07T15:29:54Z

I am still facing the same issue with pickles that were created using pandas 1.3.5 and display the same error messages when trying to be read using pandas 2.1.3

Precisely, pickles whose compression was gzip.

mroeschke · 2023-12-07T17:01:33Z

A fix for this issue should be available when 2.1.4 is released (either end of the week or next week)

tgutzler · 2024-01-09T13:33:01Z

Hi, I have a pickle file (uncertain which version it was created with, it's protocol version 4). When I try to open it with pandas 2.1.4, I get this error:
File "[...]/miniconda3/envs/terratest/lib/python3.9/site-packages/pandas/core/internals/blocks.py", line 2400, in new_block
return klass(values, ndim=ndim, placement=placement, refs=refs)
TypeError: Argument 'placement' has incorrect type (expected pandas._libs.internals.BlockPlacement, got slice)

When I load this with pandas 1.4.2 and dump it again, without changing anything, I can load it fine with pandas 2.1.4.
This is all with python 3.9.18.
How can I get to the bottom of this? I don't want to attach the file here but happy to pm it

lithomas1 · 2024-01-11T03:39:47Z

Hi, I have a pickle file (uncertain which version it was created with, it's protocol version 4). When I try to open it with pandas 2.1.4, I get this error: File "[...]/miniconda3/envs/terratest/lib/python3.9/site-packages/pandas/core/internals/blocks.py", line 2400, in new_block return klass(values, ndim=ndim, placement=placement, refs=refs) TypeError: Argument 'placement' has incorrect type (expected pandas._libs.internals.BlockPlacement, got slice)

When I load this with pandas 1.4.2 and dump it again, without changing anything, I can load it fine with pandas 2.1.4. This is all with python 3.9.18. How can I get to the bottom of this? I don't want to attach the file here but happy to pm it

Can you open a new issue?

tgutzler · 2024-01-11T04:50:10Z

Of course: #56825

liuty1999 · 2024-03-11T07:33:56Z

I downgrade pandas to 2.0.4, then it works without bugs

sahilm89 · 2024-06-12T00:09:08Z

I still have this problem in 2.2.2! I had to downgrade to 2.0.3 for pandas to be compatible.

elerys added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 14, 2023

rhshadrach added the IO Pickle read_pickle, to_pickle label Sep 14, 2023

jorisvandenbossche mentioned this issue Sep 16, 2023

Potential recent bug reports to investigate jorisvandenbossche/pydata-amsterdam-pandas-sprint#5

Open

12 tasks

aterrel referenced this issue Sep 17, 2023

PERF: new_block (#52326)

15fd7d7

* PERF: new_block * fix arraymanager build

SarahAlidoost mentioned this issue Sep 20, 2023

pd.read_pickle returns error VegeWaterDynamics/motrainer#76

Closed

Emmavt mentioned this issue Oct 15, 2023

Specific pandas version required to read in pickled results dlab-berkeley/Python-Web-APIs#4

Closed

cliffckerr mentioned this issue Oct 28, 2023

Fix tests sciris/sciris#543

Closed

lithomas1 added Regression Functionality that used to work in a prior pandas version and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Dec 3, 2023

lithomas1 self-assigned this Dec 3, 2023

lithomas1 added this to the 2.1.4 milestone Dec 3, 2023

lithomas1 mentioned this issue Dec 3, 2023

REGR: Fix reading old pickles from pandas 1.3.5 #56308

Merged

5 tasks

mroeschke closed this as completed in #56308 Dec 5, 2023

dbrattli mentioned this issue Dec 11, 2023

fix: upgrade pandas to 2.1.4 cognitedata/indsl#69

Merged

21 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: pd.read_pickle not backwards compatible from v2.1.0 to v1.3.4 #55137

BUG: pd.read_pickle not backwards compatible from v2.1.0 to v1.3.4 #55137

elerys commented Sep 14, 2023

INSTALLED VERSIONS

aterrel commented Sep 15, 2023

aterrel commented Sep 16, 2023

aterrel commented Sep 17, 2023 •

edited

Loading

timvdlee commented Nov 8, 2023

PennyWieser commented Nov 8, 2023

PapaNoyel commented Nov 9, 2023

v0lle85 commented Nov 17, 2023 •

edited

Loading

patticat commented Nov 21, 2023

aterrel commented Nov 30, 2023

DanielFPerez commented Dec 1, 2023

ferrannoguera commented Dec 7, 2023

mroeschke commented Dec 7, 2023

tgutzler commented Jan 9, 2024

lithomas1 commented Jan 11, 2024

tgutzler commented Jan 11, 2024

liuty1999 commented Mar 11, 2024

sahilm89 commented Jun 12, 2024

BUG: pd.read_pickle not backwards compatible from v2.1.0 to v1.3.4 #55137

BUG: pd.read_pickle not backwards compatible from v2.1.0 to v1.3.4 #55137

Comments

elerys commented Sep 14, 2023

Pandas version checks

Reproducible Example

Issue Description

Expected Behavior

Installed Versions

INSTALLED VERSIONS

aterrel commented Sep 15, 2023

aterrel commented Sep 16, 2023

aterrel commented Sep 17, 2023 • edited Loading

timvdlee commented Nov 8, 2023

PennyWieser commented Nov 8, 2023

PapaNoyel commented Nov 9, 2023

v0lle85 commented Nov 17, 2023 • edited Loading

patticat commented Nov 21, 2023

aterrel commented Nov 30, 2023

DanielFPerez commented Dec 1, 2023

ferrannoguera commented Dec 7, 2023

mroeschke commented Dec 7, 2023

tgutzler commented Jan 9, 2024

lithomas1 commented Jan 11, 2024

tgutzler commented Jan 11, 2024

liuty1999 commented Mar 11, 2024

sahilm89 commented Jun 12, 2024

aterrel commented Sep 17, 2023 •

edited

Loading

v0lle85 commented Nov 17, 2023 •

edited

Loading