-
-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BUG: Inconsistent behavior between None
and pd.NA
#47628
Comments
Agree that there is a problem, but I disagree with the diagnosis & expected behaviour. I think pandas should follow the principle of "least surprise"
|
I understand how the conversion can come as a surprise, but isn't one of the reasons for introducing |
I think the conversion was an intentional decision, we are doing the same with np.nan, but not completely sure. You would have to check the issue tracker for information about this Edit: https://pandas.pydata.org/docs/user_guide/integer_na.html#scalar-na-value |
Indeed, it is the design goal that the StringArray only uses NA as missing value. And for example upon construction this also happens (conversion from None to NA), like with the scalar setitem:
This is the documented behaviour (see https://pandas.pydata.org/docs/reference/api/pandas.arrays.StringArray.html in addition to the link @phofl provided), and intentional: pandas/pandas/core/arrays/string_.py Lines 309 to 323 in 2eca7e1
But so it seems that for the case of Thanks for the report! |
It seems this is a regression compared to pandas 1.3 (but already buggy in 1.4 as well). The StringArray itself seems to do this correctly:
But the strange thing is that doing this on the Series level doesn't end up calling |
I opened a PR with a potential fix -> #47763 |
first bad commit: [164aa28] ENH: implement EA._putmask (#44387)
Is this correct to bypass a EA __setitem__ method or could this also affect 3rd party EAs |
Pandas version checks
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
Issue Description
There is inconsistent behavior when assigning missing values to a string series. As shown in the reproducible example, using an integer to assign
None
will result in a conversion topd.NA
, whereas using a boolean series to assignNone
will result inNone
with no conversion.Expected Behavior
I'd expect there to be a single representation for missing values, so assigning
x[invalid] = None
would result in a conversion topd.NA
.Installed Versions
INSTALLED VERSIONS
commit : e8093ba
python : 3.9.7.final.0
python-bits : 64
OS : Darwin
OS-release : 21.3.0
Version : Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.4.3
numpy : 1.22.1
pytz : 2021.3
dateutil : 2.8.2
setuptools : 58.0.4
pip : 22.1
Cython : None
pytest : 7.0.1
hypothesis : None
sphinx : 4.2.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.3
IPython : 7.31.1
pandas_datareader: None
bs4 : 4.10.0
bottleneck : None
brotli : None
fastparquet : None
fsspec : 2022.01.0
gcsfs : None
markupsafe : 2.0.1
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 6.0.1
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.7.3
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
The text was updated successfully, but these errors were encountered: