BUG: Inconsistent behavior between `None` and `pd.NA` #47628

jeff-hernandez · 2022-07-07T20:38:49Z

Pandas version checks

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> import pandas as pd
>>> x = pd.Series(['90210', '60018-0123', '10010', 'text', '6.7', "<class 'object'>", '123456'], dtype="string")
>>> invalid = pd.Series([False, False, False, True, True, True, True])
>>> x[invalid] = None
>>> x[6] = pd.NA
>>> x[2] = None
>>> x
0         90210
1    60018-0123
2          <NA>
3          None
4          None
5          None
6          <NA>
dtype: string

Issue Description

There is inconsistent behavior when assigning missing values to a string series. As shown in the reproducible example, using an integer to assign None will result in a conversion to pd.NA, whereas using a boolean series to assign None will result in None with no conversion.

Expected Behavior

I'd expect there to be a single representation for missing values, so assigning x[invalid] = None would result in a conversion to pd.NA.

>>> import pandas as pd
>>> x = pd.Series(['90210', '60018-0123', '10010', 'text', '6.7', "<class 'object'>", '123456'], dtype="string")
>>> invalid = pd.Series([False, False, False, True, True, True, True])
>>> x[invalid] = None
>>> x
0         90210
1    60018-0123
2         10010
3          <NA>
4          <NA>
5          <NA>
6          <NA>
dtype: string

Installed Versions

INSTALLED VERSIONS

commit : e8093ba
python : 3.9.7.final.0
python-bits : 64
OS : Darwin
OS-release : 21.3.0
Version : Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.4.3
numpy : 1.22.1
pytz : 2021.3
dateutil : 2.8.2
setuptools : 58.0.4
pip : 22.1
Cython : None
pytest : 7.0.1
hypothesis : None
sphinx : 4.2.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.3
IPython : 7.31.1
pandas_datareader: None
bs4 : 4.10.0
bottleneck : None
brotli : None
fastparquet : None
fsspec : 2022.01.0
gcsfs : None
markupsafe : 2.0.1
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 6.0.1
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.7.3
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None

The text was updated successfully, but these errors were encountered:

JMBurley · 2022-07-08T14:37:38Z

I'd expect there to be a single representation for missing values, so assigning x[invalid] = None would result in a conversion to pd.NA.

Agree that there is a problem, but I disagree with the diagnosis & expected behaviour. I think pandas should follow the principle of "least surprise"

x[invalid] = None  # inserts None.  Good behaviour
x[6] = pd.NA  # inserts NA.  Good behaviour
x[2] = None  # inserts NA.  Bad behaviour

jeff-hernandez · 2022-07-08T15:57:21Z

I understand how the conversion can come as a surprise, but isn't one of the reasons for introducing pd.NA to have a single representation for missing values? Other dtypes like boolean and Float64 already insert pd.NA when x[invalid] = None.

phofl · 2022-07-08T19:41:14Z

I think the conversion was an intentional decision, we are doing the same with np.nan, but not completely sure. You would have to check the issue tracker for information about this

Edit: https://pandas.pydata.org/docs/user_guide/integer_na.html#scalar-na-value

jorisvandenbossche · 2022-07-12T09:48:14Z

but isn't one of the reasons for introducing pd.NA to have a single representation for missing values?

Indeed, it is the design goal that the StringArray only uses NA as missing value. And for example upon construction this also happens (conversion from None to NA), like with the scalar setitem:

In [14]: pd.Series(["a", None], dtype="string")
Out[14]: 
0       a
1    <NA>
dtype: string

This is the documented behaviour (see https://pandas.pydata.org/docs/reference/api/pandas.arrays.StringArray.html in addition to the link @phofl provided), and intentional:

pandas/pandas/core/arrays/string_.py

Lines 309 to 323 in 2eca7e1

    
           def _validate(self): 
        
               """Validate that we only store NA or strings.""" 
        
               if len(self._ndarray) and not lib.is_string_array(self._ndarray, skipna=True): 
        
                   raise ValueError("StringArray requires a sequence of strings or pandas.NA") 
        
               if self._ndarray.dtype != "object": 
        
                   raise ValueError( 
        
                       "StringArray requires a sequence of strings or pandas.NA. Got " 
        
                       f"'{self._ndarray.dtype}' dtype instead." 
        
                   ) 
        
               # Check to see if need to convert Na values to pd.NA 
        
               if self._ndarray.ndim > 2: 
        
                   # Ravel if ndims > 2 b/c no cythonized version available 
        
                   lib.convert_nans_to_NA(self._ndarray.ravel("K")) 
        
               else: 
        
                   lib.convert_nans_to_NA(self._ndarray)

But so it seems that for the case of __setitem__ with a mask, we don't do this validation on the input, and that is certainly a bug.

Thanks for the report!

jorisvandenbossche · 2022-07-17T13:11:04Z

It seems this is a regression compared to pandas 1.3 (but already buggy in 1.4 as well).

The StringArray itself seems to do this correctly:

In [17]: arr = pd.array(['a', 'b', 'c'], dtype="string")

In [19]: arr[arr == 'a'] = None

In [20]: arr
Out[20]: 
<StringArray>
[<NA>, 'b', 'c']
Length: 3, dtype: string

But the strange thing is that doing this on the Series level doesn't end up calling StringArray.__setitem__, it seems to go through Series._where and eventually BlockManager.putmask, and ExtensionArray._putmask, and that last one is not correctly implemented for StringArray.

jorisvandenbossche · 2022-07-17T13:47:50Z

I opened a PR with a potential fix -> #47763

simonjayhawkins · 2022-07-20T13:22:32Z

It seems this is a regression compared to pandas 1.3 (but already buggy in 1.4 as well).

first bad commit: [164aa28] ENH: implement EA._putmask (#44387)

But the strange thing is that doing this on the Series level doesn't end up calling StringArray.__setitem__, it seems to go through Series._where and eventually BlockManager.putmask, and ExtensionArray._putmask, and that last one is not correctly implemented for StringArray.

Is this correct to bypass a EA __setitem__ method or could this also affect 3rd party EAs

jeff-hernandez added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jul 7, 2022

jeff-hernandez mentioned this issue Jul 7, 2022

Null invalid PostalCode alteryx/woodwork#1467

Merged

jorisvandenbossche added Strings String extension data type and string data NA - MaskedArrays Related to pd.NA and nullable extension arrays and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Jul 12, 2022

jorisvandenbossche added Regression Functionality that used to work in a prior pandas version and removed Bug labels Jul 17, 2022

jorisvandenbossche mentioned this issue Jul 17, 2022

BUG: fix regression in Series[string] setitem setting a scalar with a mask #47763

Merged

4 tasks

jorisvandenbossche added this to the 1.4.4 milestone Jul 17, 2022

mroeschke closed this as completed in #47763 Jul 18, 2022

phofl mentioned this issue Jul 20, 2022

TST: test for inconsistency due to dtype=string #46512 #47793

Closed

3 tasks

simonjayhawkins added a commit to simonjayhawkins/pandas that referenced this issue Jul 20, 2022

code sample for pandas-dev#47628

9c99d90

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Inconsistent behavior between `None` and `pd.NA` #47628

BUG: Inconsistent behavior between `None` and `pd.NA` #47628

jeff-hernandez commented Jul 7, 2022 •

edited

Loading

INSTALLED VERSIONS

JMBurley commented Jul 8, 2022

jeff-hernandez commented Jul 8, 2022 •

edited

Loading

phofl commented Jul 8, 2022 •

edited

Loading

jorisvandenbossche commented Jul 12, 2022

jorisvandenbossche commented Jul 17, 2022

jorisvandenbossche commented Jul 17, 2022

simonjayhawkins commented Jul 20, 2022

BUG: Inconsistent behavior between None and pd.NA #47628

BUG: Inconsistent behavior between None and pd.NA #47628

Comments

jeff-hernandez commented Jul 7, 2022 • edited Loading

Pandas version checks

Reproducible Example

Issue Description

Expected Behavior

Installed Versions

INSTALLED VERSIONS

JMBurley commented Jul 8, 2022

jeff-hernandez commented Jul 8, 2022 • edited Loading

phofl commented Jul 8, 2022 • edited Loading

jorisvandenbossche commented Jul 12, 2022

jorisvandenbossche commented Jul 17, 2022

jorisvandenbossche commented Jul 17, 2022

simonjayhawkins commented Jul 20, 2022

BUG: Inconsistent behavior between `None` and `pd.NA` #47628

BUG: Inconsistent behavior between `None` and `pd.NA` #47628

jeff-hernandez commented Jul 7, 2022 •

edited

Loading

jeff-hernandez commented Jul 8, 2022 •

edited

Loading

phofl commented Jul 8, 2022 •

edited

Loading