take

Hey @macks22. I was able to recreate this issue on a different OS as well. I am going to start looking into the sourcecode, specifically the df.select_dtypes function to see if I can find what's up.

@macks22
I agree that there is an issue for this function in differentiating between what it considers to be bool vs boolean datatypes which could be a larger issue all toghether.

Here are my examples:

In [1]

import pandas as pd

df = pd.DataFrame({
'a': [1, 2, 3],
'b': [True, False, True],
'c': [1.0, 2.0,3.0]})

print(df.select_dtypes(include='number').columns)
print('')
print(df.info())

Out [1]

Index(['a', 'c'], dtype='object')

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   a       3 non-null      int64  
 1   b       3 non-null      bool
 2   c       3 non-null      float64
dtypes: bool(1), float64(1), int64(1)
memory usage: 179.0 bytes
None

In [2]

import pandas as pd

df = pd.DataFrame({
'a': [1, 2, 3],
'b': pd.Series([True, False, True], dtype=pd.BooleanDtype()),
'c': [1.0, 2.0,3.0]})

print(df.select_dtypes(include='number').columns)
print('')
print(df.info())

Out [2]

Index(['a', 'b', 'c'], dtype='object')

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   a       3 non-null      int64  
 1   b       3 non-null      boolean
 2   c       3 non-null      float64
dtypes: boolean(1), float64(1), int64(1)
memory usage: 182.0 bytes
None

I forked pandas and created the following branch for my debugging:
https://github.com/MichaelRocchio/pandas/tree/issue_46870:

Courses of action:
Check source documentation for differentiation of bool vs boolean and if there should be one.
Create ability of .select_dtypes to work with boolean dtypes.
Check if the creation of boolean datatypes is a singular issue to pandas.BooleanDtype()

Thanks @macks22 for the report.

The DataFrame.select_dtypes method includes columns with dtype of BooleanDtype when include='number' is given as a kwarg.

in 1.2.5 as EAs were not considered and therefore the BooleanDtype column was not included in the result. so will label as a regression.

This is different than the behavior for columns with dtype of bool and seems like a bug.

agreed. numpy backed bool and EA backed boolean should be consistent.

first bad commit: [deaf138] BUG: Allow numeric ExtensionDtypes in DataFrame.select_dtypes (#38246)

@simonjayhawkins
I was able to solve this problem but I am not 100% sure on how to proceed. For now I will create a pull request from my fork and compare it to main.

The bug was in the following location:
pandas/core/arrays/boolean.py

my change is pushed to https://github.com/MichaelRocchio/pandas/tree/issue_46870

It was pretty small:

    @property
    def _is_numeric(self) -> bool:
        return True

changed to:

    @property
    def _is_numeric(self) -> bool:
        return False

Additionally I will need help on how to test if my fix didn't cause other issues.

moving to 1.4.4