Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: Pandas Index fails with bitwise ufuncs #46769

Closed
2 of 3 tasks
galipremsagar opened this issue Apr 13, 2022 · 3 comments · Fixed by #48149
Closed
2 of 3 tasks

BUG: Pandas Index fails with bitwise ufuncs #46769

galipremsagar opened this issue Apr 13, 2022 · 3 comments · Fixed by #48149
Labels
Bug Compat pandas objects compatability with Numpy or Python functions Index Related to the Index class or subclasses Numeric Operations Arithmetic, Comparison, and Logical operations Regression Functionality that used to work in a prior pandas version
Milestone
1.4.4

Comments

@galipremsagar
Copy link

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np
print(pd.__version__)
print(np.__version__)
pandas_idx = [pd.Int64Index([6, 5, 3, 6, 5, 6, 3, 7, 7, 7, 2, 2, 9, 4, 2, 7, 4, 4, 7, 2, 2, 8,
            3, 9, 9, 5, 4, 2, 4, 8, 9, 3, 7, 4, 6, 1, 3, 6, 7, 7, 6, 4, 2, 4,
            6, 5, 1, 9, 6, 2, 9, 1, 5, 4, 7, 2, 1, 1, 7, 7, 3, 4, 6, 1, 1, 5,
            6, 3, 7, 3, 7, 2, 1, 2, 8, 7, 3, 1, 9, 5, 7, 7, 2, 1, 5, 6, 2, 3,
            4, 2, 2, 6, 2, 4, 8, 2, 8, 2, 2, 2],
           dtype='int64'), pd.Int64Index([5, 6, 3, 1, 8, 3, 7, 2, 8, 8, 7, 5, 5, 3, 5, 2, 8, 5, 6, 2, 1, 7,
            9, 6, 6, 7, 2, 1, 5, 3, 7, 3, 4, 3, 1, 8, 8, 8, 1, 2, 8, 3, 3, 3,
            2, 2, 5, 5, 9, 1, 4, 5, 8, 3, 3, 9, 3, 9, 9, 5, 1, 6, 8, 2, 1, 6,
            5, 7, 9, 4, 8, 6, 5, 8, 8, 7, 2, 6, 4, 1, 4, 7, 2, 4, 4, 7, 2, 3,
            8, 1, 4, 1, 2, 3, 3, 9, 6, 8, 1, 1],
           dtype='int64')]
res = np.bitwise_and(*pandas_idx)
print(res)
print(np.bitwise_and(6, 5))


1.4.2
1.22.3
del3.py:5: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  pandas_idx = [pd.Int64Index([6, 5, 3, 6, 5, 6, 3, 7, 7, 7, 2, 2, 9, 4, 2, 7, 4, 4, 7, 2, 2, 8,
del3.py:10: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  dtype='int64'), pd.Int64Index([5, 6, 3, 1, 8, 3, 7, 2, 8, 8, 7, 5, 5, 3, 5, 2, 8, 5, 6, 2, 1, 7,
del3.py:16: FutureWarning: Index.__and__ operating as a set operation is deprecated, in the future this will be a logical operation matching Series.__and__.  Use index.intersection(other) instead.
  res = np.bitwise_and(*pandas_idx)
Int64Index([6, 5, 3, 7, 2, 9, 4, 8, 1], dtype='int64')
4

Issue Description

When calling bitwise numpy ufuncs on Index objects, they seem to return wrong results. For instance numpy.bitwise_and, numpy.bitwise_or and numpy.bitwise_xor are the ones that worked correctly until pandas 1.3.5 and returning wrong results afterwards.

Expected Behavior

1.3.5
1.22.3
Int64Index([4, 4, 3, 0, 0, 2, 3, 2, 0, 0, 2, 0, 1, 0, 0, 2, 0, 4, 6, 2, 0, 0,
            1, 0, 0, 5, 0, 0, 4, 0, 1, 3, 4, 0, 0, 0, 0, 0, 1, 2, 0, 0, 2, 0,
            2, 0, 1, 1, 0, 0, 0, 1, 0, 0, 3, 0, 1, 1, 1, 5, 1, 4, 0, 0, 1, 4,
            4, 3, 1, 0, 0, 2, 1, 0, 8, 7, 2, 0, 0, 1, 4, 7, 2, 0, 4, 6, 2, 3,
            0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0],
           dtype='int64')
4

Installed Versions

Replace this line with the output of pd.show_versions()
@galipremsagar galipremsagar added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Apr 13, 2022
@simonjayhawkins simonjayhawkins added this to the 1.4.3 milestone Apr 14, 2022
@simonjayhawkins simonjayhawkins added Numeric Operations Arithmetic, Comparison, and Logical operations Index Related to the Index class or subclasses labels Apr 14, 2022
@simonjayhawkins simonjayhawkins changed the title BUG: Pandas Index fails with bitwise unfuncs BUG: Pandas Index fails with bitwise ufuncs Apr 14, 2022
simonjayhawkins added a commit to simonjayhawkins/pandas that referenced this issue Apr 14, 2022
@simonjayhawkins
code sample for pandas-dev#46769
51b0306
@simonjayhawkins
Copy link
Member

Thanks @galipremsagar for the report.

For instance numpy.bitwise_and, numpy.bitwise_or and numpy.bitwise_xor are the ones that worked correctly until pandas 1.3.5 and returning wrong results afterwards.

first bad commit: [538d2fa] ENH: implement Index.array_ufunc (#43904)

cc @jbrockmendel

@simonjayhawkins simonjayhawkins added Compat pandas objects compatability with Numpy or Python functions Regression Functionality that used to work in a prior pandas version and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Apr 25, 2022
@jbrockmendel
Copy link
Member

Looks like it is dispatching to the set operation, probably through maybe_dispatch_ufunc_to_dunder_op. We'll need to patch __array_ufunc__ for those cases until 2.0, when the patch can be removed

@simonjayhawkins
Copy link
Member

moving to 1.4.4

@simonjayhawkins simonjayhawkins modified the milestones: 1.4.3, 1.4.4 Jun 22, 2022
@jorisvandenbossche jorisvandenbossche mentioned this issue Aug 19, 2022
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Compat pandas objects compatability with Numpy or Python functions Index Related to the Index class or subclasses Numeric Operations Arithmetic, Comparison, and Logical operations Regression Functionality that used to work in a prior pandas version
Projects
None yet
Milestone
1.4.4
Development

Successfully merging a pull request may close this issue.

REGR: fix calling numpy bitwise ufunc with Index objects jorisvandenbossche/pandas
3 participants
@jbrockmendel @galipremsagar @simonjayhawkins