-
-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
REGR: preserve reindexed array object (instead of creating new array) for concat with all-NA array #47762
REGR: preserve reindexed array object (instead of creating new array) for concat with all-NA array #47762
Conversation
@jorisvandenbossche the added test passes without the code change? |
Ah, I get that indeed as well. It was certainly failing, so it seems that in the meantime, there was another change in main that affected this. It seems that because of removing the usage of internal |
makes sense.
indeed. the two merge tests pass in 1.3.5, fail in 1.4.3 and then pass again on main. the added concat test passes in 1.3.5 and fails in 1.4.3 and on main. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @jorisvandenbossche generally lgtm.
type = int | ||
name = "int_attr" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
these should be float?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't matter for the test, but yet that's less confusing ;)
return FloatAttrArray(make_data()) | ||
|
||
|
||
def test_concat_with_all_na(data): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we don't use the fixture
|
||
result = pd.concat([df1.set_index("key"), df2.set_index("key")], axis=1) | ||
expected = pd.DataFrame( | ||
{"col": arr.take([0, 0, 1]), "col2": [1, np.nan, 2], "key": [0, 1, 2]} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should be arr.take([ 0, 1, -1]?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since it is all NaNs, it actually doesn't matter what we are taking. But changed it to make it less confusing
pandas/core/internals/concat.py
Outdated
@@ -471,7 +471,11 @@ def get_reindexed_values(self, empty_dtype: DtypeObj, upcasted_na) -> ArrayLike: | |||
if len(values) and values[0] is None: | |||
fill_value = None | |||
|
|||
if is_datetime64tz_dtype(empty_dtype): | |||
if is_dtype_equal(blk_dtype, empty_dtype) and self.indexers: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is there a reason this is caught here, instead of in the elif is_1d_only_ea_dtype
block. In the OP you mention the benefits for EA types. What are the benefits for using this path for the builtin dtypes? (and for a targeted backport?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can move it into the is_1d_only_ea_dtype
block. That would fix it for the specific use case of custom extension dtypes.
Looking a bit more in detail, if we have a numpy dtype, the approach with np.empty might actually faster than take
:
In [6]: arr = np.array([np.nan]*10000)
In [7]: indexer = np.arange(10000, dtype="int32")
In [9]: %timeit pd.core.algorithms.take_nd(arr, indexer, axis=0)
28.4 µs ± 752 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
In [11]: %%timeit
...: arr2 = np.empty(arr.shape, dtype=arr.dtype)
...: arr2.fill(np.nan)
5.68 µs ± 135 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
So that's a good reason to keep the check specific for extension dtypes.
@simonjayhawkins thanks for the review! |
pandas/core/internals/concat.py
Outdated
@@ -471,21 +471,26 @@ def get_reindexed_values(self, empty_dtype: DtypeObj, upcasted_na) -> ArrayLike: | |||
if len(values) and values[0] is None: | |||
fill_value = None | |||
|
|||
if is_datetime64tz_dtype(empty_dtype): | |||
elif is_datetime64tz_dtype(empty_dtype): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this didn't need to change? (just a leftover from the code move?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, good catch
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @jorisvandenbossche lgtm. can merge on green(ish).
@meeseeksdev backport 1.5.x |
…instead of creating new array) for concat with all-NA array
…instead of creating new array) for concat with all-NA array
Thanks @jorisvandenbossche |
* add-recommended-dependencies-as-extras_require-in-setup.cfg See issue #47335. #47335 recommended dependencies should have package mgmt facilitated through pandas. This will make mgmt of pandas in production docker environments a lot simpler. * Update v1.4.3.rst * double backtick code in rst * rebundle under extras_require `recommended` * [options.extras_require] bundled dependencies by featureset see #39164 for discussion. * note: [options.extras_require] bundled dependencies by featureset * Update setup.cfg rollback numba from recommended. This would necessitate and update to documentation that requires broad agreement from pd-dev-core team that would slow down the overall PR * add adlfs for azure to `access data in cloud` see #39164 for discussion. 0.6.0 might be an overly restrictive version, but it is compatible * fix extras_require: PyTables is actually `tables` on PyPi * Update setup.cfg * add `all` option to [options.extras_require] * moved changelog to 1.4.4 as 1.4.3 released while this PR was stalled * Updated to 1.5.0 compliance * simplify sql option names * extras rename: recommended -> performance * remove azure support is currently unofficial as of 1.5.0 * align with actions-38-minimum_versions.yaml add specific installs and, where required, missing install documentation for - odfpy - pyreadstat - compression options * Pandas -> pandas in doc Co-authored-by: Matthew Roeschke <emailformattr@gmail.com> * extras rename: s3 -> aws see #47336 (comment) * extras rename: table -> output_formatting to be more general in case of future changes * bug: `>=` not `=` * Apply suggestions from code review Co-authored-by: Simon Hawkins <simonjayhawkins@gmail.com> * align 1.5.0.rst to latest extras_require updates * 1.5.0.rst example updated to use valid extras * add optional dep mgmt instructions to install.rst * lint scipy optional import Co-authored-by: Matthew Roeschke <emailformattr@gmail.com> * Apply suggestions from code review * detailed extras guidance in install.rst - updated numbas to a full recommended dependency with a promotional bullet point like bottleneck and numexpr - clarified the extra to use for each set of optional dependencies - made xml an optional extra, because is does have usage outside of read_html. * _optional.py note to keep track of setup.cfg * bug: indent after bullet in install.rst * remove numba from computation extra. * Backport PR #48197 on branch 1.5.x (DOC: Cleanup 1.5 whatsnew) (#48228) Backport PR #48197: DOC: Cleanup 1.5 whatsnew Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR #48232 on branch 1.5.x (CI: Ensure jobs run on 1.5.x branch) (#48235) Backport PR #48232: CI: Ensure jobs run on 1.5.x branch Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR #48180 on branch 1.5.x (CI: Switch to large for circleci) (#48251) Backport PR #48180: CI: Switch to large for circleci Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48245 on branch 1.5.x (CI: Skip test_round_sanity tests due to failures) (#48257) Backport PR #48245: CI: Skip test_round_sanity tests due to failures Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48240 on branch 1.5.x (Fix mypy erroring on backport branches) (#48259) Backport PR #48240: Fix mypy erroring on backport branches Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48215 on branch 1.5.x (REGR: properly update DataFrame cache in Series.__setitem__) (#48268) Backport PR #48215: REGR: properly update DataFrame cache in Series.__setitem__ Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR #48272 on branch 1.5.x (CI: Require s3fs greater than minumum version in builds) (#48276) Backport PR #48272: CI: Require s3fs greater than minumum version in builds Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48299 on branch 1.5.x (Bump s3fs to 2021.08.00) (#48305) Backport PR #48299: Bump s3fs to 2021.08.00 Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48027 on branch 1.5.x (ENH: Support masks in groupby prod) (#48302) Backport PR #48027: ENH: Support masks in groupby prod Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #47762 on branch 1.5.x (REGR: preserve reindexed array object (instead of creating new array) for concat with all-NA array) (#48309) Backport PR #47762: REGR: preserve reindexed array object (instead of creating new array) for concat with all-NA array Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR #48246 on branch 1.5.x (REGR: iloc not possible for sparse DataFrame) (#48311) Backport PR #48246: REGR: iloc not possible for sparse DataFrame Co-authored-by: Simon Hawkins <simonjayhawkins@gmail.com> * Backport PR #48314 on branch 1.5.x (DOC: v1.4.4 release date and tidy up release notes) (#48320) Backport PR #48314: DOC: v1.4.4 release date and tidy up release notes Co-authored-by: Simon Hawkins <simonjayhawkins@gmail.com> * Backport PR #48301 on branch 1.5.x (DEPR: Deprecate positional arguments in pivot) (#48326) Backport PR #48301: DEPR: Deprecate positional arguments in pivot Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48214 on branch 1.5.x (WEB: Removing links to pdf version of the docs from web and docs) (#48242) Backport PR #48214: WEB: Removing links to pdf version of the docs from web and docs * Backport PR #48159 on branch 1.5.x (TST: Fix interchange/plotting/groupby test warnings) (#48279) Backport PR #48159: TST: Fix interchange/plotting/groupby test warnings Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR #48324 on branch 1.5.x (BUG: Add note in whatsnew for DataFrame.at behavior change) (#48345) Backport PR #48324: BUG: Add note in whatsnew for DataFrame.at behavior change Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR #48336 on branch 1.5.x (DOC: Add whatsnew note for #45404) (#48341) Backport PR #48336: DOC: Add whatsnew note for #45404 Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR #48254 on branch 1.5.x (REF: avoid FutureWarning about using deprecates loc.__setitem__ non-inplace usage) (#48353) Backport PR #48254: REF: avoid FutureWarning about using deprecates loc.__setitem__ non-inplace usage Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR #48334 on branch 1.5.x (BUG: read_html(extract_links=all) with no header) (#48350) Backport PR #48334: BUG: read_html(extract_links=all) with no header Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR #48265 on branch 1.5.x (CI: Setting up ssh key to upload prod docs) (#48370) Backport PR #48265: CI: Setting up ssh key to upload prod docs Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR #48381 on branch 1.5.x (CI: Pin mambaforge image) (#48401) Backport PR #48381: CI: Pin mambaforge image Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48229 on branch 1.5.x (TST: Test Nullable int floordiv by 0) (#48413) Backport PR #48229: TST: Test Nullable int floordiv by 0 Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR #48414 on branch 1.5.x (DOC: Add deprecation to is_categorical) (#48418) Backport PR #48414: DOC: Add deprecation to is_categorical Co-authored-by: Kevin Sheppard <bashtage@users.noreply.github.com> * Backport PR #48264 on branch 1.5.x (BUG: ArrowExtensionArray._from_* accepts pyarrow arrays) (#48422) * Backport PR #48264: BUG: ArrowExtensionArray._from_* accepts pyarrow arrays * Add missing import Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR #48411 on branch 1.5.x (REGR: get_loc for ExtensionEngine not returning bool indexer for na) (#48430) Backport PR #48411: REGR: get_loc for ExtensionEngine not returning bool indexer for na Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48419 on branch 1.5.x (BUG: ensure to return writable buffer in __dataframe__ interchange for categorical column) (#48441) Backport PR #48419: BUG: ensure to return writable buffer in __dataframe__ interchange for categorical column Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR #48444 on branch 1.5.x (CI: Pin ipython version) (#48449) Backport PR #48444: CI: Pin ipython version Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48380 on branch 1.5.x (DOC: Clarify that objects dtype takes precedence in where) (#48445) * Backport PR #48380: DOC: Clarify that objects dtype takes precedence in where * Update generic.py Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Manual Backport PR #48427 on branch 1.5.x (BLD: Refactor Dockerfile to not install dev enviornment on base) (#48450) Backport PR #48427: BLD: Refactor Dockerfile to not install dev enviornment on base * Backport PR #48426 on branch 1.5.x (BUG: Column.size should be a method) (#48465) Backport PR #48426: BUG: Column.size should be a method Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR #48398 on branch 1.5.x (WARN: Avoid FutureWarnings in tests) (#48420) * Backport PR #48398: WARN: Avoid FutureWarnings in tests * Update Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR #48416 on branch 1.5.x (REF: ensure to apply suffixes before concat step in merge code) (#48470) Backport PR #48416: REF: ensure to apply suffixes before concat step in merge code Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR #48354 on branch 1.5.x (CI: Bump timeout to 180 minutes) (#48474) Backport PR #48354: CI: Bump timeout to 180 minutes Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR #48472 on branch 1.5.x (PERF: keep using ObjectEngine for ExtensionArrays for 1.5) (#48486) Backport PR #48472: PERF: keep using ObjectEngine for ExtensionArrays for 1.5 Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR #48473 on branch 1.5.x (REGR: .describe on unsigned dtypes results in object) (#48501) Backport PR #48473: REGR: .describe on unsigned dtypes results in object Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR #48443 on branch 1.5.x (BUG: Fix pyarrow groupby tests) (#48494) * BUG: Fix pyarrow groupby tests (#48443) # Conflicts: # pandas/tests/extension/test_arrow.py * CI: Fix failing tests (#48493) Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR #48490 on branch 1.5.x (CI: Use -j1 for python-dev build to avoid flaky build error) (#48517) Backport PR #48490: CI: Use -j1 for python-dev build to avoid flaky build error Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Manual Backport PR #48428 on branch 1.5.x (BUG/TST: fix a bunch of arraymanager+pyarrow tests) (#48518) Backport PR #48428: BUG/TST: fix a bunch of arraymanager+pyarrow tests Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR #48525 on branch 1.5.x (CI: Fix py311 builds different exception message) (#48529) Backport PR #48525: CI: Fix py311 builds different exception message Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48489 on branch 1.5.x (BUG: fix test_arrow.py tests) (#48532) Backport PR #48489: BUG: fix test_arrow.py tests Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR #48543 on branch 1.5.x (DOC: Update footer and include OVH) (#48548) Backport PR #48543: DOC: Update footer and include OVH Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Manual Backport PR #48417 on branch 1.5.x (Revert set_index inplace and copy keyword changes) (#48552) Backport PR #48417: Revert set_index inplace and copy keyword changes Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR #48550 on branch 1.5.x (TST: remove 2D tests irrelevant for pyarrow) (#48554) Backport PR #48550: TST: remove 2D tests irrelevant for pyarrow Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR #48556 on branch 1.5.x (DOC: Fix docs footer) (#48558) Backport PR #48556: DOC: Fix docs footer Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR #48562 on branch 1.5.x (TST: Testing that no warnings are emitted and that inplace fillna produces the correct result (GH48480)) (#48564) Backport PR #48562: TST: Testing that no warnings are emitted and that inplace fillna produces the correct result (GH48480) Co-authored-by: RaphSku <45042665+RaphSku@users.noreply.github.com> * Backport PR #48563 on branch 1.5.x (DOC: Fix read_sas 1.5 release notes) (#48565) Backport PR #48563: DOC: Fix read_sas 1.5 release notes Co-authored-by: Jonas Haag <jonas@lophus.org> * Backport PR #48539 on branch 1.5.x (REGR: groupby doesn't identify null values when sort=False) (#48568) Backport PR #48539: REGR: groupby doesn't identify null values when sort=False Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR #48555 on branch 1.5.x (DEPR: Series.astype(np.datetime64)) (#48569) Backport PR #48555: DEPR: Series.astype(np.datetime64) Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR #48557 on branch 1.5.x (WEB: Add new footer to web) (#48571) Backport PR #48557: WEB: Add new footer to web Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR #48285 on branch 1.5.x (WEB: Unpin pydata sphinx theme) (#48585) Backport PR #48285: WEB: Unpin pydata sphinx theme Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48572 on branch 1.5.x (DOC: Fixing styles for the dark theme) (#48584) Backport PR #48572: DOC: Fixing styles for the dark theme Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR #48397 on branch 1.5.x (WARN: Remove false positive warning for iloc inplaceness) (#48583) Backport PR #48397: WARN: Remove false positive warning for iloc inplaceness Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48587 on branch 1.5.x (Fix `series.str.startswith(tuple)`) (#48593) Backport PR #48587: Fix `series.str.startswith(tuple)` Co-authored-by: Janosh Riebesell <janosh.riebesell@gmail.com> * Backport PR #48601 on branch 1.5.x (CI: Fix matplolib release issues) (#48617) Backport PR #48601: CI: Fix matplolib release issues Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48623 on branch 1.5.x (REGR/DOC: Docs left navbar broke) (#48625) Backport PR #48623: REGR/DOC: Docs left navbar broke Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR #48619 on branch 1.5.x (REGR: Loc.setitem with enlargement raises for nested data) (#48629) Backport PR #48619: REGR: Loc.setitem with enlargement raises for nested data Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48627 on branch 1.5.x (DOC: Last changes to release notes for 1.5.0 release) (#48630) Backport PR #48627: DOC: Last changes to release notes for 1.5.0 release Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * RLS: 1.5.0 * Backport PR #48642 on branch 1.5.x (DOC: Add release notes for 1.5.1) (#48647) Backport PR #48642: DOC: Add release notes for 1.5.1 Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48639 on branch 1.5.x (CI: Fix directory name for published prod docs) (#48648) Backport PR #48639: CI: Fix directory name for published prod docs Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR #48651 on branch 1.5.x (REGR: TextIOWrapper raising an error in read_csv) (#48666) Backport PR #48651: REGR: TextIOWrapper raising an error in read_csv Co-authored-by: Torsten Wörtwein <twoertwein@users.noreply.github.com> * Backport PR #48599 on branch 1.5.x (DOC: Add deprecation infos to deprecated functions) (#48690) Backport PR #48599: DOC: Add deprecation infos to deprecated functions Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48620 on branch 1.5.x (REGR: Performance decrease in factorize) (#48710) Backport PR #48620: REGR: Performance decrease in factorize Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR #48711 on branch 1.5.x (REGR: Regression in DataFrame.loc when setting df with all True indexer) (#48717) Backport PR #48711: REGR: Regression in DataFrame.loc when setting df with all True indexer Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48696 on branch 1.5.x (REGR: to_hdf raising AssertionError with boolean index) (#48716) Backport PR #48696: REGR: to_hdf raising AssertionError with boolean index Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48734 on branch 1.5.x (REGR: Raise on invalid colormap for scatter plot) (#48744) Backport PR #48734: REGR: Raise on invalid colormap for scatter plot Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48713 on branch 1.5.x (BUG: pivot_table raising Future Warning with datetime column as index) (#48742) Backport PR #48713: BUG: pivot_table raising Future Warning with datetime column as index Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48703 on branch 1.5.x (Added theme_switcher ) (#48741) Backport PR #48703: Added theme_switcher Co-authored-by: Deepak Sirohiwal <38135521+deepaksirohiwal@users.noreply.github.com> * Backport PR #48697 on branch 1.5.x (REGR: None converted to NaN when enlarging Series) (#48745) Backport PR #48697: REGR: None converted to NaN when enlarging Series Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48702 on branch 1.5.x (REGR: dropna affects observed in groupby) (#48750) Backport PR #48702: REGR: dropna affects observed in groupby Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR #48782 on branch 1.5.x (REGR: describe raising when result contains NA) (#48793) Backport PR #48782: REGR: describe raising when result contains NA Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48662 on branch 1.5.x (BUG: Series.getitem not falling back to positional for bool index) (#48799) Backport PR #48662: BUG: Series.getitem not falling back to positional for bool index Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48751 on branch 1.5.x (WEB: Update link to datapythonista blog url) (#48798) Backport PR #48751: WEB: Update link to datapythonista blog url Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR #48608 on branch 1.5.x (REGR: assert_index_equal raising with non matching pd.NA) (#48800) * Backport PR #48608: REGR: assert_index_equal raising with non matching pd.NA Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48785 on branch 1.5.x (BUG: still emitting unnecessary FutureWarning in DataFrame.sort_values with sparse columns) (#48807) Backport PR #48785: BUG: still emitting unnecessary FutureWarning in DataFrame.sort_values with sparse columns Co-authored-by: Marco Edward Gorelli <33491632+MarcoGorelli@users.noreply.github.com> * Backport PR #48693 on branch 1.5.x (ENH: Make deprecate_nonkeyword_arguments alter function signature) (#48795) Backport PR #48693: ENH: Make deprecate_nonkeyword_arguments alter function signature Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com> * Backport PR #48579 on branch 1.5.x (BUG: Fix calling groupBy(...).apply(func) on an empty dataframe invokes func) (#48817) BUG: Fix calling groupBy(...).apply(func) on an empty dataframe invokes func (#48579) (cherry picked from commit 8b0ad71) Co-authored-by: Dennis Chukwunta <chuksmcdennis@yahoo.com> * Backport PR #48760 on branch 1.5.x (REGR: groupby.size with axis=1 doesn't return a Series) (#48825) * Backport PR #48820 on branch 1.5.x (BUG: to_datetime(format='...%f') parses nanoseconds) (#48860) BUG: to_datetime(format='...%f') parses nanoseconds (#48820) Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR #48850 on branch 1.5.x (TYP: Fix typing errors caused by new numpy) (#48859) * Backport PR #48790 on branch 1.5.x (Created icons for dark theme) (#48875) Backport PR #48790: Created icons for dark theme Co-authored-by: Lorenzo Vainigli <lorenzovngl@gmail.com> * Backport PR #48805 on branch 1.5.x (Added padding and fixed columns for sponsor logos in mobile view) (#48874) Backport PR #48805: Added padding and fixed columns for sponsor logos in mobile view Co-authored-by: Amay Patel <92037532+amay-patel@users.noreply.github.com> * Backport PR #48866 on branch 1.5.x (REGR: replace replacing wrong values with inplace and datetime) (#48872) Backport PR #48866: REGR: replace replacing wrong values with inplace and datetime Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48853 on branch 1.5.x (REGR: Avoid unnecessary warning when setting empty dataframe) (#48873) Backport PR #48853: REGR: Avoid unnecessary warning when setting empty dataframe Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR #48833 on branch 1.5.x (BUG: ArrowExtensionArray compared to invalid object not raising) (#48878) Backport PR #48833: BUG: ArrowExtensionArray compared to invalid object not raising Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR #48686 on branch 1.5.x (BUG: to_datetime(tz_mix, utc=True) converts to UTC) (#48882) Backport PR #48686: BUG: to_datetime(tz_mix, utc=True) converts to UTC Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR #48736 on branch 1.5.x ( BUG: AttributeError: 'function' object has no attribute 'currentframe') (#48887) * Backport PR #48797 on branch 1.5.x (REGR: fix df.apply with keyword non-zero axis) (#48886) REGR: fix df.apply with keyword non-zero axis (#48797) Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com> * add pandas[foo] throughout to explain the optional extras process Also `Recommended dependencies` to `Performance dependencies (recommended)` * extend optional_extra`all` to cover tests and redundant packages * add tzdata from pandas 1.5.0 * move summary to latest whatsnew doc Code is now stable and updated. Attempt a port from doc/source/whatsnew/v1.5.0.rst over to latest document doc/source/whatsnew/v1.5.1.rst. * explicitly pair packages to optional_extra in installs * fix sphinx errors in install.rst extra padding on column + empty return before table * add: pytest-asyncio>=0.19.0 pytest-asyncio>=0.19.0 Closes #48361. No version guidance available so went for latest. No-one should be messing around with tests without modern installs. * fsspec note * repin pytest-asyncio>=0.17.0 matches pytest-asyncio>=0.17.0 in ci/deps/actions-38-minimum_versions.yaml * move summary from whatsnew/v1.5.1 to v1.6.0 plus add (:issue:`48361`) resolution note * linting double-backtick ``test`` underline tilde match title length * Add `clipboard` as optional extra * Review comments - fss optional extra - better numba description * remove unneeded comment * fix: leave 1.5.1.rst unchanged by this PR * Update doc/source/whatsnew/v1.5.1.rst Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * pyarrow bump 1.0.1 -> 6.0.0 Co-authored-by: Matthew Roeschke <emailformattr@gmail.com> Co-authored-by: Simon Hawkins <simonjayhawkins@gmail.com> Co-authored-by: MeeseeksMachine <39504233+meeseeksmachine@users.noreply.github.com> Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> Co-authored-by: Marc Garcia <garcia.marc@gmail.com> Co-authored-by: Kevin Sheppard <bashtage@users.noreply.github.com> Co-authored-by: RaphSku <45042665+RaphSku@users.noreply.github.com> Co-authored-by: Jonas Haag <jonas@lophus.org> Co-authored-by: Janosh Riebesell <janosh.riebesell@gmail.com> Co-authored-by: Pandas Development Team <pandas-dev@python.org> Co-authored-by: Torsten Wörtwein <twoertwein@users.noreply.github.com> Co-authored-by: Deepak Sirohiwal <38135521+deepaksirohiwal@users.noreply.github.com> Co-authored-by: Marco Edward Gorelli <33491632+MarcoGorelli@users.noreply.github.com> Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com> Co-authored-by: Dennis Chukwunta <chuksmcdennis@yahoo.com> Co-authored-by: Lorenzo Vainigli <lorenzovngl@gmail.com> Co-authored-by: Amay Patel <92037532+amay-patel@users.noreply.github.com>
…dev#47336) * add-recommended-dependencies-as-extras_require-in-setup.cfg See issue pandas-dev#47335. pandas-dev#47335 recommended dependencies should have package mgmt facilitated through pandas. This will make mgmt of pandas in production docker environments a lot simpler. * Update v1.4.3.rst * double backtick code in rst * rebundle under extras_require `recommended` * [options.extras_require] bundled dependencies by featureset see pandas-dev#39164 for discussion. * note: [options.extras_require] bundled dependencies by featureset * Update setup.cfg rollback numba from recommended. This would necessitate and update to documentation that requires broad agreement from pd-dev-core team that would slow down the overall PR * add adlfs for azure to `access data in cloud` see pandas-dev#39164 for discussion. 0.6.0 might be an overly restrictive version, but it is compatible * fix extras_require: PyTables is actually `tables` on PyPi * Update setup.cfg * add `all` option to [options.extras_require] * moved changelog to 1.4.4 as 1.4.3 released while this PR was stalled * Updated to 1.5.0 compliance * simplify sql option names * extras rename: recommended -> performance * remove azure support is currently unofficial as of 1.5.0 * align with actions-38-minimum_versions.yaml add specific installs and, where required, missing install documentation for - odfpy - pyreadstat - compression options * Pandas -> pandas in doc Co-authored-by: Matthew Roeschke <emailformattr@gmail.com> * extras rename: s3 -> aws see pandas-dev#47336 (comment) * extras rename: table -> output_formatting to be more general in case of future changes * bug: `>=` not `=` * Apply suggestions from code review Co-authored-by: Simon Hawkins <simonjayhawkins@gmail.com> * align 1.5.0.rst to latest extras_require updates * 1.5.0.rst example updated to use valid extras * add optional dep mgmt instructions to install.rst * lint scipy optional import Co-authored-by: Matthew Roeschke <emailformattr@gmail.com> * Apply suggestions from code review * detailed extras guidance in install.rst - updated numbas to a full recommended dependency with a promotional bullet point like bottleneck and numexpr - clarified the extra to use for each set of optional dependencies - made xml an optional extra, because is does have usage outside of read_html. * _optional.py note to keep track of setup.cfg * bug: indent after bullet in install.rst * remove numba from computation extra. * Backport PR pandas-dev#48197 on branch 1.5.x (DOC: Cleanup 1.5 whatsnew) (pandas-dev#48228) Backport PR pandas-dev#48197: DOC: Cleanup 1.5 whatsnew Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48232 on branch 1.5.x (CI: Ensure jobs run on 1.5.x branch) (pandas-dev#48235) Backport PR pandas-dev#48232: CI: Ensure jobs run on 1.5.x branch Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48180 on branch 1.5.x (CI: Switch to large for circleci) (pandas-dev#48251) Backport PR pandas-dev#48180: CI: Switch to large for circleci Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48245 on branch 1.5.x (CI: Skip test_round_sanity tests due to failures) (pandas-dev#48257) Backport PR pandas-dev#48245: CI: Skip test_round_sanity tests due to failures Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48240 on branch 1.5.x (Fix mypy erroring on backport branches) (pandas-dev#48259) Backport PR pandas-dev#48240: Fix mypy erroring on backport branches Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48215 on branch 1.5.x (REGR: properly update DataFrame cache in Series.__setitem__) (pandas-dev#48268) Backport PR pandas-dev#48215: REGR: properly update DataFrame cache in Series.__setitem__ Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR pandas-dev#48272 on branch 1.5.x (CI: Require s3fs greater than minumum version in builds) (pandas-dev#48276) Backport PR pandas-dev#48272: CI: Require s3fs greater than minumum version in builds Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48299 on branch 1.5.x (Bump s3fs to 2021.08.00) (pandas-dev#48305) Backport PR pandas-dev#48299: Bump s3fs to 2021.08.00 Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48027 on branch 1.5.x (ENH: Support masks in groupby prod) (pandas-dev#48302) Backport PR pandas-dev#48027: ENH: Support masks in groupby prod Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#47762 on branch 1.5.x (REGR: preserve reindexed array object (instead of creating new array) for concat with all-NA array) (pandas-dev#48309) Backport PR pandas-dev#47762: REGR: preserve reindexed array object (instead of creating new array) for concat with all-NA array Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR pandas-dev#48246 on branch 1.5.x (REGR: iloc not possible for sparse DataFrame) (pandas-dev#48311) Backport PR pandas-dev#48246: REGR: iloc not possible for sparse DataFrame Co-authored-by: Simon Hawkins <simonjayhawkins@gmail.com> * Backport PR pandas-dev#48314 on branch 1.5.x (DOC: v1.4.4 release date and tidy up release notes) (pandas-dev#48320) Backport PR pandas-dev#48314: DOC: v1.4.4 release date and tidy up release notes Co-authored-by: Simon Hawkins <simonjayhawkins@gmail.com> * Backport PR pandas-dev#48301 on branch 1.5.x (DEPR: Deprecate positional arguments in pivot) (pandas-dev#48326) Backport PR pandas-dev#48301: DEPR: Deprecate positional arguments in pivot Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48214 on branch 1.5.x (WEB: Removing links to pdf version of the docs from web and docs) (pandas-dev#48242) Backport PR pandas-dev#48214: WEB: Removing links to pdf version of the docs from web and docs * Backport PR pandas-dev#48159 on branch 1.5.x (TST: Fix interchange/plotting/groupby test warnings) (pandas-dev#48279) Backport PR pandas-dev#48159: TST: Fix interchange/plotting/groupby test warnings Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48324 on branch 1.5.x (BUG: Add note in whatsnew for DataFrame.at behavior change) (pandas-dev#48345) Backport PR pandas-dev#48324: BUG: Add note in whatsnew for DataFrame.at behavior change Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR pandas-dev#48336 on branch 1.5.x (DOC: Add whatsnew note for pandas-dev#45404) (pandas-dev#48341) Backport PR pandas-dev#48336: DOC: Add whatsnew note for pandas-dev#45404 Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48254 on branch 1.5.x (REF: avoid FutureWarning about using deprecates loc.__setitem__ non-inplace usage) (pandas-dev#48353) Backport PR pandas-dev#48254: REF: avoid FutureWarning about using deprecates loc.__setitem__ non-inplace usage Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR pandas-dev#48334 on branch 1.5.x (BUG: read_html(extract_links=all) with no header) (pandas-dev#48350) Backport PR pandas-dev#48334: BUG: read_html(extract_links=all) with no header Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48265 on branch 1.5.x (CI: Setting up ssh key to upload prod docs) (pandas-dev#48370) Backport PR pandas-dev#48265: CI: Setting up ssh key to upload prod docs Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR pandas-dev#48381 on branch 1.5.x (CI: Pin mambaforge image) (pandas-dev#48401) Backport PR pandas-dev#48381: CI: Pin mambaforge image Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48229 on branch 1.5.x (TST: Test Nullable int floordiv by 0) (pandas-dev#48413) Backport PR pandas-dev#48229: TST: Test Nullable int floordiv by 0 Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48414 on branch 1.5.x (DOC: Add deprecation to is_categorical) (pandas-dev#48418) Backport PR pandas-dev#48414: DOC: Add deprecation to is_categorical Co-authored-by: Kevin Sheppard <bashtage@users.noreply.github.com> * Backport PR pandas-dev#48264 on branch 1.5.x (BUG: ArrowExtensionArray._from_* accepts pyarrow arrays) (pandas-dev#48422) * Backport PR pandas-dev#48264: BUG: ArrowExtensionArray._from_* accepts pyarrow arrays * Add missing import Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48411 on branch 1.5.x (REGR: get_loc for ExtensionEngine not returning bool indexer for na) (pandas-dev#48430) Backport PR pandas-dev#48411: REGR: get_loc for ExtensionEngine not returning bool indexer for na Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48419 on branch 1.5.x (BUG: ensure to return writable buffer in __dataframe__ interchange for categorical column) (pandas-dev#48441) Backport PR pandas-dev#48419: BUG: ensure to return writable buffer in __dataframe__ interchange for categorical column Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR pandas-dev#48444 on branch 1.5.x (CI: Pin ipython version) (pandas-dev#48449) Backport PR pandas-dev#48444: CI: Pin ipython version Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48380 on branch 1.5.x (DOC: Clarify that objects dtype takes precedence in where) (pandas-dev#48445) * Backport PR pandas-dev#48380: DOC: Clarify that objects dtype takes precedence in where * Update generic.py Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Manual Backport PR pandas-dev#48427 on branch 1.5.x (BLD: Refactor Dockerfile to not install dev enviornment on base) (pandas-dev#48450) Backport PR pandas-dev#48427: BLD: Refactor Dockerfile to not install dev enviornment on base * Backport PR pandas-dev#48426 on branch 1.5.x (BUG: Column.size should be a method) (pandas-dev#48465) Backport PR pandas-dev#48426: BUG: Column.size should be a method Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48398 on branch 1.5.x (WARN: Avoid FutureWarnings in tests) (pandas-dev#48420) * Backport PR pandas-dev#48398: WARN: Avoid FutureWarnings in tests * Update Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR pandas-dev#48416 on branch 1.5.x (REF: ensure to apply suffixes before concat step in merge code) (pandas-dev#48470) Backport PR pandas-dev#48416: REF: ensure to apply suffixes before concat step in merge code Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR pandas-dev#48354 on branch 1.5.x (CI: Bump timeout to 180 minutes) (pandas-dev#48474) Backport PR pandas-dev#48354: CI: Bump timeout to 180 minutes Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48472 on branch 1.5.x (PERF: keep using ObjectEngine for ExtensionArrays for 1.5) (pandas-dev#48486) Backport PR pandas-dev#48472: PERF: keep using ObjectEngine for ExtensionArrays for 1.5 Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR pandas-dev#48473 on branch 1.5.x (REGR: .describe on unsigned dtypes results in object) (pandas-dev#48501) Backport PR pandas-dev#48473: REGR: .describe on unsigned dtypes results in object Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR pandas-dev#48443 on branch 1.5.x (BUG: Fix pyarrow groupby tests) (pandas-dev#48494) * BUG: Fix pyarrow groupby tests (pandas-dev#48443) # Conflicts: # pandas/tests/extension/test_arrow.py * CI: Fix failing tests (pandas-dev#48493) Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR pandas-dev#48490 on branch 1.5.x (CI: Use -j1 for python-dev build to avoid flaky build error) (pandas-dev#48517) Backport PR pandas-dev#48490: CI: Use -j1 for python-dev build to avoid flaky build error Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Manual Backport PR pandas-dev#48428 on branch 1.5.x (BUG/TST: fix a bunch of arraymanager+pyarrow tests) (pandas-dev#48518) Backport PR pandas-dev#48428: BUG/TST: fix a bunch of arraymanager+pyarrow tests Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR pandas-dev#48525 on branch 1.5.x (CI: Fix py311 builds different exception message) (pandas-dev#48529) Backport PR pandas-dev#48525: CI: Fix py311 builds different exception message Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48489 on branch 1.5.x (BUG: fix test_arrow.py tests) (pandas-dev#48532) Backport PR pandas-dev#48489: BUG: fix test_arrow.py tests Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR pandas-dev#48543 on branch 1.5.x (DOC: Update footer and include OVH) (pandas-dev#48548) Backport PR pandas-dev#48543: DOC: Update footer and include OVH Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Manual Backport PR pandas-dev#48417 on branch 1.5.x (Revert set_index inplace and copy keyword changes) (pandas-dev#48552) Backport PR pandas-dev#48417: Revert set_index inplace and copy keyword changes Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR pandas-dev#48550 on branch 1.5.x (TST: remove 2D tests irrelevant for pyarrow) (pandas-dev#48554) Backport PR pandas-dev#48550: TST: remove 2D tests irrelevant for pyarrow Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR pandas-dev#48556 on branch 1.5.x (DOC: Fix docs footer) (pandas-dev#48558) Backport PR pandas-dev#48556: DOC: Fix docs footer Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR pandas-dev#48562 on branch 1.5.x (TST: Testing that no warnings are emitted and that inplace fillna produces the correct result (GH48480)) (pandas-dev#48564) Backport PR pandas-dev#48562: TST: Testing that no warnings are emitted and that inplace fillna produces the correct result (GH48480) Co-authored-by: RaphSku <45042665+RaphSku@users.noreply.github.com> * Backport PR pandas-dev#48563 on branch 1.5.x (DOC: Fix read_sas 1.5 release notes) (pandas-dev#48565) Backport PR pandas-dev#48563: DOC: Fix read_sas 1.5 release notes Co-authored-by: Jonas Haag <jonas@lophus.org> * Backport PR pandas-dev#48539 on branch 1.5.x (REGR: groupby doesn't identify null values when sort=False) (pandas-dev#48568) Backport PR pandas-dev#48539: REGR: groupby doesn't identify null values when sort=False Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR pandas-dev#48555 on branch 1.5.x (DEPR: Series.astype(np.datetime64)) (pandas-dev#48569) Backport PR pandas-dev#48555: DEPR: Series.astype(np.datetime64) Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR pandas-dev#48557 on branch 1.5.x (WEB: Add new footer to web) (pandas-dev#48571) Backport PR pandas-dev#48557: WEB: Add new footer to web Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR pandas-dev#48285 on branch 1.5.x (WEB: Unpin pydata sphinx theme) (pandas-dev#48585) Backport PR pandas-dev#48285: WEB: Unpin pydata sphinx theme Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48572 on branch 1.5.x (DOC: Fixing styles for the dark theme) (pandas-dev#48584) Backport PR pandas-dev#48572: DOC: Fixing styles for the dark theme Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR pandas-dev#48397 on branch 1.5.x (WARN: Remove false positive warning for iloc inplaceness) (pandas-dev#48583) Backport PR pandas-dev#48397: WARN: Remove false positive warning for iloc inplaceness Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48587 on branch 1.5.x (Fix `series.str.startswith(tuple)`) (pandas-dev#48593) Backport PR pandas-dev#48587: Fix `series.str.startswith(tuple)` Co-authored-by: Janosh Riebesell <janosh.riebesell@gmail.com> * Backport PR pandas-dev#48601 on branch 1.5.x (CI: Fix matplolib release issues) (pandas-dev#48617) Backport PR pandas-dev#48601: CI: Fix matplolib release issues Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48623 on branch 1.5.x (REGR/DOC: Docs left navbar broke) (pandas-dev#48625) Backport PR pandas-dev#48623: REGR/DOC: Docs left navbar broke Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR pandas-dev#48619 on branch 1.5.x (REGR: Loc.setitem with enlargement raises for nested data) (pandas-dev#48629) Backport PR pandas-dev#48619: REGR: Loc.setitem with enlargement raises for nested data Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48627 on branch 1.5.x (DOC: Last changes to release notes for 1.5.0 release) (pandas-dev#48630) Backport PR pandas-dev#48627: DOC: Last changes to release notes for 1.5.0 release Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * RLS: 1.5.0 * Backport PR pandas-dev#48642 on branch 1.5.x (DOC: Add release notes for 1.5.1) (pandas-dev#48647) Backport PR pandas-dev#48642: DOC: Add release notes for 1.5.1 Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48639 on branch 1.5.x (CI: Fix directory name for published prod docs) (pandas-dev#48648) Backport PR pandas-dev#48639: CI: Fix directory name for published prod docs Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR pandas-dev#48651 on branch 1.5.x (REGR: TextIOWrapper raising an error in read_csv) (pandas-dev#48666) Backport PR pandas-dev#48651: REGR: TextIOWrapper raising an error in read_csv Co-authored-by: Torsten Wörtwein <twoertwein@users.noreply.github.com> * Backport PR pandas-dev#48599 on branch 1.5.x (DOC: Add deprecation infos to deprecated functions) (pandas-dev#48690) Backport PR pandas-dev#48599: DOC: Add deprecation infos to deprecated functions Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48620 on branch 1.5.x (REGR: Performance decrease in factorize) (pandas-dev#48710) Backport PR pandas-dev#48620: REGR: Performance decrease in factorize Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR pandas-dev#48711 on branch 1.5.x (REGR: Regression in DataFrame.loc when setting df with all True indexer) (pandas-dev#48717) Backport PR pandas-dev#48711: REGR: Regression in DataFrame.loc when setting df with all True indexer Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48696 on branch 1.5.x (REGR: to_hdf raising AssertionError with boolean index) (pandas-dev#48716) Backport PR pandas-dev#48696: REGR: to_hdf raising AssertionError with boolean index Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48734 on branch 1.5.x (REGR: Raise on invalid colormap for scatter plot) (pandas-dev#48744) Backport PR pandas-dev#48734: REGR: Raise on invalid colormap for scatter plot Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48713 on branch 1.5.x (BUG: pivot_table raising Future Warning with datetime column as index) (pandas-dev#48742) Backport PR pandas-dev#48713: BUG: pivot_table raising Future Warning with datetime column as index Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48703 on branch 1.5.x (Added theme_switcher ) (pandas-dev#48741) Backport PR pandas-dev#48703: Added theme_switcher Co-authored-by: Deepak Sirohiwal <38135521+deepaksirohiwal@users.noreply.github.com> * Backport PR pandas-dev#48697 on branch 1.5.x (REGR: None converted to NaN when enlarging Series) (pandas-dev#48745) Backport PR pandas-dev#48697: REGR: None converted to NaN when enlarging Series Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48702 on branch 1.5.x (REGR: dropna affects observed in groupby) (pandas-dev#48750) Backport PR pandas-dev#48702: REGR: dropna affects observed in groupby Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR pandas-dev#48782 on branch 1.5.x (REGR: describe raising when result contains NA) (pandas-dev#48793) Backport PR pandas-dev#48782: REGR: describe raising when result contains NA Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48662 on branch 1.5.x (BUG: Series.getitem not falling back to positional for bool index) (pandas-dev#48799) Backport PR pandas-dev#48662: BUG: Series.getitem not falling back to positional for bool index Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48751 on branch 1.5.x (WEB: Update link to datapythonista blog url) (pandas-dev#48798) Backport PR pandas-dev#48751: WEB: Update link to datapythonista blog url Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR pandas-dev#48608 on branch 1.5.x (REGR: assert_index_equal raising with non matching pd.NA) (pandas-dev#48800) * Backport PR pandas-dev#48608: REGR: assert_index_equal raising with non matching pd.NA Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48785 on branch 1.5.x (BUG: still emitting unnecessary FutureWarning in DataFrame.sort_values with sparse columns) (pandas-dev#48807) Backport PR pandas-dev#48785: BUG: still emitting unnecessary FutureWarning in DataFrame.sort_values with sparse columns Co-authored-by: Marco Edward Gorelli <33491632+MarcoGorelli@users.noreply.github.com> * Backport PR pandas-dev#48693 on branch 1.5.x (ENH: Make deprecate_nonkeyword_arguments alter function signature) (pandas-dev#48795) Backport PR pandas-dev#48693: ENH: Make deprecate_nonkeyword_arguments alter function signature Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com> * Backport PR pandas-dev#48579 on branch 1.5.x (BUG: Fix calling groupBy(...).apply(func) on an empty dataframe invokes func) (pandas-dev#48817) BUG: Fix calling groupBy(...).apply(func) on an empty dataframe invokes func (pandas-dev#48579) (cherry picked from commit 8b0ad71) Co-authored-by: Dennis Chukwunta <chuksmcdennis@yahoo.com> * Backport PR pandas-dev#48760 on branch 1.5.x (REGR: groupby.size with axis=1 doesn't return a Series) (pandas-dev#48825) * Backport PR pandas-dev#48820 on branch 1.5.x (BUG: to_datetime(format='...%f') parses nanoseconds) (pandas-dev#48860) BUG: to_datetime(format='...%f') parses nanoseconds (pandas-dev#48820) Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48850 on branch 1.5.x (TYP: Fix typing errors caused by new numpy) (pandas-dev#48859) * Backport PR pandas-dev#48790 on branch 1.5.x (Created icons for dark theme) (pandas-dev#48875) Backport PR pandas-dev#48790: Created icons for dark theme Co-authored-by: Lorenzo Vainigli <lorenzovngl@gmail.com> * Backport PR pandas-dev#48805 on branch 1.5.x (Added padding and fixed columns for sponsor logos in mobile view) (pandas-dev#48874) Backport PR pandas-dev#48805: Added padding and fixed columns for sponsor logos in mobile view Co-authored-by: Amay Patel <92037532+amay-patel@users.noreply.github.com> * Backport PR pandas-dev#48866 on branch 1.5.x (REGR: replace replacing wrong values with inplace and datetime) (pandas-dev#48872) Backport PR pandas-dev#48866: REGR: replace replacing wrong values with inplace and datetime Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48853 on branch 1.5.x (REGR: Avoid unnecessary warning when setting empty dataframe) (pandas-dev#48873) Backport PR pandas-dev#48853: REGR: Avoid unnecessary warning when setting empty dataframe Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48833 on branch 1.5.x (BUG: ArrowExtensionArray compared to invalid object not raising) (pandas-dev#48878) Backport PR pandas-dev#48833: BUG: ArrowExtensionArray compared to invalid object not raising Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48686 on branch 1.5.x (BUG: to_datetime(tz_mix, utc=True) converts to UTC) (pandas-dev#48882) Backport PR pandas-dev#48686: BUG: to_datetime(tz_mix, utc=True) converts to UTC Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48736 on branch 1.5.x ( BUG: AttributeError: 'function' object has no attribute 'currentframe') (pandas-dev#48887) * Backport PR pandas-dev#48797 on branch 1.5.x (REGR: fix df.apply with keyword non-zero axis) (pandas-dev#48886) REGR: fix df.apply with keyword non-zero axis (pandas-dev#48797) Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com> * add pandas[foo] throughout to explain the optional extras process Also `Recommended dependencies` to `Performance dependencies (recommended)` * extend optional_extra`all` to cover tests and redundant packages * add tzdata from pandas 1.5.0 * move summary to latest whatsnew doc Code is now stable and updated. Attempt a port from doc/source/whatsnew/v1.5.0.rst over to latest document doc/source/whatsnew/v1.5.1.rst. * explicitly pair packages to optional_extra in installs * fix sphinx errors in install.rst extra padding on column + empty return before table * add: pytest-asyncio>=0.19.0 pytest-asyncio>=0.19.0 Closes pandas-dev#48361. No version guidance available so went for latest. No-one should be messing around with tests without modern installs. * fsspec note * repin pytest-asyncio>=0.17.0 matches pytest-asyncio>=0.17.0 in ci/deps/actions-38-minimum_versions.yaml * move summary from whatsnew/v1.5.1 to v1.6.0 plus add (:issue:`48361`) resolution note * linting double-backtick ``test`` underline tilde match title length * Add `clipboard` as optional extra * Review comments - fss optional extra - better numba description * remove unneeded comment * fix: leave 1.5.1.rst unchanged by this PR * Update doc/source/whatsnew/v1.5.1.rst Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * pyarrow bump 1.0.1 -> 6.0.0 Co-authored-by: Matthew Roeschke <emailformattr@gmail.com> Co-authored-by: Simon Hawkins <simonjayhawkins@gmail.com> Co-authored-by: MeeseeksMachine <39504233+meeseeksmachine@users.noreply.github.com> Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> Co-authored-by: Marc Garcia <garcia.marc@gmail.com> Co-authored-by: Kevin Sheppard <bashtage@users.noreply.github.com> Co-authored-by: RaphSku <45042665+RaphSku@users.noreply.github.com> Co-authored-by: Jonas Haag <jonas@lophus.org> Co-authored-by: Janosh Riebesell <janosh.riebesell@gmail.com> Co-authored-by: Pandas Development Team <pandas-dev@python.org> Co-authored-by: Torsten Wörtwein <twoertwein@users.noreply.github.com> Co-authored-by: Deepak Sirohiwal <38135521+deepaksirohiwal@users.noreply.github.com> Co-authored-by: Marco Edward Gorelli <33491632+MarcoGorelli@users.noreply.github.com> Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com> Co-authored-by: Dennis Chukwunta <chuksmcdennis@yahoo.com> Co-authored-by: Lorenzo Vainigli <lorenzovngl@gmail.com> Co-authored-by: Amay Patel <92037532+amay-patel@users.noreply.github.com>
… for concat with all-NA array (pandas-dev#47762)
…dev#47336) * add-recommended-dependencies-as-extras_require-in-setup.cfg See issue pandas-dev#47335. pandas-dev#47335 recommended dependencies should have package mgmt facilitated through pandas. This will make mgmt of pandas in production docker environments a lot simpler. * Update v1.4.3.rst * double backtick code in rst * rebundle under extras_require `recommended` * [options.extras_require] bundled dependencies by featureset see pandas-dev#39164 for discussion. * note: [options.extras_require] bundled dependencies by featureset * Update setup.cfg rollback numba from recommended. This would necessitate and update to documentation that requires broad agreement from pd-dev-core team that would slow down the overall PR * add adlfs for azure to `access data in cloud` see pandas-dev#39164 for discussion. 0.6.0 might be an overly restrictive version, but it is compatible * fix extras_require: PyTables is actually `tables` on PyPi * Update setup.cfg * add `all` option to [options.extras_require] * moved changelog to 1.4.4 as 1.4.3 released while this PR was stalled * Updated to 1.5.0 compliance * simplify sql option names * extras rename: recommended -> performance * remove azure support is currently unofficial as of 1.5.0 * align with actions-38-minimum_versions.yaml add specific installs and, where required, missing install documentation for - odfpy - pyreadstat - compression options * Pandas -> pandas in doc Co-authored-by: Matthew Roeschke <emailformattr@gmail.com> * extras rename: s3 -> aws see pandas-dev#47336 (comment) * extras rename: table -> output_formatting to be more general in case of future changes * bug: `>=` not `=` * Apply suggestions from code review Co-authored-by: Simon Hawkins <simonjayhawkins@gmail.com> * align 1.5.0.rst to latest extras_require updates * 1.5.0.rst example updated to use valid extras * add optional dep mgmt instructions to install.rst * lint scipy optional import Co-authored-by: Matthew Roeschke <emailformattr@gmail.com> * Apply suggestions from code review * detailed extras guidance in install.rst - updated numbas to a full recommended dependency with a promotional bullet point like bottleneck and numexpr - clarified the extra to use for each set of optional dependencies - made xml an optional extra, because is does have usage outside of read_html. * _optional.py note to keep track of setup.cfg * bug: indent after bullet in install.rst * remove numba from computation extra. * Backport PR pandas-dev#48197 on branch 1.5.x (DOC: Cleanup 1.5 whatsnew) (pandas-dev#48228) Backport PR pandas-dev#48197: DOC: Cleanup 1.5 whatsnew Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48232 on branch 1.5.x (CI: Ensure jobs run on 1.5.x branch) (pandas-dev#48235) Backport PR pandas-dev#48232: CI: Ensure jobs run on 1.5.x branch Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48180 on branch 1.5.x (CI: Switch to large for circleci) (pandas-dev#48251) Backport PR pandas-dev#48180: CI: Switch to large for circleci Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48245 on branch 1.5.x (CI: Skip test_round_sanity tests due to failures) (pandas-dev#48257) Backport PR pandas-dev#48245: CI: Skip test_round_sanity tests due to failures Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48240 on branch 1.5.x (Fix mypy erroring on backport branches) (pandas-dev#48259) Backport PR pandas-dev#48240: Fix mypy erroring on backport branches Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48215 on branch 1.5.x (REGR: properly update DataFrame cache in Series.__setitem__) (pandas-dev#48268) Backport PR pandas-dev#48215: REGR: properly update DataFrame cache in Series.__setitem__ Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR pandas-dev#48272 on branch 1.5.x (CI: Require s3fs greater than minumum version in builds) (pandas-dev#48276) Backport PR pandas-dev#48272: CI: Require s3fs greater than minumum version in builds Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48299 on branch 1.5.x (Bump s3fs to 2021.08.00) (pandas-dev#48305) Backport PR pandas-dev#48299: Bump s3fs to 2021.08.00 Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48027 on branch 1.5.x (ENH: Support masks in groupby prod) (pandas-dev#48302) Backport PR pandas-dev#48027: ENH: Support masks in groupby prod Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#47762 on branch 1.5.x (REGR: preserve reindexed array object (instead of creating new array) for concat with all-NA array) (pandas-dev#48309) Backport PR pandas-dev#47762: REGR: preserve reindexed array object (instead of creating new array) for concat with all-NA array Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR pandas-dev#48246 on branch 1.5.x (REGR: iloc not possible for sparse DataFrame) (pandas-dev#48311) Backport PR pandas-dev#48246: REGR: iloc not possible for sparse DataFrame Co-authored-by: Simon Hawkins <simonjayhawkins@gmail.com> * Backport PR pandas-dev#48314 on branch 1.5.x (DOC: v1.4.4 release date and tidy up release notes) (pandas-dev#48320) Backport PR pandas-dev#48314: DOC: v1.4.4 release date and tidy up release notes Co-authored-by: Simon Hawkins <simonjayhawkins@gmail.com> * Backport PR pandas-dev#48301 on branch 1.5.x (DEPR: Deprecate positional arguments in pivot) (pandas-dev#48326) Backport PR pandas-dev#48301: DEPR: Deprecate positional arguments in pivot Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48214 on branch 1.5.x (WEB: Removing links to pdf version of the docs from web and docs) (pandas-dev#48242) Backport PR pandas-dev#48214: WEB: Removing links to pdf version of the docs from web and docs * Backport PR pandas-dev#48159 on branch 1.5.x (TST: Fix interchange/plotting/groupby test warnings) (pandas-dev#48279) Backport PR pandas-dev#48159: TST: Fix interchange/plotting/groupby test warnings Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48324 on branch 1.5.x (BUG: Add note in whatsnew for DataFrame.at behavior change) (pandas-dev#48345) Backport PR pandas-dev#48324: BUG: Add note in whatsnew for DataFrame.at behavior change Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR pandas-dev#48336 on branch 1.5.x (DOC: Add whatsnew note for pandas-dev#45404) (pandas-dev#48341) Backport PR pandas-dev#48336: DOC: Add whatsnew note for pandas-dev#45404 Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48254 on branch 1.5.x (REF: avoid FutureWarning about using deprecates loc.__setitem__ non-inplace usage) (pandas-dev#48353) Backport PR pandas-dev#48254: REF: avoid FutureWarning about using deprecates loc.__setitem__ non-inplace usage Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR pandas-dev#48334 on branch 1.5.x (BUG: read_html(extract_links=all) with no header) (pandas-dev#48350) Backport PR pandas-dev#48334: BUG: read_html(extract_links=all) with no header Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48265 on branch 1.5.x (CI: Setting up ssh key to upload prod docs) (pandas-dev#48370) Backport PR pandas-dev#48265: CI: Setting up ssh key to upload prod docs Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR pandas-dev#48381 on branch 1.5.x (CI: Pin mambaforge image) (pandas-dev#48401) Backport PR pandas-dev#48381: CI: Pin mambaforge image Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48229 on branch 1.5.x (TST: Test Nullable int floordiv by 0) (pandas-dev#48413) Backport PR pandas-dev#48229: TST: Test Nullable int floordiv by 0 Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48414 on branch 1.5.x (DOC: Add deprecation to is_categorical) (pandas-dev#48418) Backport PR pandas-dev#48414: DOC: Add deprecation to is_categorical Co-authored-by: Kevin Sheppard <bashtage@users.noreply.github.com> * Backport PR pandas-dev#48264 on branch 1.5.x (BUG: ArrowExtensionArray._from_* accepts pyarrow arrays) (pandas-dev#48422) * Backport PR pandas-dev#48264: BUG: ArrowExtensionArray._from_* accepts pyarrow arrays * Add missing import Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48411 on branch 1.5.x (REGR: get_loc for ExtensionEngine not returning bool indexer for na) (pandas-dev#48430) Backport PR pandas-dev#48411: REGR: get_loc for ExtensionEngine not returning bool indexer for na Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48419 on branch 1.5.x (BUG: ensure to return writable buffer in __dataframe__ interchange for categorical column) (pandas-dev#48441) Backport PR pandas-dev#48419: BUG: ensure to return writable buffer in __dataframe__ interchange for categorical column Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR pandas-dev#48444 on branch 1.5.x (CI: Pin ipython version) (pandas-dev#48449) Backport PR pandas-dev#48444: CI: Pin ipython version Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48380 on branch 1.5.x (DOC: Clarify that objects dtype takes precedence in where) (pandas-dev#48445) * Backport PR pandas-dev#48380: DOC: Clarify that objects dtype takes precedence in where * Update generic.py Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Manual Backport PR pandas-dev#48427 on branch 1.5.x (BLD: Refactor Dockerfile to not install dev enviornment on base) (pandas-dev#48450) Backport PR pandas-dev#48427: BLD: Refactor Dockerfile to not install dev enviornment on base * Backport PR pandas-dev#48426 on branch 1.5.x (BUG: Column.size should be a method) (pandas-dev#48465) Backport PR pandas-dev#48426: BUG: Column.size should be a method Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48398 on branch 1.5.x (WARN: Avoid FutureWarnings in tests) (pandas-dev#48420) * Backport PR pandas-dev#48398: WARN: Avoid FutureWarnings in tests * Update Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR pandas-dev#48416 on branch 1.5.x (REF: ensure to apply suffixes before concat step in merge code) (pandas-dev#48470) Backport PR pandas-dev#48416: REF: ensure to apply suffixes before concat step in merge code Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR pandas-dev#48354 on branch 1.5.x (CI: Bump timeout to 180 minutes) (pandas-dev#48474) Backport PR pandas-dev#48354: CI: Bump timeout to 180 minutes Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48472 on branch 1.5.x (PERF: keep using ObjectEngine for ExtensionArrays for 1.5) (pandas-dev#48486) Backport PR pandas-dev#48472: PERF: keep using ObjectEngine for ExtensionArrays for 1.5 Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR pandas-dev#48473 on branch 1.5.x (REGR: .describe on unsigned dtypes results in object) (pandas-dev#48501) Backport PR pandas-dev#48473: REGR: .describe on unsigned dtypes results in object Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR pandas-dev#48443 on branch 1.5.x (BUG: Fix pyarrow groupby tests) (pandas-dev#48494) * BUG: Fix pyarrow groupby tests (pandas-dev#48443) # Conflicts: # pandas/tests/extension/test_arrow.py * CI: Fix failing tests (pandas-dev#48493) Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR pandas-dev#48490 on branch 1.5.x (CI: Use -j1 for python-dev build to avoid flaky build error) (pandas-dev#48517) Backport PR pandas-dev#48490: CI: Use -j1 for python-dev build to avoid flaky build error Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Manual Backport PR pandas-dev#48428 on branch 1.5.x (BUG/TST: fix a bunch of arraymanager+pyarrow tests) (pandas-dev#48518) Backport PR pandas-dev#48428: BUG/TST: fix a bunch of arraymanager+pyarrow tests Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR pandas-dev#48525 on branch 1.5.x (CI: Fix py311 builds different exception message) (pandas-dev#48529) Backport PR pandas-dev#48525: CI: Fix py311 builds different exception message Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48489 on branch 1.5.x (BUG: fix test_arrow.py tests) (pandas-dev#48532) Backport PR pandas-dev#48489: BUG: fix test_arrow.py tests Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR pandas-dev#48543 on branch 1.5.x (DOC: Update footer and include OVH) (pandas-dev#48548) Backport PR pandas-dev#48543: DOC: Update footer and include OVH Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Manual Backport PR pandas-dev#48417 on branch 1.5.x (Revert set_index inplace and copy keyword changes) (pandas-dev#48552) Backport PR pandas-dev#48417: Revert set_index inplace and copy keyword changes Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> * Backport PR pandas-dev#48550 on branch 1.5.x (TST: remove 2D tests irrelevant for pyarrow) (pandas-dev#48554) Backport PR pandas-dev#48550: TST: remove 2D tests irrelevant for pyarrow Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR pandas-dev#48556 on branch 1.5.x (DOC: Fix docs footer) (pandas-dev#48558) Backport PR pandas-dev#48556: DOC: Fix docs footer Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR pandas-dev#48562 on branch 1.5.x (TST: Testing that no warnings are emitted and that inplace fillna produces the correct result (GH48480)) (pandas-dev#48564) Backport PR pandas-dev#48562: TST: Testing that no warnings are emitted and that inplace fillna produces the correct result (GH48480) Co-authored-by: RaphSku <45042665+RaphSku@users.noreply.github.com> * Backport PR pandas-dev#48563 on branch 1.5.x (DOC: Fix read_sas 1.5 release notes) (pandas-dev#48565) Backport PR pandas-dev#48563: DOC: Fix read_sas 1.5 release notes Co-authored-by: Jonas Haag <jonas@lophus.org> * Backport PR pandas-dev#48539 on branch 1.5.x (REGR: groupby doesn't identify null values when sort=False) (pandas-dev#48568) Backport PR pandas-dev#48539: REGR: groupby doesn't identify null values when sort=False Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR pandas-dev#48555 on branch 1.5.x (DEPR: Series.astype(np.datetime64)) (pandas-dev#48569) Backport PR pandas-dev#48555: DEPR: Series.astype(np.datetime64) Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> * Backport PR pandas-dev#48557 on branch 1.5.x (WEB: Add new footer to web) (pandas-dev#48571) Backport PR pandas-dev#48557: WEB: Add new footer to web Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR pandas-dev#48285 on branch 1.5.x (WEB: Unpin pydata sphinx theme) (pandas-dev#48585) Backport PR pandas-dev#48285: WEB: Unpin pydata sphinx theme Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48572 on branch 1.5.x (DOC: Fixing styles for the dark theme) (pandas-dev#48584) Backport PR pandas-dev#48572: DOC: Fixing styles for the dark theme Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR pandas-dev#48397 on branch 1.5.x (WARN: Remove false positive warning for iloc inplaceness) (pandas-dev#48583) Backport PR pandas-dev#48397: WARN: Remove false positive warning for iloc inplaceness Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48587 on branch 1.5.x (Fix `series.str.startswith(tuple)`) (pandas-dev#48593) Backport PR pandas-dev#48587: Fix `series.str.startswith(tuple)` Co-authored-by: Janosh Riebesell <janosh.riebesell@gmail.com> * Backport PR pandas-dev#48601 on branch 1.5.x (CI: Fix matplolib release issues) (pandas-dev#48617) Backport PR pandas-dev#48601: CI: Fix matplolib release issues Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48623 on branch 1.5.x (REGR/DOC: Docs left navbar broke) (pandas-dev#48625) Backport PR pandas-dev#48623: REGR/DOC: Docs left navbar broke Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR pandas-dev#48619 on branch 1.5.x (REGR: Loc.setitem with enlargement raises for nested data) (pandas-dev#48629) Backport PR pandas-dev#48619: REGR: Loc.setitem with enlargement raises for nested data Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48627 on branch 1.5.x (DOC: Last changes to release notes for 1.5.0 release) (pandas-dev#48630) Backport PR pandas-dev#48627: DOC: Last changes to release notes for 1.5.0 release Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * RLS: 1.5.0 * Backport PR pandas-dev#48642 on branch 1.5.x (DOC: Add release notes for 1.5.1) (pandas-dev#48647) Backport PR pandas-dev#48642: DOC: Add release notes for 1.5.1 Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48639 on branch 1.5.x (CI: Fix directory name for published prod docs) (pandas-dev#48648) Backport PR pandas-dev#48639: CI: Fix directory name for published prod docs Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR pandas-dev#48651 on branch 1.5.x (REGR: TextIOWrapper raising an error in read_csv) (pandas-dev#48666) Backport PR pandas-dev#48651: REGR: TextIOWrapper raising an error in read_csv Co-authored-by: Torsten Wörtwein <twoertwein@users.noreply.github.com> * Backport PR pandas-dev#48599 on branch 1.5.x (DOC: Add deprecation infos to deprecated functions) (pandas-dev#48690) Backport PR pandas-dev#48599: DOC: Add deprecation infos to deprecated functions Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48620 on branch 1.5.x (REGR: Performance decrease in factorize) (pandas-dev#48710) Backport PR pandas-dev#48620: REGR: Performance decrease in factorize Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR pandas-dev#48711 on branch 1.5.x (REGR: Regression in DataFrame.loc when setting df with all True indexer) (pandas-dev#48717) Backport PR pandas-dev#48711: REGR: Regression in DataFrame.loc when setting df with all True indexer Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48696 on branch 1.5.x (REGR: to_hdf raising AssertionError with boolean index) (pandas-dev#48716) Backport PR pandas-dev#48696: REGR: to_hdf raising AssertionError with boolean index Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48734 on branch 1.5.x (REGR: Raise on invalid colormap for scatter plot) (pandas-dev#48744) Backport PR pandas-dev#48734: REGR: Raise on invalid colormap for scatter plot Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48713 on branch 1.5.x (BUG: pivot_table raising Future Warning with datetime column as index) (pandas-dev#48742) Backport PR pandas-dev#48713: BUG: pivot_table raising Future Warning with datetime column as index Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48703 on branch 1.5.x (Added theme_switcher ) (pandas-dev#48741) Backport PR pandas-dev#48703: Added theme_switcher Co-authored-by: Deepak Sirohiwal <38135521+deepaksirohiwal@users.noreply.github.com> * Backport PR pandas-dev#48697 on branch 1.5.x (REGR: None converted to NaN when enlarging Series) (pandas-dev#48745) Backport PR pandas-dev#48697: REGR: None converted to NaN when enlarging Series Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48702 on branch 1.5.x (REGR: dropna affects observed in groupby) (pandas-dev#48750) Backport PR pandas-dev#48702: REGR: dropna affects observed in groupby Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> * Backport PR pandas-dev#48782 on branch 1.5.x (REGR: describe raising when result contains NA) (pandas-dev#48793) Backport PR pandas-dev#48782: REGR: describe raising when result contains NA Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48662 on branch 1.5.x (BUG: Series.getitem not falling back to positional for bool index) (pandas-dev#48799) Backport PR pandas-dev#48662: BUG: Series.getitem not falling back to positional for bool index Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48751 on branch 1.5.x (WEB: Update link to datapythonista blog url) (pandas-dev#48798) Backport PR pandas-dev#48751: WEB: Update link to datapythonista blog url Co-authored-by: Marc Garcia <garcia.marc@gmail.com> * Backport PR pandas-dev#48608 on branch 1.5.x (REGR: assert_index_equal raising with non matching pd.NA) (pandas-dev#48800) * Backport PR pandas-dev#48608: REGR: assert_index_equal raising with non matching pd.NA Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48785 on branch 1.5.x (BUG: still emitting unnecessary FutureWarning in DataFrame.sort_values with sparse columns) (pandas-dev#48807) Backport PR pandas-dev#48785: BUG: still emitting unnecessary FutureWarning in DataFrame.sort_values with sparse columns Co-authored-by: Marco Edward Gorelli <33491632+MarcoGorelli@users.noreply.github.com> * Backport PR pandas-dev#48693 on branch 1.5.x (ENH: Make deprecate_nonkeyword_arguments alter function signature) (pandas-dev#48795) Backport PR pandas-dev#48693: ENH: Make deprecate_nonkeyword_arguments alter function signature Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com> * Backport PR pandas-dev#48579 on branch 1.5.x (BUG: Fix calling groupBy(...).apply(func) on an empty dataframe invokes func) (pandas-dev#48817) BUG: Fix calling groupBy(...).apply(func) on an empty dataframe invokes func (pandas-dev#48579) (cherry picked from commit 8b0ad71) Co-authored-by: Dennis Chukwunta <chuksmcdennis@yahoo.com> * Backport PR pandas-dev#48760 on branch 1.5.x (REGR: groupby.size with axis=1 doesn't return a Series) (pandas-dev#48825) * Backport PR pandas-dev#48820 on branch 1.5.x (BUG: to_datetime(format='...%f') parses nanoseconds) (pandas-dev#48860) BUG: to_datetime(format='...%f') parses nanoseconds (pandas-dev#48820) Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48850 on branch 1.5.x (TYP: Fix typing errors caused by new numpy) (pandas-dev#48859) * Backport PR pandas-dev#48790 on branch 1.5.x (Created icons for dark theme) (pandas-dev#48875) Backport PR pandas-dev#48790: Created icons for dark theme Co-authored-by: Lorenzo Vainigli <lorenzovngl@gmail.com> * Backport PR pandas-dev#48805 on branch 1.5.x (Added padding and fixed columns for sponsor logos in mobile view) (pandas-dev#48874) Backport PR pandas-dev#48805: Added padding and fixed columns for sponsor logos in mobile view Co-authored-by: Amay Patel <92037532+amay-patel@users.noreply.github.com> * Backport PR pandas-dev#48866 on branch 1.5.x (REGR: replace replacing wrong values with inplace and datetime) (pandas-dev#48872) Backport PR pandas-dev#48866: REGR: replace replacing wrong values with inplace and datetime Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48853 on branch 1.5.x (REGR: Avoid unnecessary warning when setting empty dataframe) (pandas-dev#48873) Backport PR pandas-dev#48853: REGR: Avoid unnecessary warning when setting empty dataframe Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> * Backport PR pandas-dev#48833 on branch 1.5.x (BUG: ArrowExtensionArray compared to invalid object not raising) (pandas-dev#48878) Backport PR pandas-dev#48833: BUG: ArrowExtensionArray compared to invalid object not raising Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48686 on branch 1.5.x (BUG: to_datetime(tz_mix, utc=True) converts to UTC) (pandas-dev#48882) Backport PR pandas-dev#48686: BUG: to_datetime(tz_mix, utc=True) converts to UTC Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Backport PR pandas-dev#48736 on branch 1.5.x ( BUG: AttributeError: 'function' object has no attribute 'currentframe') (pandas-dev#48887) * Backport PR pandas-dev#48797 on branch 1.5.x (REGR: fix df.apply with keyword non-zero axis) (pandas-dev#48886) REGR: fix df.apply with keyword non-zero axis (pandas-dev#48797) Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com> * add pandas[foo] throughout to explain the optional extras process Also `Recommended dependencies` to `Performance dependencies (recommended)` * extend optional_extra`all` to cover tests and redundant packages * add tzdata from pandas 1.5.0 * move summary to latest whatsnew doc Code is now stable and updated. Attempt a port from doc/source/whatsnew/v1.5.0.rst over to latest document doc/source/whatsnew/v1.5.1.rst. * explicitly pair packages to optional_extra in installs * fix sphinx errors in install.rst extra padding on column + empty return before table * add: pytest-asyncio>=0.19.0 pytest-asyncio>=0.19.0 Closes pandas-dev#48361. No version guidance available so went for latest. No-one should be messing around with tests without modern installs. * fsspec note * repin pytest-asyncio>=0.17.0 matches pytest-asyncio>=0.17.0 in ci/deps/actions-38-minimum_versions.yaml * move summary from whatsnew/v1.5.1 to v1.6.0 plus add (:issue:`48361`) resolution note * linting double-backtick ``test`` underline tilde match title length * Add `clipboard` as optional extra * Review comments - fss optional extra - better numba description * remove unneeded comment * fix: leave 1.5.1.rst unchanged by this PR * Update doc/source/whatsnew/v1.5.1.rst Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * pyarrow bump 1.0.1 -> 6.0.0 Co-authored-by: Matthew Roeschke <emailformattr@gmail.com> Co-authored-by: Simon Hawkins <simonjayhawkins@gmail.com> Co-authored-by: MeeseeksMachine <39504233+meeseeksmachine@users.noreply.github.com> Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> Co-authored-by: Patrick Hoefler <61934744+phofl@users.noreply.github.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> Co-authored-by: Marc Garcia <garcia.marc@gmail.com> Co-authored-by: Kevin Sheppard <bashtage@users.noreply.github.com> Co-authored-by: RaphSku <45042665+RaphSku@users.noreply.github.com> Co-authored-by: Jonas Haag <jonas@lophus.org> Co-authored-by: Janosh Riebesell <janosh.riebesell@gmail.com> Co-authored-by: Pandas Development Team <pandas-dev@python.org> Co-authored-by: Torsten Wörtwein <twoertwein@users.noreply.github.com> Co-authored-by: Deepak Sirohiwal <38135521+deepaksirohiwal@users.noreply.github.com> Co-authored-by: Marco Edward Gorelli <33491632+MarcoGorelli@users.noreply.github.com> Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com> Co-authored-by: Dennis Chukwunta <chuksmcdennis@yahoo.com> Co-authored-by: Lorenzo Vainigli <lorenzovngl@gmail.com> Co-authored-by: Amay Patel <92037532+amay-patel@users.noreply.github.com>
Originally reported in GeoPandas: geopandas/geopandas#2493
This worked in pandas 1.3.5, was then originally "broken" by #43043 (as an unintended side effect of some of the (proper) changes in that PR) somewhere between 1.3 and 1.4.0 but got fixed again before the final 1.4.0 because of subsequent refactoring in the internal concat code. Some of those refactors were then reverted in 1.4.3 for being able to revert the all-NA change (#47372), surfacing the "bug" again.
This is not necessarily strictly a "bug" in pandas (I also don't know if we make any guarantees about preserving array objects, let alone its attributes), but I think this is a decent change nevertheless.
Currently, when concatting/merging a column with all-NA data, we create the "empty" (all-NA) array for the result from scratch. While if there is no dtype change, and so the original column already has the correct dtype, we can actually just reindex that array instead of creating a new all-NA array. For builtin dtypes that might not matter much, but for custom ExtensionArrays that avoids going through some complex code to create this new all-NA array:
pandas/pandas/core/internals/concat.py
Lines 479 to 488 in fc68a9a
In GeoPandas, our ExtensionArray holds some optional attribute information (eg the coordinate reference system, a spatial index, ..) that is not part of the dtype. And so in that case, reindexing the original array lets geopandas preserve that information (if appropriate), while if creating an empty array from scratch with the code above, this information is definitely lost.
Apart from the two-line actual code change, this PR adds a new, minimal test ExtensionArray that exhibits this behaviour of having an attribute, to be able to test this without dependency on GeoPandas.