Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: timezone comparisions are inconsistent, manifesting in bugs in .concat #19281

Merged
merged 4 commits into from
Jan 18, 2018

Conversation

jreback
Copy link
Contributor

@jreback jreback commented Jan 17, 2018

closes #18523
superseded #18596

@jreback jreback added Bug Reshaping Concat, Merge/Join, Stack/Unstack, Explode Timezones Timezone data dtype labels Jan 17, 2018
@jreback jreback added this to the 0.23.0 milestone Jan 17, 2018
@jreback
Copy link
Contributor Author

jreback commented Jan 17, 2018

cc @jschendel
cc @jbrockmendel
cc @MridulS
cc @pganssle
cc @rockg

@jreback jreback changed the title BUG: timezone comparisions are inconsistent, manifesting in bugs in .reset_index BUG: timezone comparisions are inconsistent, manifesting in bugs in .concat Jan 17, 2018
timezones. For example
`<DstTzInfo 'Europe/Paris' LMT+0:09:00 STD>` and
`<DstTzInfo 'Europe/Paris' CET+1:00:00 STD>` are essentially same
timezones but aren't evaluted such, but the string representation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When you say "aren't evaluated as such" are you talking about ==? i.e. start == end is False, but tz_compare(start, end) is True?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TomAugspurger Yes, the reason for this is that pytz generates a separate tzinfo object for each unique offset/tzname combination, and that is what is attached when you call .localize or .normalize. These time zones don't compare equal to one another.

It might be worth noting that this function exists only to add a notion of equality to pytz-style zones that is compatible with the notion of equality expected of tzinfo subclasses.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another thing that might be worth noting is that, as currently written (and probably it would be out of scope to fix this), this will not disambiguate zones that are actually links to other zones.

See the backward file, US/Eastern is a link to America/New_York, for example, but that information is not accessible either from pytz or from the zoneinfo files, as far as I know. Even though pytz.timezone(US/Eastern) and pytz.timezone(America/New_York) always generate the same exact rules, they will be treated as different zones for purposes of tz_compare (which also means concat will merge them and convert to UTC).

@jreback jreback merged commit 88d0238 into pandas-dev:master Jan 18, 2018
@codecov
Copy link

codecov bot commented Jan 18, 2018

Codecov Report

Merging #19281 into master will not change coverage.
The diff coverage is 100%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master   #19281   +/-   ##
=======================================
  Coverage   91.56%   91.56%           
=======================================
  Files         148      148           
  Lines       48874    48874           
=======================================
  Hits        44751    44751           
  Misses       4123     4123
Flag Coverage Δ
#multiple 89.93% <100%> (ø) ⬆️
#single 41.69% <66.66%> (ø) ⬆️
Impacted Files Coverage Δ
pandas/core/indexes/datetimes.py 95.23% <100%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 65d1b62...e95f16f. Read the comment docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Reshaping Concat, Merge/Join, Stack/Unstack, Explode Timezones Timezone data dtype
Projects
None yet
Development

Successfully merging this pull request may close these issues.

pd.concat reset the tz-aware index to UTC
4 participants