Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Appending to DataFrame removes the timezone-awareness #30238

Closed
MarcoGorelli opened this issue Dec 12, 2019 · 2 comments · Fixed by #30277
Closed

Appending to DataFrame removes the timezone-awareness #30238

MarcoGorelli opened this issue Dec 12, 2019 · 2 comments · Fixed by #30277
Labels
Reshaping Concat, Merge/Join, Stack/Unstack, Explode Timezones Timezone data dtype
Milestone
1.0.0

Comments

@MarcoGorelli
Copy link
Member

Code Sample, a copy-pastable example if possible

import pytz

import pandas as pd

df = pd.DataFrame(
    {"a": [pd.Timestamp("2019-07-19 07:04:57+0100", tz=pytz.FixedOffset(60))]}
)
df.append(df.iloc[0])

Problem description

This returns:

|    | a                         |
|---:|:--------------------------|
|  0 | 2019-07-19 07:04:57+01:00 |
|  0 | 2019-07-19 06:04:57       |

However, I was expecting

|    | a                         |
|---:|:--------------------------|
|  0 | 2019-07-19 07:04:57+01:00 |
|  0 | 2019-07-19 07:04:57+01:00 |

which indeed is what happens if I instead write

df.append(df)

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.5.final.0
python-bits : 64
OS : Linux
OS-release : 4.15.0-72-generic
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8

pandas : 0.25.3
numpy : 1.17.4
pytz : 2019.3
dateutil : 2.8.0
pip : 19.3.1
setuptools : 41.6.0.post20191030
Cython : 0.29.14
pytest : 5.3.1
hypothesis : None
sphinx : 2.2.1
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10.3
IPython : 7.9.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.1.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.15.1
pytables : None
s3fs : 0.4.0
scipy : 1.3.3
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
@mroeschke
Copy link
Member

Thanks for the report. Can reproduce on master. Investigations and PRs welcome!

@mroeschke mroeschke added Reshaping Concat, Merge/Join, Stack/Unstack, Explode Timezones Timezone data dtype labels Dec 12, 2019
@MarcoGorelli
Copy link
Member Author

MarcoGorelli commented Dec 12, 2019 •
edited
Loading

OK, I'll have a go at this on Saturday!

UPDATE

in frame.py the problematic line seems to be

            other = DataFrame(
                other.values.reshape((1, len(other))),
                index=index,
                columns=combined_columns,
            )

will narrow this down further (and hopefully fix it) later

@MarcoGorelli MarcoGorelli mentioned this issue Dec 14, 2019
Merged
5 tasks
@MarcoGorelli MarcoGorelli mentioned this issue Dec 24, 2019
Closed
@jreback jreback added this to the 1.0 milestone Dec 24, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Reshaping Concat, Merge/Join, Stack/Unstack, Explode Timezones Timezone data dtype
Projects
None yet
Milestone
1.0.0
Development

Successfully merging a pull request may close this issue.

[bug] don't remove timezone-awareness when using the method from Dat… MarcoGorelli/pandas
3 participants
@jreback @mroeschke @MarcoGorelli