Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Shifting a datetime column with timezone after groupby loses the timezone. #19995

Closed
borispf opened this issue Mar 5, 2018 · 3 comments · Fixed by #27992
Closed

Shifting a datetime column with timezone after groupby loses the timezone. #19995

borispf opened this issue Mar 5, 2018 · 3 comments · Fixed by #27992
Labels
Duplicate Report Duplicate issue or pull request Groupby Timezones Timezone data dtype
Milestone
0.25.2

Comments

@borispf
Copy link

borispf commented Mar 5, 2018

Code Sample, a copy-pastable example if possible

weeks = pd.Series(pd.date_range('2018-01', '2018-02', freq='7D', tz='America/New_York'))
week_ago = weeks.groupby([1, 1, 1, 1, 1]).shift()
print(week_ago)

Outputs:

0                   NaT
1   2018-01-01 05:00:00
2   2018-01-08 05:00:00
3   2018-01-15 05:00:00
4   2018-01-22 05:00:00
dtype: datetime64[ns]

Problem description

Shifting a groupby'd datetime column removes the timezone. It's not mentioned in the documentation so either the timezone should be preserved or the documentation should mention that timezone is removed (by converting to UTC it seems).

I had a quick look and couldn't find another issue for this. I glanced at the GroupBy shift implementation and my guess is that it's due to cython/numpy not supporting timezones.

I'm happy to work on a fix if someone says what the preferred solution is.

Expected Output

0                         NaT
1   2018-01-01 00:00:00-05:00
2   2018-01-08 00:00:00-05:00
3   2018-01-15 00:00:00-05:00
4   2018-01-22 00:00:00-05:00
dtype: datetime64[ns, America/New_York]

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Linux
OS-release: 3.10.0-693.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.21.0
pytest: 3.4.1
pip: 9.0.1
setuptools: 38.5.1
Cython: None
numpy: 1.13.3
scipy: 1.0.0
pyarrow: 0.8.0
xarray: None
IPython: 6.2.1
sphinx: None
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2018.3
blosc: None
bottleneck: 1.2.1
tables: None
numexpr: 2.6.4
feather: 0.4.0
matplotlib: 2.1.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.9999999
sqlalchemy: 1.2.1
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: 0.1.4
pandas_gbq: None
pandas_datareader: None
@jreback
Copy link
Contributor

jreback commented Mar 7, 2018

duplicate of #15884

@jreback jreback closed this as completed Mar 7, 2018
@jreback jreback added Groupby Duplicate Report Duplicate issue or pull request Timezones Timezone data dtype labels Mar 7, 2018
@jreback jreback added this to the No action milestone Mar 7, 2018
@akdor1154
Copy link

@jreback the linked issue is closed as fixed but shift still has this behaviour (only first and last were fixed), can this be reopened?

@TomAugspurger
Copy link
Contributor

Thanks @akdor1154.

@TomAugspurger TomAugspurger reopened this Aug 12, 2019
@TomAugspurger TomAugspurger modified the milestones: No action, Contributions Welcome Aug 12, 2019
noritada added a commit to noritada/pandas that referenced this issue Sep 9, 2019
@noritada
Document additionally an existing issue number pandas-dev#19995 for G…
6eb0d28 
…roupBy.shift
@noritada noritada mentioned this issue Sep 9, 2019
Merged
5 tasks
@jreback jreback modified the milestones: Contributions Welcome, 0.25.2 Sep 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Duplicate Report Duplicate issue or pull request Groupby Timezones Timezone data dtype
Projects
None yet
Milestone
0.25.2
Development

Successfully merging a pull request may close this issue.

BUG/TST: fix and test for timezone drop in GroupBy.shift/bfill/ffill noritada/pandas
4 participants
@borispf @jreback @TomAugspurger @akdor1154