Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

numpy.datetime64 precision cause dict indexing failure in index_data.py #1806

Open
GeorgeGuo1202 opened this issue Jun 8, 2024 · 2 comments
Open

numpy.datetime64 precision cause dict indexing failure in index_data.py #1806

GeorgeGuo1202 opened this issue Jun 8, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@GeorgeGuo1202
Copy link

🐛 Bug Description

ns precision numpy.datetime64 is equal to second precision
however when doing indexing of a dict keys, it will cause failure

numpy.datetime64('2017-01-04T00:00:00.000000000')==numpy.datetime64('2017-01-04T00:00:00')
Out[24]: True
self.index_map[numpy.datetime64('2017-01-04T00:00:00.000000000')]
Out[25]: 1
self.index_map[numpy.datetime64('2017-01-04T00:00:00')]
Traceback (most recent call last):
File "F:\work\env\py311\Lib\site-packages\IPython\core\interactiveshell.py", line 3508, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
self.index_map[numpy.datetime64('2017-01-04T00:00:00')]
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: numpy.datetime64('2017-01-04T00:00:00')

it occurs at qlib/qlib/utils/index_data.py, line 157:
try:
return self.index_map[self._convert_type(item)]
except IndexError as index_e:
raise KeyError(f"{item} can't be found in {self}") from index_e

maybe I did something wrong, hoping for help
@GeorgeGuo1202 GeorgeGuo1202 added the bug Something isn't working label Jun 8, 2024
@SunsetWolf
Copy link
Collaborator

Can you describe, in detail, how this can be surfaced? This will help us to solve the problem.

@akazeakari
Copy link
Contributor

akazeakari commented Jun 17, 2024 •
edited
Loading

Here is a code reproduction based on the issue description. Although the two numpy.datetime64 values are equal, the difference in precision causes a dict indexing failure.

The following code successfully reproduces the issue on Python 3.9.19, pyqlib 0.9.5.99, and numpy 1.23.5.

import numpy as np
from qlib.utils.index_data import Index, SingleData

index = Index([np.datetime64('2017-01-04T00:00:00.000000000'),
               np.datetime64('2017-01-05T00:00:00.000000000'),
               np.datetime64('2017-01-06T00:00:00.000000000')])

data = SingleData([1, 2, 3], index=index)

# print: True
print(np.datetime64('2017-01-04T00:00:00.000000000') == np.datetime64('2017-01-04T00:00:00'))

# print: 0
print(data.index.index_map[np.datetime64('2017-01-04T00:00:00.000000000')])

# False
print(data.index.index_map[np.datetime64('2017-01-04T00:00:00')])

# print: 0
print(data.index.index_map[np.datetime64(np.datetime64('2017-01-04T00:00:00'), 'ns')])

@taozhiwang taozhiwang mentioned this issue Jun 22, 2024
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@SunsetWolf @GeorgeGuo1202 @akazeakari