BUG: Series constructor with nullable unsigned integer dtype fails with large number #38798

arw2019 · 2020-12-30T06:22:36Z

The following runs fine:

In [30]: s = pd.Series([np.iinfo(np.uint64).max-1, 1], dtype="uint64")

In [31]: s
Out[31]: 
0    18446744073709551614
1                       1
dtype: uint64

but with the equivalent extension dtype the constructor throws:

In [29]: s = pd.Series([np.iinfo(np.uint64).max-1, 1], dtype="UInt64")
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/repos/pandas/pandas/core/arrays/integer.py in safe_cast(values, dtype, copy)
    100     try:
--> 101         return values.astype(dtype, casting="safe", copy=copy)
    102     except TypeError as err:

TypeError: Cannot cast array data from dtype('float64') to dtype('uint64') according to the rule 'safe'

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
<ipython-input-29-7fde9ec5cff9> in <module>
----> 1 s = pd.Series([np.iinfo(np.uint64).max-1, 1], dtype="UInt64")

~/repos/pandas/pandas/core/series.py in __init__(self, data, index, dtype, name, copy, fastpath)
    339                     data = data.copy()
    340             else:
--> 341                 data = sanitize_array(data, index, dtype, copy, raise_cast_failure=True)
    342 
    343                 data = SingleBlockManager.from_array(data, index)

~/repos/pandas/pandas/core/construction.py in sanitize_array(data, index, dtype, copy, raise_cast_failure)
    485 
    486         if dtype is not None:
--> 487             subarr = _try_cast(data, dtype, copy, raise_cast_failure)
    488         else:
    489             subarr = maybe_convert_platform(data)

~/repos/pandas/pandas/core/construction.py in _try_cast(arr, dtype, copy, raise_cast_failure)
    863         # SparseDtype does not
    595         array_type = dtype.construct_array_type()._from_sequence
--> 596         subarr = array_type(arr, dtype=dtype, copy=copy)
    597         return subarr
    598 

~/repos/pandas/pandas/core/arrays/integer.py in _from_sequence(cls, scalars, dtype, copy)
    307         cls, scalars, *, dtype=None, copy: bool = False
    308     ) -> "IntegerArray":
--> 309         values, mask = coerce_to_array(scalars, dtype=dtype, copy=copy)
    310         return IntegerArray(values, mask)
    311 

~/repos/pandas/pandas/core/arrays/integer.py in coerce_to_array(values, dtype, mask, copy)
    204         values = safe_cast(values, dtype, copy=False)
    205     else:
--> 206         values = safe_cast(values, dtype, copy=False)
    207 
    208     return values, mask

~/repos/pandas/pandas/core/arrays/integer.py in safe_cast(values, dtype, copy)
    106             return casted
    107 
--> 108         raise TypeError(
    109             f"cannot safely cast non-equivalent {values.dtype} to {np.dtype(dtype)}"
    110         ) from err

TypeError: cannot safely cast non-equivalent float64 to uint64

jorisvandenbossche · 2020-12-30T07:25:29Z

This is related to #30268, in the sense that it has the same cause: we let numpy do the inference on the list of values first, and in this case of large integers, numpy apparently converts it to a float array (loosing precision) instead of a uint64 array.
So I think the solution will probably be similar. There was a PR #30282, but was blocked by other issues.

arw2019 · 2020-12-30T07:35:10Z

This is related to #30268, in the sense that it has the same cause: we let numpy do the inference on the list of values first, and in this case of large integers, numpy apparently converts it to a float array (loosing precision) instead of a uint64 array.

So I think the solution will probably be similar. There was a PR #30282, but was blocked by other issues.

Thanks for linking these! I'll look into resuscitating 30282, probably by resolving the blocker first

jorisvandenbossche · 2020-12-30T07:38:26Z

I need to look back at the original PR, but I think the blocker is at the moment mainly #38315

arw2019 · 2020-12-30T07:44:15Z

I need to look back at the original PR, but I think the blocker is at the moment mainly #38315

I think that's right

fangchenli · 2021-01-03T16:38:55Z

This test has passed on arm64 #38905

We test on more architectures, so upstream's xfails are not always correct everywhere. On those known to fail: arm64 xfail -> all non-x86 xfail x86 or unconditional strict xfail -> unconditional nonstrict xfail Author: Rebecca N. Palmer <[email protected]> Bug: pandas-dev/pandas#38921, pandas-dev/pandas#38798, pandas-dev/pandas#41740, numpy/numpy#19146 Forwarded: no Gbp-Pq: Name fix_overly_arch_specific_xfails.patch

We test on more architectures, so upstream's xfails are not always correct everywhere. On those known to fail: arm64 xfail -> all non-x86 xfail x86 or unconditional strict xfail -> unconditional nonstrict xfail pandas/tests/window/test_rolling.py also gets an i386 xfail for rounding error that may be x87 excess precision Author: Rebecca N. Palmer <[email protected]> Bug: pandas-dev/pandas#38921, pandas-dev/pandas#38798, pandas-dev/pandas#41740, numpy/numpy#19146 Forwarded: no Gbp-Pq: Name fix_overly_arch_specific_xfails.patch

We test on more architectures, so upstream's xfails are not always correct everywhere. On those known to fail: arm64 xfail -> all non-x86 xfail x86 or unconditional strict xfail -> unconditional nonstrict xfail Author: Rebecca N. Palmer <[email protected]> Bug: pandas-dev/pandas#38921, pandas-dev/pandas#38798, pandas-dev/pandas#41740, numpy/numpy#19146 Forwarded: no Gbp-Pq: Name fix_overly_arch_specific_xfails.patch

arw2019 added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Dec 30, 2020

arw2019 mentioned this issue Dec 30, 2020

ENH: support downcasting of nullable EAs in pd.to_numeric #38746

Merged

5 tasks

jorisvandenbossche added NA - MaskedArrays Related to pd.NA and nullable extension arrays and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Dec 30, 2020

fangchenli mentioned this issue Mar 8, 2021

CI: enable arm64 build on drone.io #39730 #39742

Closed

1 task

jbrockmendel added the Constructors Series/DataFrame/Index/pd.array Constructors label Jun 8, 2021

phofl mentioned this issue Jan 15, 2023

BUG: Series constructor overflowing for UInt64 #50757

Merged

6 tasks

phofl closed this as completed in #50757 Jan 18, 2023

ggold7046 mentioned this issue Aug 10, 2023

Modified doc/make.py to run sphinx-build -b linkcheck #54265

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Series constructor with nullable unsigned integer dtype fails with large number #38798

BUG: Series constructor with nullable unsigned integer dtype fails with large number #38798

arw2019 commented Dec 30, 2020

jorisvandenbossche commented Dec 30, 2020

arw2019 commented Dec 30, 2020

jorisvandenbossche commented Dec 30, 2020

arw2019 commented Dec 30, 2020

fangchenli commented Jan 3, 2021

BUG: Series constructor with nullable unsigned integer dtype fails with large number #38798

BUG: Series constructor with nullable unsigned integer dtype fails with large number #38798

Comments

arw2019 commented Dec 30, 2020

jorisvandenbossche commented Dec 30, 2020

arw2019 commented Dec 30, 2020

jorisvandenbossche commented Dec 30, 2020

arw2019 commented Dec 30, 2020

fangchenli commented Jan 3, 2021