Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

merging two empty dataframes can incur a division by zero #17776

Closed
micklat opened this issue Oct 4, 2017 · 4 comments · Fixed by #17846
Closed

merging two empty dataframes can incur a division by zero #17776

micklat opened this issue Oct 4, 2017 · 4 comments · Fixed by #17846
Labels
Compat pandas objects compatability with Numpy or Python functions good first issue Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Milestone
0.21.1

Comments

@micklat
Copy link

micklat commented Oct 4, 2017

Code Sample, a copy-pastable example if possible

import numpy
import pandas

pandas.show_versions()

a = pandas.DataFrame({'a':[],'b':[],'c':[]})
numpy.seterr(divide='raise')
pandas.merge(a,a,on=('a','b')) # no problem if we only merge on 'a'.

Problem description

The call to merge triggers a division by zero.

Traceback (most recent call last):
File "/homes/mickyl/work/bugs/pandas_merge_div_by_0.py", line 8, in
pandas.merge(a,a,on=('a','b'))
File "/homes/mickyl/venvs/debian8/local/lib/python2.7/site-packages/pandas/core/reshape/merge.py", line 54, in merge
return op.get_result()
File "/homes/mickyl/venvs/debian8/local/lib/python2.7/site-packages/pandas/core/reshape/merge.py", line 569, in get_result
join_index, left_indexer, right_indexer = self._get_join_info()
File "/homes/mickyl/venvs/debian8/local/lib/python2.7/site-packages/pandas/core/reshape/merge.py", line 734, in _get_join_info
right_indexer) = self._get_join_indexers()
File "/homes/mickyl/venvs/debian8/local/lib/python2.7/site-packages/pandas/core/reshape/merge.py", line 713, in _get_join_indexers
how=self.how)
File "/homes/mickyl/venvs/debian8/local/lib/python2.7/site-packages/pandas/core/reshape/merge.py", line 985, in _get_join_indexers
lkey, rkey = _get_join_keys(llab, rlab, shape, sort)
File "/homes/mickyl/venvs/debian8/local/lib/python2.7/site-packages/pandas/core/reshape/merge.py", line 1457, in _get_join_keys
stride //= shape[i]
FloatingPointError: divide by zero encountered in long_scalars

The expeted behaviour is for merge to return an empty dataframe without causing division by 0.

Expected Output

just a print-out of all the version numbers with no exception.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 2.7.9.final.0
python-bits: 64
OS: Linux
OS-release: 4.7.0-0.bpo.1-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.20.3
pytest: 2.6.3
pip: 9.0.1
setuptools: 5.5.1
Cython: 0.25.2
numpy: 1.13.3
scipy: 0.19.1
xarray: None
IPython: 5.4.1
sphinx: 1.2.3
patsy: None
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.1.1
numexpr: 2.6.4
feather: None
matplotlib: 2.0.0
openpyxl: 2.4.8
xlrd: 0.9.2
xlwt: 0.7.5
xlsxwriter: 0.5.2
lxml: 3.4.0
bs4: None
html5lib: 0.999999999
sqlalchemy: 0.9.8
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None
@jreback
Copy link
Contributor

jreback commented Oct 4, 2017

you could make a patch to fix this. note that we surround pandas operations with np.seterr(divide='ignore') as a matter of course; we DO want to propagate. In this case it would be checked directly though.

@jreback jreback added Compat pandas objects compatability with Numpy or Python functions Difficulty Novice Reshaping Concat, Merge/Join, Stack/Unstack, Explode labels Oct 4, 2017
@jreback jreback added this to the Next Major Release milestone Oct 4, 2017
@micklat
Copy link
Author

micklat commented Oct 8, 2017

I'm sorry, I don't know what the right fix would be. Please remember that I don't understand this codebase like you do.

@jreback
Copy link
Contributor

jreback commented Oct 8, 2017

well you would have to debug this and see where the error is occurring. then surround that with a np.seterr(divid='ignore')

@yeemey yeemey mentioned this issue Oct 10, 2017
Closed
4 tasks
@yeemey
Copy link
Contributor

yeemey commented Oct 10, 2017

Hi, does "make a patch" mean edit the code directly? Sorry, I'm new to this and thought I'd try it, but now am not sure if I did the right thing.

yeemey added a commit to yeemey/pandas that referenced this issue Oct 11, 2017
@yeemey
BUG: Ignore division by 0 when merging empty dataframes (pandas-dev#1…
ff8e1bb 
…7776)
@yeemey yeemey mentioned this issue Oct 11, 2017
Merged
4 tasks
yeemey added a commit to yeemey/pandas that referenced this issue Oct 13, 2017
@yeemey
BUG: Ignore division by 0 when merging empty dataframes (pandas-dev#1…
5c5acd1 
…7776)
yeemey added a commit to yeemey/pandas that referenced this issue Oct 13, 2017
@yeemey
BUG: Use context manager; add test from pandas-dev#17776
039315a
yeemey added a commit to yeemey/pandas that referenced this issue Oct 14, 2017
@yeemey
Rebase BUG: Ignore division by 0 when merging empty dataframes (panda…
ccbf927 
…s-dev#17776)
yeemey added a commit to yeemey/pandas that referenced this issue Oct 14, 2017
@yeemey
BUG: Ignore division by 0 when merging empty dataframes (pandas-dev#1…
7aee0b2 
…7776)
yeemey added a commit to yeemey/pandas that referenced this issue Oct 16, 2017
@yeemey
BUG: Ignore division by 0 when merging empty dataframes (pandas-dev#1…
bbf1849 
…7776)
@jreback jreback modified the milestones: Next Major Release, 0.21.1 Nov 26, 2017
jreback pushed a commit that referenced this issue Nov 27, 2017
@yeemey @jreback
BUG: Ignore division by 0 when merging empty dataframes (#17776) (#17846
262e8ff 
)
TomAugspurger pushed a commit to TomAugspurger/pandas that referenced this issue Dec 8, 2017
@yeemey @TomAugspurger
BUG: Ignore division by 0 when merging empty dataframes (pandas-dev#1…
792c868 
…7776) (pandas-dev#17846)

(cherry picked from commit 262e8ff)
TomAugspurger pushed a commit that referenced this issue Dec 11, 2017
@yeemey @TomAugspurger
BUG: Ignore division by 0 when merging empty dataframes (#17776) (#17846
d25a8a1 
)

(cherry picked from commit 262e8ff)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Compat pandas objects compatability with Numpy or Python functions good first issue Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Projects
None yet
Milestone
0.21.1
4 participants
@jreback @TomAugspurger @micklat @yeemey