Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DataFrame.iloc raises ValueError: buffer source array is read-only when applied to read-only indices #17192

Closed
lesteve opened this issue Aug 7, 2017 · 1 comment · Fixed by #27375
Closed

DataFrame.iloc raises ValueError: buffer source array is read-only when applied to read-only indices #17192

lesteve opened this issue Aug 7, 2017 · 1 comment · Fixed by #27375
Labels
Indexing Related to indexing on series/frames, not to indexes themselves
Milestone
0.25.0

Comments

@lesteve
Copy link
Contributor

lesteve commented Aug 7, 2017

Snippet reproducing the problem:

import numpy as np
import pandas as pd

df = pd.DataFrame({'data': np.ones(100, dtype='float64')})
indices = np.array([1, 3, 6])
indices.flags.writeable = False
df.iloc[indices]

This seems like a variation of #10043 (fixed by #10070). The difference here is that the indices is read-only not the numpy arrays inside the DataFrame. For completeness this problem was seen in scikit-learn as a by-product of debugging scikit-learn/scikit-learn#9483. We can very likely work-around the problem on the scikit-learn side by copying the indices if they are read-only.

Full stack-trace:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-fb86e43fb729> in <module>()
      5 indices = np.array([1, 3, 6])
      6 indices.flags.writeable = False
----> 7 df.iloc[indices]

/volatile/le243287/miniconda3/lib/python3.6/site-packages/pandas/core/indexing.py in __getitem__(self, key)
   1326         else:
   1327             key = com._apply_if_callable(key, self.obj)
-> 1328             return self._getitem_axis(key, axis=0)
   1329 
   1330     def _is_scalar_access(self, key):

/volatile/le243287/miniconda3/lib/python3.6/site-packages/pandas/core/indexing.py in _getitem_axis(self, key, axis)
   1736         # a list of integers
   1737         elif is_list_like_indexer(key):
-> 1738             return self._get_list_axis(key, axis=axis)
   1739 
   1740         # a single integer

/volatile/le243287/miniconda3/lib/python3.6/site-packages/pandas/core/indexing.py in _get_list_axis(self, key, axis)
   1713         """
   1714         try:
-> 1715             return self.obj.take(key, axis=axis, convert=False)
   1716         except IndexError:
   1717             # re-raise with different error message

/volatile/le243287/miniconda3/lib/python3.6/site-packages/pandas/core/generic.py in take(self, indices, axis, convert, is_copy, **kwargs)                                                                                                 
   1926         new_data = self._data.take(indices,
   1927                                    axis=self._get_block_manager_axis(axis),
-> 1928                                    convert=True, verify=True)
   1929         result = self._constructor(new_data).__finalize__(self)
   1930 

/volatile/le243287/miniconda3/lib/python3.6/site-packages/pandas/core/internals.py in take(self, indexer, axis, verify, convert)
   4009         new_labels = self.axes[axis].take(indexer)
   4010         return self.reindex_indexer(new_axis=new_labels, indexer=indexer,
-> 4011                                     axis=axis, allow_dups=True)
   4012 
   4013     def merge(self, other, lsuffix='', rsuffix=''):

/volatile/le243287/miniconda3/lib/python3.6/site-packages/pandas/core/internals.py in reindex_indexer(self, new_axis, indexer, axis, fill_value, allow_dups, copy)
   3895             new_blocks = [blk.take_nd(indexer, axis=axis, fill_tuple=(
   3896                 fill_value if fill_value is not None else blk.fill_value,))
-> 3897                 for blk in self.blocks]
   3898 
   3899         new_axes = list(self.axes)

/volatile/le243287/miniconda3/lib/python3.6/site-packages/pandas/core/internals.py in <listcomp>(.0)
   3895             new_blocks = [blk.take_nd(indexer, axis=axis, fill_tuple=(
   3896                 fill_value if fill_value is not None else blk.fill_value,))
-> 3897                 for blk in self.blocks]
   3898 
   3899         new_axes = list(self.axes)

/volatile/le243287/miniconda3/lib/python3.6/site-packages/pandas/core/internals.py in take_nd(self, indexer, axis, new_mgr_locs, fill_tuple)
   1044             fill_value = fill_tuple[0]
   1045             new_values = algos.take_nd(values, indexer, axis=axis,
-> 1046                                        allow_fill=True, fill_value=fill_value)
   1047 
   1048         if new_mgr_locs is None:

/volatile/le243287/miniconda3/lib/python3.6/site-packages/pandas/core/algorithms.py in take_nd(arr, indexer, axis, out, fill_value, mask_info, allow_fill)
   1469     func = _get_take_nd_function(arr.ndim, arr.dtype, out.dtype, axis=axis,
   1470                                  mask_info=mask_info)
-> 1471     func(arr, indexer, out, fill_value)
   1472 
   1473     if flip_order:

pandas/_libs/algos_take_helper.pxi in pandas._libs.algos.take_2d_axis0_float64_float64 (pandas/_libs/algos.c:110417)()

/volatile/le243287/miniconda3/lib/python3.6/site-packages/pandas/_libs/algos.cpython-36m-x86_64-linux-gnu.so in View.MemoryView.memoryview_cwrapper (pandas/_libs/algos.c:124730)()

/volatile/le243287/miniconda3/lib/python3.6/site-packages/pandas/_libs/algos.cpython-36m-x86_64-linux-gnu.so in View.MemoryView.memoryview.__cinit__ (pandas/_libs/algos.c:120965)()

ValueError: buffer source array is read-only

Pandas version information:

INSTALLED VERSIONS ------------------https://github.com/scikit-learn/scikit-learn/issues/9483 commit: None python: 3.6.2.final.0 python-bits: 64 OS: Linux OS-release: 4.4.0-83-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: en_GB.UTF-8

pandas: 0.20.3
pytest: 3.1.3
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.26
numpy: 1.13.1
scipy: 0.19.1
xarray: None
IPython: 6.1.0
sphinx: 1.6.2
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 0.9999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None
@lesteve lesteve mentioned this issue Aug 7, 2017
Closed
@chris-b1
Copy link
Contributor

chris-b1 commented Aug 7, 2017

It'd be nice if this were addressed in Cython, xref cython/cython#1605

But until then could do something similar to #10070

@chris-b1 chris-b1 added Difficulty Intermediate Indexing Related to indexing on series/frames, not to indexes themselves labels Aug 7, 2017
@chris-b1 chris-b1 added this to the Next Major Release milestone Aug 7, 2017
@lesteve lesteve mentioned this issue Aug 7, 2017
Merged
@ogrisel ogrisel mentioned this issue Aug 8, 2017
Closed
@jreback jreback mentioned this issue Oct 31, 2017
Closed
@shoyer shoyer mentioned this issue Dec 14, 2017
Closed
jbrockmendel added a commit to jbrockmendel/pandas that referenced this issue Jul 13, 2019
@jbrockmendel
BUG: Fix take with read-only indexer, closes pandas-dev#17192
c49f39c
@jbrockmendel jbrockmendel mentioned this issue Jul 13, 2019
Merged
5 tasks
@jreback jreback modified the milestones: Contributions Welcome, 0.25.0 Jul 15, 2019
jreback pushed a commit that referenced this issue Jul 15, 2019
@jbrockmendel @jreback
BUG: Fix take with read-only indexer, closes #17192 (#27375)
0f725bf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Indexing Related to indexing on series/frames, not to indexes themselves
Projects
None yet
Milestone
0.25.0
Development

Successfully merging a pull request may close this issue.

BUG: Fix take with read-only indexer, closes #17192 jbrockmendel/pandas
3 participants
@jreback @lesteve @chris-b1