BUG: assignment fails with copy_on_write = True

[kameamea]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

pd.options.mode.copy_on_write = True
# pd.options.mode.copy_on_write = "warn"
dftest = pd.DataFrame({"A":[1,4,1,5], "B":[2,5,2,6], "C":[3,6,1,7]})
df=dftest[["B","C"]]
df.iloc[[1,3],:] = [[2, 2],[2 ,2]]

Issue Description

The result is the following error output:

Traceback (most recent call last):
File "/home/rolf/jupyter/venv/lib/python3.12/site-packages/pandas/core/internals/blocks.py", line 1429, in setitem
values[indexer] = casted
~~~~~~^^^^^^^^^
ValueError: shape mismatch: value array of shape (2,2) could not be broadcast to indexing result of shape (2,)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/rolf/code/python/bug.py", line 7, in
df.iloc[[1,3],:] = [[2, 2],[2 ,2]]
~~~~~~~^^^^^^^^^
File "/home/rolf/jupyter/venv/lib/python3.12/site-packages/pandas/core/indexing.py", line 911, in setitem
iloc._setitem_with_indexer(indexer, value, self.name)
File "/home/rolf/jupyter/venv/lib/python3.12/site-packages/pandas/core/indexing.py", line 1944, in _setitem_with_indexer
self._setitem_single_block(indexer, value, name)
File "/home/rolf/jupyter/venv/lib/python3.12/site-packages/pandas/core/indexing.py", line 2218, in _setitem_single_block
self.obj._mgr = self.obj._mgr.setitem(indexer=indexer, value=value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rolf/jupyter/venv/lib/python3.12/site-packages/pandas/core/internals/managers.py", line 409, in setitem
self.blocks[0].setitem((indexer[0], np.arange(len(blk_loc))), value)
File "/home/rolf/jupyter/venv/lib/python3.12/site-packages/pandas/core/internals/blocks.py", line 1432, in setitem
raise ValueError(
ValueError: setting an array element with a sequence.

Expected Behavior

Should not give an error. When uncommenting the copy_on_write="warn" line the code runs with no error message as expected (I'd expect a warning in this case, but thats a separate issue)

Installed Versions

INSTALLED VERSIONS ------------------ commit : 0691c5c python : 3.12.3 python-bits : 64 OS : Linux OS-release : 6.8.0-47-generic Version : #47-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 21:40:26 UTC 2024 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : de_DE.UTF-8 LOCALE : de_DE.UTF-8

pandas : 2.2.3
numpy : 2.0.2
pytz : 2024.2
dateutil : 2.9.0.post0
pip : 24.0
Cython : None
sphinx : None
IPython : 8.27.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.4
lxml.etree : None
matplotlib : 3.9.2
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : None
pyreadstat : None
pytest : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.14.1
sqlalchemy : 2.0.36
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2024.2
qtpy : None
pyqt5 : None

[sooooooing]

take

[jorisvandenbossche]

@kameamea thanks a lot for testing the CoW mode and for the report! I can confirm that I see the same for 2.2 (it works fine without enabling CoW, but gives that error when enabled), and it also appears on the main development branch for 3.0. So something we should definitely fix.

[jorisvandenbossche]

FWIW I think is related to #51435 (comment), i.e. the fact that we pass integer indices instead of a slice(None) (:) to the underlying method, which ends up using this for indexing the numpy array, which fails in this case.

[bobcorthals]

FYI. Remarkably, the reported error is raised only on the first call. Demonstration of this behavior (pandas 2.2.3):

import pandas as pd
# 2.2.3

pd.options.mode.copy_on_write = True
dftest = pd.DataFrame({"A":[1,4,1,5], "B":[2,5,2,6], "C":[3,6,1,7]})
df=dftest[["B","C"]]
try:
    print('try: I failed\n')
    df.iloc[[1,3],:] = [[2, 2],[2 ,2]]
except:
    df.iloc[[1,3],:] = [[2, 2],[2 ,2]]
    print('except: I worked\n')
    
print(df)

Prints:

try: I failed

except: I worked

   B  C
0  2  3
1  2  2
2  2  1
3  2  2

[jorisvandenbossche]

@bobcorthals I think the reason for that is because the initial dataframes starts under the hood with the three columns stored in a single "blcok" (a single 2d numpy array). Then in the first assignment (which fails) it splits the block, and then the second time you try the assignment, because the block is already split at that point, it takes a different code path that does not have the bug.