BUG: Drop duplicates bug when using pyarrow (version 21.0.0) backend

[flori-ko]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

# run 'pip install pyarrow==21.0.0' before running this code to reproduce
import pandas as pd

# Having the empty string as well as a None value seems required to trigger the bug
df = pd.DataFrame(data={"A": ["a", "b", "", "a", None, "b"]})
# If this column is added, drop duplicates even removes a wrong column
df["B"] = "b"
# Drop duplicates misses a duplicated column the first time its called with pyarrow backend
df = df.convert_dtypes(dtype_backend="pyarrow")

print(df)
print("\n\n")
# The first call doesn't drop all duplicates, and even drops the <NA> row if the column B is added
print(df.drop_duplicates())
print("\n\n")
# The second call drop the remaining duplicate
print(df.drop_duplicates().drop_duplicates())

Issue Description

When calling df.drop_duplicates on a pandas Dataframe using the pyarrow backend, it incorrectly removes unique columns and keeps non unique columns. This only occurs with the pyarrow version 21.0.0, earlier versions seem to work without issue.
I could only trigger this bug, having both another unique value and a None value in the column containing duplicate entries.
Adding a second column with a fixed value (like "B"), leads to drop_duplicates removing the row containing the value, even though the row is unique.

Expected Behavior

I would expect drop_duplicates to only drop duplicate rows and keep unique rows when using a pyarrow backend (as it does correctly for the numpy nullable backend).

Installed Versions

INSTALLED VERSIONS ------------------ commit : 9c8bc3e python : 3.11.9 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.22631 machine : AMD64 processor : Intel64 Family 6 Model 143 Stepping 8, GenuineIntel byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : English_United States.1252

pandas : 2.3.3
numpy : 2.3.3
pytz : 2025.2
dateutil : 2.9.0.post0
pip : 25.2
Cython : None
sphinx : 8.2.3
IPython : 9.5.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.13.5
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.6
lxml.etree : 6.0.2
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 20.0.0
pyreadstat : None
pytest : 8.4.2
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.15.3
sqlalchemy : 2.0.43
tables : None
tabulate : 0.9.0
xarray : 2025.9.0
xlrd : None
xlsxwriter : 3.2.9
zstandard : None
tzdata : 2025.2
qtpy : None
pyqt5 : None

[rahulratho15]

take

[rahulratho15]

HI,I am a first-time contributor and will start by reproducing & writing a failing test.

[rahulratho15]

I've successfully reproduced this bug with pandas 2.3.3 and PyArrow 21.0.0. The first call to drop_duplicates() returns 4 rows instead of the expected 3 rows when using PyArrow backend. The second call correctly returns 3 rows. I'm now investigating the root cause in the PyArrow backend implementation.

[flori-ko]

I've successfully reproduced this bug with pandas 2.3.3 and PyArrow 21.0.0. The first call to drop_duplicates() returns 4 rows instead of the expected 3 rows when using PyArrow backend. The second call correctly returns 3 rows. I'm now investigating the root cause in the PyArrow backend implementation.

Actually there are only two duplicte rows, so 4 rows is the correct number of rows. However row index 3 and 4 are dropped instead of index 3 and 5 the first time it is called. If you comment out the line where column "B" is added, the first drop duplictes call only drops row index 3 and incorrectly keeps row index 5.