Document ART index limitation with concurrent transactions and deletes

[rynoV]

Addresses #5685

[taniabogatsch]

committed on a table with an index (happens for all our indexes, also those in extensions). There, we just don't notice, as we don't use them for constraint checking.

[taniabogatsch]

referencing the deleted record. Note that constraint violations are only relevant for PK, FK, and UNIQUE indexes. Maybe expand on this? Not sure.

[taniabogatsch]

with an index enforcing a constraint violation - I think this can also happen for FKs, no (not only duplicate checks)? Oh, can we actually enter an inconsistent state here? I.e., I delete from the PK table, but keep the value alive. Then, I insert into the FK table, and now the delete goes through. ??

[rynoV]

I just checked this using the .NET SDK and it does seem to get into an inconsistent state. This is the pseudocode:

// Setup: Create a primary table with UUID primary key and a secondary table with foreign key reference
primaryId = generateNewGUID()
conn = duckdbConnectInMemory()

// Create tables and insert initial record in primary table
duckdb(conn, "CREATE TABLE primary_table (id UUID PRIMARY KEY)")
duckdb(conn, "CREATE TABLE secondary_table (primary_id UUID, FOREIGN KEY (primary_id) REFERENCES primary_table(id))")
duckdbInsert(conn, "primary_table", {id: primaryId})

// Start transaction tx1 which will read from primary_table
tx1 = duckdbTxStart(conn)
readRecord = duckdb(tx1, "SELECT id FROM primary_table LIMIT 1")
// Note: tx1 remains open, holding locks/resources

// Outside of tx1, delete the record from primary_table
duckdbDelete(conn, "primary_table", {id: primaryId})

// Try to insert into secondary_table with foreign key reference to the now-deleted primary record
// This succeeds because tx1 is still open and the constraint isn't fully enforced yet
duckdbInsert(conn, "secondary_table", {primary_id: primaryId})

// Commit tx1, releasing any locks/resources
duckdbTxCommit(tx1)

// Verify the primary record is indeed deleted
count = duckdb(conn, "SELECT COUNT(*) FROM primary_table WHERE id = $primaryId",{primaryId: primaryId})
assert(count == 0, "Record should be deleted")

// Verify the secondary record with the foreign key reference exists
// This demonstrates the inconsistent state
count = duckdb(conn, "SELECT COUNT(*) FROM secondary_table WHERE primary_id = $primaryId",{primaryId: primaryId})
assert(count == 1, "Foreign key reference should exist")

If tx1 is committed before the secondary_table insert, the insert fails as expected, but if not the assertions succeed

[rynoV]

Thanks @taniabogatsch , I updated the section, let me know if that covers your suggestions

I also opened duckdb/duckdb#19002 based on your comment

[taniabogatsch]

Thanks! Looks good, I think covering that inconsistent FK state makes sense, even though it is a bug (as far as I can tell) and has to be fixed - at which point we can update the documentation again. And thanks for opening that issue 👍