fix(test): test_create_with_query_cache

[BuonOmo]

Problem

test_create_with_query_cache fails quite often, but it is a flaky test, and fast local reproduction is difficult.

Exploration

I started a bunch of parallel tests (256 is the maximum allowed by Github Actions), and checked if one was failing fast. Then isolated the related tests, and restarted. Once with a limited amount of tests, I could check in a more linear fashion which tests were leading to the bad conditions. And here a constant reproduction (Rails 8.0.3, CRDB 25.2):

TESTOPTS="--seed=1 --name='/test_truncate_with_query_cache|test_create_with_query_cache/'" \
COCKROACH_SKIP_LOAD_SCHEMA=1 bundle exec rake test

Reproduction time is 5s and we isolated two test. Now we should be able to pinpoint the issue.

Bug reproduction

# frozen_string_literal: true

require "activerecord-cockroachdb-adapter"
require "minitest/autorun"
require "logger"

FIXTURES_ROOT = Bundler.load.specs["activerecord"].first.full_gem_path + "/test/fixtures"

ActiveRecord::Base.establish_connection("cockroachdb://root@localhost:26257/defaultdb")

# ActiveRecord::Base.logger = Logger.new(STDOUT)

def title(str)
  puts
  puts str
  puts "=" * (str.length)
  puts
end

title "ActiveRecord Version: #{ActiveRecord::VERSION::STRING}"

title ActiveRecord::Base.connection.select_one("select version()")["version"]

ActiveRecord::Schema.define do
  create_table :posts, force: true do |t|
    t.references :author
    t.string :title, null: false
    t.text    :body, null: false
    t.string  :type
    t.integer :legacy_comments_count, default: 0
    t.integer :taggings_with_delete_all_count, default: 0
    t.integer :taggings_with_destroy_count, default: 0
    t.integer :tags_count, default: 0
    t.integer :indestructible_tags_count, default: 0
    t.integer :tags_with_destroy_count, default: 0
    t.integer :tags_with_nullify_count, default: 0
  end
end

class Post < ActiveRecord::Base
end

class BugTest < ActiveSupport::TestCase
  def test_query_cache_should_not_hit
    @connection = ActiveRecord::Base.connection
    reset_fixtures("posts")

    begin # test_truncate_with_query_cache
      @connection.enable_query_cache!

      assert_operator Post.count, :>, 0

      @connection.truncate("posts")
      assert_equal 0, Post.count
    ensure
      reset_fixtures("posts")
      @connection.disable_query_cache!
    end

    begin # test_create_with_query_cache
      @connection.enable_query_cache!

      count = Post.count

      @connection.create("INSERT INTO posts(title, body) VALUES ('', '')")

      assert_equal count + 1, Post.count
    ensure
      reset_fixtures("posts")
      @connection.disable_query_cache!
    end
  end

  private def reset_fixtures(fixture_name)
    ActiveRecord::FixtureSet.reset_cache

    ActiveRecord::FixtureSet.create_fixtures(FIXTURES_ROOT, fixture_name)
  end
end

If we use this reproduction with git bisect, we can see that the first bug come from commit 63e8e86

Conclusion

Based on this search and some debugging, I saw that the root cause came from rails/rails#52428. The fix is trivial though.

I took the opportunity to extract my way of isolating the test through a new 'flaky.yml' workflow. That will allow us to be more efficient on flaky tests. And there are still some...

[BuonOmo]

@rafiss I would rebase and merge this if approved, the two commits are separated to make more sense :)

[rafiss]

this was a great investigation, thank you for tracking this down! the new testing workflow should be very useful.