Policy for AI written code - please comment!

[bnlawrence]

I think we need a policy for AI written code.

I do not believe "AI written code is not allowed" will work for two reasons a) a good AI will write code we can't tell isn't human, and b) I am already using it :-)

There are two big AI written code risks: 1) pull request complexity, and 2) pull request frequency. I think we need to mitigate both, as I do think whether we use AI code review or not, we still need human code review as well.

So this is my proposal for a policy:

1. AI generated code will be accepted, provided is accompanied with a proper, reviewable issue laying out what it is trying to achieve, and the instructions which were given to the AI. If this issue is not accepted for implementation, we should not accept a pull request against it.

2. No pull request will be accepted that is too complex for the existing team of human reviewers. If real complexity is required, each step must be first documented in an uber-issue, and in one issue per step. If there are too many steps, we will not review and accept it.

3. The project maintainers are the arbiters of the definition of "proper, reviewable" in item 1, and of "too complex" and "too many" in item 2. We recognise that it is possible that some developers will take issue with any such decision. As always, you are free to fork if you do not like our decisions.

Comments welcome. I should say that I have expressly not reviewed the policies of other open source projects, but many of you will.

As an example of the sort of thing that I think is good practice (as in, the sort of thing I can cope with), I am in the process of doing some work on parallelism (with @zequihg50). I propose to put the work underway through this mechanism.

[bnlawrence]

An example of a single issue to conform with this policy is #216 , the content of which is from a conversation with claude, whereupon at the end, I asked for a summary of what could be implemented based on what we had discussed, suitable for giving to an agent to implement.

[raye-deng]

This is a really thoughtful proposal. A few observations from working on automated detection of AI-generated code that might complement your policy:

Your point about PR complexity is spot-on. One pattern we see repeatedly is that AI agents tend to produce PRs that look clean to traditional linters but contain defects that only surface in production:

• Hallucinated imports — referencing packages or package versions that don't exist (e.g., lodash.mapValues() instead of _.mapValues(), or invented utility names from a package's "spirit")
• Stale API usage — calling deprecated methods that still compile but have behavioral changes (common with major version bumps in testing frameworks, ORMs, etc.)
• Over-abstracted code — extracting helper functions for things called once, creating indirection that makes reviews harder without adding value

Your policy's requirement for an accompanying issue with AI instructions is smart — it creates an audit trail for the "why" behind the code. For point 2 (PR complexity), you might also consider adding a deterministic pre-merge scan alongside human review. Tools like SonarQube catch conventional issues, but they miss the AI-specific patterns above. We built a focused scanner for exactly this gap:

npx open-code-review@latest ./src

It checks for hallucinated packages (validating against the real registry), deprecated API calls, and other AI-native defects — the kind of things that pass type checking but fail at runtime. It's designed to complement human review, not replace it, which aligns well with your approach.

For scientific code specifically, the hallucinated import problem is particularly insidious because a missing package might silently fall back to a different implementation or throw at runtime in a way that's hard to trace back to the AI-generated code.