Optionally allow git filters to be applied when reading blobs

[GrahamDennis]

Motivation

Nix 2.20 introduced a regression (a breaking change) in which git repositories do not properly have filters applied (specifically filters that are configured via .gitattributes). This manifests as "NAR hash mismatch" errors when users are upgrading from earlier versions and they have a dependency that uses .gitattributes to manipulate line endings. curl is the most notable example I am aware of which, for example, specifies in .gitattributes that *.bat files should have CRLF line endings.

This change differs from #13428 in that the nix 2.20 behavior is retained as the default, with the previous behavior now being opt-in via the optional applyFilters attribute to git flake inputs. As such, this PR does not introduce a breaking change.

Context

Fixes #11428, alternative to #13428.

The current nix behaviour seems completely broken when it comes to handling git filters.

#!/bin/bash

set -o errexit
set -o nounset
set -o pipefail

readonly REPO="https://github.com/curl/curl.git"
readonly COMMIT="6b951a6928811507d493303b2878e848c077b471"

readonly FILES=(
  # This file does not contain CRLF but has `eol=crlf` in `.gitattributes`.
  buildconf.bat

  # This file does not contain CRLF.
  README.md

  # This file contains CRLF and isn't matched by `.gitattributes`.
  winbuild/README.md
)

# Don't respect user/system configuration.
export GIT_CONFIG_GLOBAL=/dev/null
export GIT_CONFIG_NOSYSTEM=true

function check_crlf() {
  OUTPUT_DIR="curl-$(echo "$@" | sha256sum | awk '{print $1}')"

  if ! test -d "${OUTPUT_DIR}"; then
    git $@ clone --quiet "${REPO}" "${OUTPUT_DIR}"
    git $@ -C "${OUTPUT_DIR}" checkout --quiet "${COMMIT}"
  fi

  for FILE in ${FILES[@]}; do
    if grep --quiet $'\r' "${OUTPUT_DIR}/${FILE}"; then
      echo "git $@ => ${FILE} contains CRLF"
    else
      echo "git $@ => ${FILE} does not contain CRLF"
    fi
  done

  echo
}

nix --version
for FILE in ${FILES[@]}; do
  if grep --quiet $'\r' "$(nix eval --raw --expr "(builtins.fetchGit { url = \"${REPO}\"; ref = \"master\"; rev = \"${COMMIT}\"; }).outPath")/${FILE}"; then
    echo "nix $(nix --version) => ${FILE} contains CRLF"
  else
    echo "nix $(nix --version) => ${FILE} does not contain CRLF"
  fi
done
echo

check_crlf
check_crlf -c core.eol=lf
check_crlf -c core.eol=crlf
check_crlf -c core.eol=native
check_crlf -c core.autocrlf=false
check_crlf -c core.autocrlf=input
check_crlf -c core.autocrlf=true

The output of the above script on my machine is as follows:

nix (Nix) 2.29.1
nix nix (Nix) 2.29.1 => buildconf.bat does not contain CRLF
nix nix (Nix) 2.29.1 => README.md does not contain CRLF
nix nix (Nix) 2.29.1 => winbuild/README.md contains CRLF

git  => buildconf.bat contains CRLF
git  => README.md does not contain CRLF
git  => winbuild/README.md contains CRLF

git -c core.eol=lf => buildconf.bat contains CRLF
git -c core.eol=lf => README.md does not contain CRLF
git -c core.eol=lf => winbuild/README.md contains CRLF

git -c core.eol=crlf => buildconf.bat contains CRLF
git -c core.eol=crlf => README.md does not contain CRLF
git -c core.eol=crlf => winbuild/README.md contains CRLF

git -c core.eol=native => buildconf.bat contains CRLF
git -c core.eol=native => README.md does not contain CRLF
git -c core.eol=native => winbuild/README.md contains CRLF

git -c core.autocrlf=false => buildconf.bat contains CRLF
git -c core.autocrlf=false => README.md does not contain CRLF
git -c core.autocrlf=false => winbuild/README.md contains CRLF

git -c core.autocrlf=input => buildconf.bat contains CRLF
git -c core.autocrlf=input => README.md does not contain CRLF
git -c core.autocrlf=input => winbuild/README.md contains CRLF

git -c core.autocrlf=true => buildconf.bat contains CRLF
git -c core.autocrlf=true => README.md contains CRLF
git -c core.autocrlf=true => winbuild/README.md contains CRLF

This shows that the way Nix is handling git repositories is inconsistent with git, irrespective of the user and/or system git configuration.

Another manifestation of this bug is that the NAR hash for a git repository can change depending on the order of evaluations. This can be demonstrated by the following script:

#!/bin/bash

set -o errexit
set -o nounset
set -o pipefail

nix --version
git clone --quiet https://github.com/joshuaspence/nix-crlf-test.git

rm -rf ~/.cache/nix
nix eval --impure --expr "builtins.fetchGit \"file://$(readlink -f nix-crlf-test)\""
nix eval --impure --expr "builtins.fetchGit { url = \"file://$(readlink -f nix-crlf-test)\"; ref = \"$(git -C nix-crlf-test branch --show-current)\"; rev = \"$(git -C nix-crlf-test rev-parse HEAD)\"; }"

rm -rf ~/.cache/nix
nix eval --impure --expr "builtins.fetchGit { url = \"file://$(readlink -f nix-crlf-test)\"; ref = \"$(git -C nix-crlf-test branch --show-current)\"; rev = \"$(git -C nix-crlf-test rev-parse HEAD)\"; }"
nix eval --impure --expr "builtins.fetchGit \"file://$(readlink -f nix-crlf-test)\""

rm -rf nix-crlf-test

The output of this script is as follows:

nix (Nix) 2.29.1
{ lastModified = 946684800; lastModifiedDate = "20000101000000"; narHash = "sha256-k7u7RAaF+OvrbtT3KCCDQA8e9uOdflUo5zSgsosoLzA="; outPath = "/nix/store/pbm7g5wjg44d1z7byaivhcs9rrv58fqf-source"; rev = "27fcdeab9b5edc4095160b6d9a15a5c5260bca38"; revCount = 2; shortRev = "27fcdea"; submodules = false; }
{ lastModified = 946684800; lastModifiedDate = "20000101000000"; narHash = "sha256-k7u7RAaF+OvrbtT3KCCDQA8e9uOdflUo5zSgsosoLzA="; outPath = "/nix/store/pbm7g5wjg44d1z7byaivhcs9rrv58fqf-source"; rev = "27fcdeab9b5edc4095160b6d9a15a5c5260bca38"; revCount = 2; shortRev = "27fcdea"; submodules = false; }
{ lastModified = 946684800; lastModifiedDate = "20000101000000"; narHash = "sha256-BBhuj+vOnwCUnk5az22PwAnF32KE1aulWAVfCQlbW7U="; outPath = "/nix/store/9vi7nc2507l5fjyd0cg6fgbrikncpjmw-source"; rev = "27fcdeab9b5edc4095160b6d9a15a5c5260bca38"; revCount = 2; shortRev = "27fcdea"; submodules = false; }
{ lastModified = 946684800; lastModifiedDate = "20000101000000"; narHash = "sha256-BBhuj+vOnwCUnk5az22PwAnF32KE1aulWAVfCQlbW7U="; outPath = "/nix/store/9vi7nc2507l5fjyd0cg6fgbrikncpjmw-source"; rev = "27fcdeab9b5edc4095160b6d9a15a5c5260bca38"; revCount = 2; shortRev = "27fcdea"; submodules = false; }

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.