USE 449 - support btrix-args-json

[ghukill]

Purpose and background context

This PR allows passing of btrix-args-json in the StepFunction input payload, and having this propagate to the browsertrix-harvester CLI command.

How can a reviewer manually see the effects of these changes?

1- Build SAM image:

make sam-build

2- Run pre-existing sample that exercise this new input payload property:

make sam-example-libguides-extract

Payload sent with this:

{
  "next-step": "extract",
  "run-date": "2026-03-13",
  "run-type": "full",
  "source": "libguides",
  "verbose": "true",
  "btrix-config-yaml-file": "s3://timdex-extract-dev-222053980223/libguides/config/libguides.yaml",
  "btrix-sitemaps": [
    "https://libguides.mit.edu/sitemap.xml"
  ],
  "btrix-sitemap-urls-output-file": "s3://timdex-extract-dev-222053980223/libguides/last-sitemaps-urls.txt",
  --------> "btrix-args-json":"{\"--scopeType\":\"custom\",\"--include\":\".*libguides.mit.edu/(c.php\\?g=.*\\&p=.*)\"}"
}

Expected output with --btrix-args-json present:

{
  "next-step": "transform",
  "run-date": "2026-03-13",
  "run-type": "full",
  "run-id": "8afb9aa8-3605-4817-a114-0a4074aa4c8d",
  "source": "libguides",
  "verbose": true,
  "harvester-type": "browsertrix",
  "extract": {
    "extract-command": [
      "--verbose",
      "harvest",
      "--config-yaml-file=s3://timdex-extract-dev-222053980223/libguides/config/libguides.yaml",
      "--records-output-file=s3://timdex-extract-dev-222053980223/libguides/libguides-2026-03-13-full-extracted-records-to-index.jsonl",
      "--sitemap=https://libguides.mit.edu/sitemap.xml",
      "--sitemap-urls-output-file=s3://timdex-extract-dev-222053980223/libguides/last-sitemaps-urls.txt",
      ------> "--btrix-args-json={\"--scopeType\":\"custom\",\"--include\":\".*libguides.mit.edu/(c.php\\?g=.*\\&p=.*)\"}"
    ]
  }
}

Includes new or updated dependencies?

YES

Changes expectations for external applications?

YES: StepFunction input payload can now pass browsertrix-harveset overrides

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/USE-449

[Copilot]

Pull request overview

This PR extends the “format input / command generation” layer of the TIMDEX pipeline lambdas to accept a new Step Function input property (btrix-args-json) and include it in the generated Browsertrix harvester extract command.

Changes:

• Add support for btrix-args-json when generating Browsertrix extract commands.
• Update unit tests for Browsertrix extract command generation to include the new argument.
• Update the SAM example fixture payload for libguides and refresh dependency lockfile entries.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 4 comments.

File	Description
lambdas/commands.py	Appends btrix-args-json into the Browsertrix extract command argument list.
tests/test_commands.py	Adds test coverage asserting the new argument is present in Browsertrix extract commands.
tests/fixtures/event_payloads/libguides-full-extract.json	Updates the sample payload used for sam local invoke to include Browsertrix inputs and btrix-args-json.
Pipfile.lock	Updates locked dependency versions and adds/updates transitive deps.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[ghukill]

This was exactly correct. Force push coming shortly. Encountered this during a trial run.