Skip to content

Fix bundle generate job to preserve nested notebook directory structure#4596

Open
varundeepsaini wants to merge 1 commit intodatabricks:mainfrom
varundeepsaini:fix/bundle-generate-job
Open

Fix bundle generate job to preserve nested notebook directory structure#4596
varundeepsaini wants to merge 1 commit intodatabricks:mainfrom
varundeepsaini:fix/bundle-generate-job

Conversation

@varundeepsaini
Copy link
Contributor

Changes

Fixes #4503.

Add MarkTasksForDownload that computes a common base path across all notebook tasks before downloading, replacing the per-task loop in job.go. This preserves nested directory structure instead of flattening all notebooks into src/.

Tests

  • Unit tests for commonDirPrefix and MarkTasksForDownload.
  • Acceptance test bundle/generate/job_nested_notebooks reproducing the issue.
  • No regressions in existing python_job, ipynb_job, git_job tests.

Test are written by Claude Code

@eng-dev-ecosystem-bot
Copy link
Collaborator

eng-dev-ecosystem-bot commented Feb 27, 2026

Commit: dc572e1

Run: 22484902946

Env 🟨​KNOWN 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
🟨​ aws linux 7 1 7 268 773 6:44
🟨​ aws windows 7 1 7 270 771 5:57
💚​ aws-ucws linux 8 7 364 689 6:43
💚​ aws-ucws windows 8 7 366 687 4:29
💚​ azure linux 2 9 271 771 5:47
💚​ azure windows 2 9 273 769 4:51
💚​ azure-ucws linux 2 9 369 685 6:58
💚​ azure-ucws windows 2 9 371 683 4:49
💚​ gcp linux 2 9 267 774 5:18
💚​ gcp windows 2 9 269 772 4:30
15 interesting tests: 7 KNOWN, 7 SKIP, 1 RECOVERED
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
🟨​ TestAccept 🟨​K 🟨​K 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 🟨​K 🟨​K 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 🟨​K 🟨​K 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 💚​R 💚​R
🙈​ TestAccept/bundle/resources/postgres_branches/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/update_protected 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/without_branch_id 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_endpoints/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/synced_database_tables/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/ssh/connection 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 20 slowest tests (at least 2 minutes):
duration env testname
3:17 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:11 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:07 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:06 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:56 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:46 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:46 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:42 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:41 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:40 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:40 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:40 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:37 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:36 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:26 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:15 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:13 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:10 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:10 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:10 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct

@varundeepsaini varundeepsaini force-pushed the fix/bundle-generate-job branch from dc572e1 to 6a23c44 Compare February 27, 2026 17:08
@varundeepsaini
Copy link
Contributor Author

@andrewnester could you run the checks, there were some issues on windows, have fixed those

Copy link
Contributor

@andrewnester andrewnester left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add a NEXT_CHANGELOG entry for this change, following the format there? Thank you!

@varundeepsaini varundeepsaini force-pushed the fix/bundle-generate-job branch from 6a23c44 to 545af5c Compare March 3, 2026 09:46
@varundeepsaini
Copy link
Contributor Author

@andrewnester done.

@andrewnester andrewnester self-requested a review March 4, 2026 16:11
@andrewnester
Copy link
Contributor

@varundeepsaini, this change could break for users who continuously run the "bundle generates job" command to sync the files back. At the very least, can you expand this PR to clean up the files at the old location?

@varundeepsaini varundeepsaini force-pushed the fix/bundle-generate-job branch from 6384a63 to 2e1b40d Compare March 4, 2026 17:14
Signed-off-by: Varun Deep Saini <varun.23bcs10048@ms.sst.scaler.com>
@varundeepsaini varundeepsaini force-pushed the fix/bundle-generate-job branch from 2e1b40d to 7f9bca0 Compare March 4, 2026 17:16
@github-actions
Copy link

github-actions bot commented Mar 4, 2026

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

  • PR number: 4596
  • Commit SHA: 7f9bca0329d296e4ffe8fe901e7912255b039cc1

Checks will be approved automatically on success.

@varundeepsaini
Copy link
Contributor Author

@andrewnester could you run the ci ?
thankss !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

databricks bundle generate job --existing-job-id flattens notebook directory structure

3 participants