[Enhancement] Support Azure Workload Identity authentication for Azure Data Lake Storage Gen2 (backport #62754)

[CosminLazar]

Why I'm doing:

PR #62754 could not be backported to branch-3.5 due to merge conflicts (see #62764). This fixes the merge conflicts.

What I'm doing:

Fixed the merge conflicts.

Fixes #62764 failed backport

What type of PR is this:

• BugFix
• Feature
• Enhancement
• Refactor
• UT
• Doc
• Tool

Does this PR entail a change in behavior?

• Yes, this PR will result in a change in behavior.
• No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• Interface/UI changes: syntax, type conversion, expression evaluation, display information
• Parameter changes: default values, similar parameters but with different default values
• Policy changes: use new policy to replace old one, functionality automatically enabled
• Feature removed
• Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• I have added test cases for my bug fix or my new feature
• This pr needs user documentation (for new or modified features or behaviors)
  • I have added documentation for my new feature or new function
• This is a backport pr

Bugfix cherry-pick branch check:

• I have checked the version labels which the pr will be auto-backported to the target branch
  • 4.0
  • 3.5
  • 3.4
  • 3.3

[CosminLazar]

As mentioned in the previous pull requests, there is one outstanding:

• AzureADLS2CloudCredential.toFileStoreInfo is not updated because it depends on com.staros, which I cannot find the source code of. It is unclear to me what are the implications on not having the oauth2TokenFile being part of it.

[CosminLazar]

If this looks good and it gets merged I can do the same backport for version 4.0

[mergify]

🧪 CI Insights

Here's what we observed from your CI run for f38befd.

✅ Passed Jobs With Interesting Signals

Pipeline	Job	Signal	Health on branch-3.5	Retries	🔍 CI Insights	📄 Logs
PR CHECKER	title-check	Base branch is broken, but the job passed. Looks like this might be a real fix 💪		0	View	View

[alvin-celerdata]

@CosminLazar, I believe that v4.0 should get merged first to avoid a fallback when upgrading.

[CosminLazar]

@alvin-celerdata Roger, will look into it. Any thoughts about this: #62882 (comment) ?

[CosminLazar]

Added #62887 to address the v4.0 backport