NIFI-15077 : Added conflict resolution strategies for file move operations in FetchFileTransfer #10405

ravinarayansingh · 2025-10-09T03:45:34Z

Summary

NIFI-15077

Introduced MOVE_CONFLICT_RESOLUTION property to handle filename collisions during file move
Implemented logic for conflict resolution strategies (REPLACE, RENAME, IGNORE, etc.)
Refactored filename normalization and path handling utilities
Updated FetchFTP and FetchSFTP processors for enhanced conflict resolution support
Added corresponding unit tests and utility methods for unique filename generation in conflicts

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Apache NiFi Jira issue created

Pull Request Tracking

Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

Pull Request based on current revision of the main branch
Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

Build completed using ./mvnw clean install -P contrib-check
- JDK 21
- JDK 25

Licensing

New dependencies are compatible with the Apache License 2.0 according to the License Policy
New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

Documentation formatting appears as expected in rendered files

…ons in FetchFileTransfer - Introduced MOVE_CONFLICT_RESOLUTION property to handle filename collisions during file move - Implemented logic for conflict resolution strategies (REPLACE, RENAME, IGNORE, etc.) - Refactored filename normalization and path handling utilities - Updated FetchFTP and FetchSFTP processors for enhanced conflict resolution support - Added corresponding unit tests and utility methods for unique filename generation in conflicts

exceptionfactory

Thanks for proposing this improvement @ravinarayansingh. The concept makes sense, and the basic approach looks good. I noted some recommendations on a few implementation and logging details.

exceptionfactory · 2025-10-22T02:40:56Z

...e-transfer/src/main/java/org/apache/nifi/processor/util/file/transfer/FetchFileTransfer.java

+        .description(String.format("Determines how to handle filename collisions when '%s' is '%s'. " +
+                        "This setting controls behavior when the target file exists in the %s.",


A multiline string with """description""".formatted() can also work and make this a bit easier to read.

This description can be reformatted as mentioned.

@exceptionfactory code already has been updated. i do not know why it's not showing here

...fer/src/main/java/org/apache/nifi/processor/util/file/transfer/FileTransferConflictUtil.java

...e-transfer/src/main/java/org/apache/nifi/processor/util/file/transfer/FetchFileTransfer.java

…/src/main/java/org/apache/nifi/processor/util/file/transfer/FetchFileTransfer.java Co-authored-by: David Handermann <[email protected]>

…lictUtil - Replaced incrementing prefix-based approach with UUID prefix for unique filename generation - Simplified conflict resolution logic in FetchFileTransfer during file move operations - Improved logging consistency and updated descriptions for MOVE_DESTINATION_DIR property

ravinarayansingh · 2025-10-23T04:44:40Z

Thanks for proposing this improvement @ravinarayansingh. The concept makes sense, and the basic approach looks good. I noted some recommendations on a few implementation and logging details.

Thanks for review, @exceptionfactory.
I’ve implemented the suggested changes. Please take another look when you have a chance.

- Adjusted indentation for property description for better readability

exceptionfactory

Thanks for updates @ravinarayansingh, there was one previous comment remaining on description formatting, and I recommended some adjustments to the logging messages, then this should be ready to go.

...fer/src/main/java/org/apache/nifi/processor/util/file/transfer/FileTransferConflictUtil.java

exceptionfactory · 2025-10-25T19:47:36Z

...e-transfer/src/main/java/org/apache/nifi/processor/util/file/transfer/FetchFileTransfer.java

+        .description(String.format("Determines how to handle filename collisions when '%s' is '%s'. " +
+                        "This setting controls behavior when the target file exists in the %s.",


This description can be reformatted as mentioned.

...e-transfer/src/main/java/org/apache/nifi/processor/util/file/transfer/FetchFileTransfer.java

exceptionfactory · 2025-10-25T19:53:58Z

...e-transfer/src/main/java/org/apache/nifi/processor/util/file/transfer/FetchFileTransfer.java

+                            getLogger().warn("Configured to {} on move conflict for {}. Original remote file will be left in place.", strategy, flowFile);
+                            return;


This appears to behave the same as the ignore strategy, does it fail elsewhere during processing despite the return?

@exceptionfactory i have updated the code please have a look

…/src/main/java/org/apache/nifi/processor/util/file/transfer/FetchFileTransfer.java Co-authored-by: David Handermann <[email protected]>

exceptionfactory

Thanks for the incremental adjustments @ravinarayansingh, it looks like just accepting the proposed changes resulted in compilation issues, and a few comments are still unaddressed. If you can work through the remaining items and update the pull request, that would be helpful.

- Moved unique filename generation method from FileTransferConflictUtil to FetchFileTransfer - Simplified conflict resolution logic by removing unused utility class

exceptionfactory

Thanks for the updates @ravinarayansingh. This looks better, but handling IGNORE, REJECT, and FAIL with the same case does not seem correct. The IGNORE and NONE options should not log a warning as previously mentioned, and the other two options require routing to a different location.

exceptionfactory · 2025-10-28T20:42:03Z

...e-transfer/src/main/java/org/apache/nifi/processor/util/file/transfer/FetchFileTransfer.java

+    private static String generateUniqueFilename(final FileTransfer transfer,
+                                                 final String path,
+                                                 final String baseFileName,
+                                                 final FlowFile flowFile,
+                                                 final ComponentLog logger) throws IOException {


With the log message where this is called, most of these arguments are no longer needed. On review, I recommend removing this method completely and building the filename inside the case statement block.

ravinarayansingh · 2025-10-29T07:32:39Z

Thanks for the updates @ravinarayansingh. This looks better, but handling IGNORE, REJECT, and FAIL with the same case does not seem correct. The IGNORE and NONE options should not log a warning as previously mentioned, and the other two options require routing to a different location.

Thanks for the feedback, @exceptionfactory.
Just to confirm, for the IGNORE and NONE conflict resolution strategies, I’ll keep the log level as INFO, like this:

case FileTransfer.CONFLICT_RESOLUTION_IGNORE:
case FileTransfer.CONFLICT_RESOLUTION_NONE:
    getLogger().info("Configured to {} on move conflict for {}. Original remote file will be left in place.", strategy, flowFile);
    return;

For the other two options — REJECT and FAIL — should these also log a warning, or should they additionally route the FlowFile to a different relationship (for example, failure or reject)?

exceptionfactory · 2025-10-29T13:55:55Z

Thanks for the updates @ravinarayansingh. This looks better, but handling IGNORE, REJECT, and FAIL with the same case does not seem correct. The IGNORE and NONE options should not log a warning as previously mentioned, and the other two options require routing to a different location.

Thanks for the feedback, @exceptionfactory. Just to confirm, for the IGNORE and NONE conflict resolution strategies, I’ll keep the log level as INFO, like this:
case FileTransfer.CONFLICT_RESOLUTION_IGNORE:
case FileTransfer.CONFLICT_RESOLUTION_NONE:
    getLogger().info("Configured to {} on move conflict for {}. Original remote file will be left in place.", strategy, flowFile);
    return;
                            

Yes, for IGNORE and NONE, and INFO log looks good.

For the other two options — REJECT and FAIL — should these also log a warning, or should they additionally route the FlowFile to a different relationship (for example, failure or reject)?

Those options should log a warning, and route to the appropriate relationship, based on the description for each value.

…TE and adjusted pre-commit handling for MOVE - Centralized MOVE and DELETE completion handling with pre-commit routing for MOVE conflicts and post-commit actions for DELETE. - Introduced detailed failure reasons for enhanced debugging. - Added REL_REJECT and REL_FAILURE relationships for comprehensive failure management. - Updated unit tests to reflect the new process flow and relationships.

ravinarayansingh · 2025-10-30T22:17:16Z

Hi @exceptionfactory
I have refactored the FetchFileTransfer processor to improve post-commit and pre-commit handling logic. Centralized MOVE and DELETE completion strategies, introduced detailed failure reasons for better debugging, and added new REL_REJECT and REL_FAILURE relationships for more robust error management. Updated corresponding unit tests to align with the revised flow.
please have a look

exceptionfactory requested changes Oct 22, 2025

View reviewed changes

ravinarayansingh and others added 2 commits October 22, 2025 21:13

Update nifi-extension-bundles/nifi-extension-utils/nifi-file-transfer…

abe0f06

…/src/main/java/org/apache/nifi/processor/util/file/transfer/FetchFileTransfer.java Co-authored-by: David Handermann <[email protected]>

ravinarayansingh added 2 commits October 22, 2025 21:49

Updated MOVE_DESTINATION_DIR property formatting in FetchFileTransfer

5bcaf61

- Adjusted indentation for property description for better readability

Updated MOVE_DESTINATION_DIR property formatting in FetchFileTransfer

1350efb

- Adjusted indentation for property description for better readability

ravinarayansingh requested a review from exceptionfactory October 23, 2025 04:54

exceptionfactory requested changes Oct 25, 2025

View reviewed changes

ravinarayansingh and others added 4 commits October 25, 2025 13:57

Update nifi-extension-bundles/nifi-extension-utils/nifi-file-transfer…

f73dfa9

…/src/main/java/org/apache/nifi/processor/util/file/transfer/FetchFileTransfer.java Co-authored-by: David Handermann <[email protected]>

Update nifi-extension-bundles/nifi-extension-utils/nifi-file-transfer…

e99ee17

…/src/main/java/org/apache/nifi/processor/util/file/transfer/FetchFileTransfer.java Co-authored-by: David Handermann <[email protected]>

Update nifi-extension-bundles/nifi-extension-utils/nifi-file-transfer…

66fb517

…/src/main/java/org/apache/nifi/processor/util/file/transfer/FetchFileTransfer.java Co-authored-by: David Handermann <[email protected]>

Update nifi-extension-bundles/nifi-extension-utils/nifi-file-transfer…

66d7dd0

…/src/main/java/org/apache/nifi/processor/util/file/transfer/FetchFileTransfer.java Co-authored-by: David Handermann <[email protected]>

exceptionfactory requested changes Oct 27, 2025

View reviewed changes

Refactored FetchFileTransfer unique filename generation

88fb177

- Moved unique filename generation method from FileTransferConflictUtil to FetchFileTransfer - Simplified conflict resolution logic by removing unused utility class

exceptionfactory requested changes Oct 28, 2025

View reviewed changes

ravinarayansingh requested a review from exceptionfactory October 30, 2025 22:17

		.description(String.format("Determines how to handle filename collisions when '%s' is '%s'. " +
		"This setting controls behavior when the target file exists in the %s.",

		getLogger().warn("Configured to {} on move conflict for {}. Original remote file will be left in place.", strategy, flowFile);
		return;

NIFI-15077 : Added conflict resolution strategies for file move operations in FetchFileTransfer #10405

Are you sure you want to change the base?

NIFI-15077 : Added conflict resolution strategies for file move operations in FetchFileTransfer #10405

Conversation

ravinarayansingh commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tracking

Issue Tracking

Pull Request Tracking

Pull Request Formatting

Verification

Build

Licensing

Documentation

Uh oh!

exceptionfactory left a comment

Choose a reason for hiding this comment

Uh oh!

exceptionfactory Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

exceptionfactory Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

ravinarayansingh Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ravinarayansingh commented Oct 23, 2025

Uh oh!

exceptionfactory left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

exceptionfactory Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

exceptionfactory Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

ravinarayansingh Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

exceptionfactory left a comment

Choose a reason for hiding this comment

Uh oh!

exceptionfactory left a comment

Choose a reason for hiding this comment

Uh oh!

exceptionfactory Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

ravinarayansingh commented Oct 29, 2025

Uh oh!

exceptionfactory commented Oct 29, 2025

Uh oh!

ravinarayansingh commented Oct 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ravinarayansingh commented Oct 9, 2025 •

edited

Loading