Skip to content

Azure path incorrectly determined to be directory by trailing slash #6427

@adamrtalbot

Description

@adamrtalbot

Bug report

If using nf-schema, Nextflow will determine an Azure path to be a directory if it's missing a trailing slash:

ERROR ~ Validation of pipeline parameters failed!

 -- Check '.nextflow.log' file for details
The following invalid input values have been detected:

* --outdir (az://pipeline/output): 'az://pipeline/output' is not a directory, but a file

 -- Check script '/home/username/.nextflow/assets/nf-core/rnaseq/subworkflows/nf-core/utils_nfschema_plugin/main.nf' at line: 39 or see '.nextflow.log' file for more details

Expected behavior and actual behavior

nf-schema checks Nextflow.file.directory for whether a file is a directory, but for Azure this is inaccurate because it's just checking for the trailing slash:

For most situations, this is fine but when using nf-schema this becomes an issue because it halts the pipeline.

Instead, we should be able to have a more robust way of checking if the file is a directory on Azure, it's probably possible using the SDKs.

I will add a bypass for nf-schema.

Steps to reproduce the problem

main.nf:

// Print the parameters for inspection
log.info """
Parameters:
s3_file: ${params.s3_file ?: 'not set'}
gs_file: ${params.gs_file ?: 'not set'}
az_file: ${params.az_file ?: 'not set'}
"""

workflow {
    println "Cloud path validation example completed successfully!"
}

nextflow_schema.json:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "$id": "https://raw.githubusercontent.com/nf-core/nf-schema/master/examples/cloudpath/nextflow_schema.json",
    "title": "Cloud path validation example",
    "description": "Example demonstrating cloud path validation",
    "type": "object",
    "properties": {
        "s3_file": {
            "type": "string",
            "format": "file-path",
            "description": "An S3 file path"
        },
        "gs_file": {
            "type": "string",
            "format": "file-path",
            "description": "A Google Cloud Storage file path"
        },
        "az_file": {
            "type": "string",
            "format": "file-path",
            "description": "An Azure Blob Storage file path"
        }
    }
}

Program output

(Copy and paste the output produced by the failing execution. Please highlight it as a code block. Whenever possible upload the .nextflow.log file.)

Environment

  • Nextflow version: 25.04.7 build 5955
  • nf-schema version: 2.5.1
  • Java version: openjdk version "17.0.3" 2022-04-19
  • Operating system: macOS
  • Bash version: zsh

Additional context

Slack thread: https://nextflow.slack.com/archives/C02TPRRLCF4/p1758724547528979?thread_ts=1727918072.021049&cid=C02TPRRLCF4
nf-schema issue: nextflow-io/nf-schema#16

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions