Support `patternProperties` for data/samplesheet validation

[aringeri]

Hi,

I would like to validate a csv file that has dynamic columns that are generated based on some sample identifier.

Example CSV file (input.csv):

fieldA, sample1, sample2, sample3,
rowA, 1, 2, 3,
rowB, 4, 5, 6

In JSON schema, I cannot use the regular items.properties to describe these columns because they can change between runs.
Instead I am using the items.patternProperties to validate fields based on regex match (csv_schema.json):

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "title": "my title",
    "type": "array",
    "items": {
        "type": "object",
        "patternProperties": {
            "^sample": {
                "type": "integer"
            }
        },
        "properties" : {
            "fieldA" : {
                ...
            }
        }
    }
}

When using the samplesheetToList function I get warnings that these columns are unidentified:

WARN: Found the following unidentified headers in testfile.csv:
	- sample1
	- ...

I think that nf-core should either:

• treat these fields in the patternProperties as defined
• or add an option to disable the warning for unidentified headers

Also, I think this issue is somewhat related to: #126

[nvnieuwk]

Hey, validation for patternProperties is already supported for file validation using the schema keyword in the parameters schema.

The nature of this keyword goes against the stable output of samplesheetToList as the amount of columns are unknown. However you could validate your samplesheet using the schema keyword for that parameter and read the file in manually (using .splitCsv() for example)

[aringeri]

Ah okay, thank you for your response. It makes more sense now.

I had initially tried something like this:

include { validate } from 'plugin/nf-schema'
...
validate(file(params.input_csv).splitCsv(header: true), "assets/csv_schema.json")

It was failing validation because splitCsv returns every field as a string (and the validation was expecting integers).

Instead this is my (more complicated) working setup where I take a directory as an input parameter, and validate that all the csv files in that directory conform to a schema:

nextflow_schema.json:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://raw.githubusercontent.com/nextflow_schema.json",
  "title": " pipeline parameters",
  "description": "",
  "type": "object",
  "$defs": {
    "input_output_options": {
      "title": "Input/output options",
      "type": "object",
      "fa_icon": "fas fa-terminal",
      "description": "Define where the pipeline should find input data and save output data.",
      "required": ["csv_dir"],
      "properties": {
        "csv_dir": {
          "type": "string",
          "format": "directory-path",
          "exists": true,
          "description": "Path to directory containing one or more '.csv' files."
        }
      }
    }
  }
}

assets/schema_csv_file_path.json:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "title": "Schema for a input csv file",
    "description": "params.csv_input",
    "type": "string",
    "format": "file-path",
    "schema": "assets/schema_csv_file_contents.json",
    "exists": true
}

assets/schema_csv_file_contents.json:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "title": "schema for the contents of an input csv file",
    "type": "array",
    "items": {
        "type": "object",
        "patternProperties": {
            "^sample": {
                "type": "integer"
            }
        },
        "properties" : {
            "fieldA" : {
                ...
            }
        }
    }
}

main.nf:

include { validate } from 'plugin/nf-schema'
...

Channel.fromPath("${params.csv_dir}/*.csv", checkIfExists: true)
    .map { csv_file ->
        validate(csv_file.toString(), "${projectDir}/assets/schema_csv_file_path.json")
        csv_file
    }

Let me know if you can find better ways of achieving this.

[nvnieuwk]

Are you planning on supporting multiple CSVs? In that case, your solution seems to be the correct one.

I'll leave this issue open for now as something in the future might come up to make this easier for you. Just not sure when yet 😁