samplesheetToList converting string values to integer in scientific notation

[juyeongkim]

example schema:

"sample": {
    "type": "string",
    "pattern": "^\\S+$",
    "errorMessage": "Sample name must be provided and cannot contain spaces"
}

example samplesheet.csv:

sample
10E8
4E10

validation error message:

-> Entry 1: Error for field 'sample' (1.0E+9): Value is [integer] but should be [string] (Sample name must be provided and cannot contain spaces)
-> Entry 2: Error for field 'sample' (4E+10): Value is [integer] but should be [string] (Sample name must be provided and cannot contain spaces)

Sample names are converted from string values (10E8 and 4E10) to integer in scientific notation (1.0E+9 and 4E+10) in samplesheetToList.

nextflow version: 25.04.5
nf-schema version: 2.4.2

[nvnieuwk]

Hi! This is a known issue with no simple solution at the moment... The plugin has to infer the data types of all values in those files and does tend to make some wrong assumptions about this.

One way to prevent this is to start using YAML or JSON samplesheets (which do have native data types integrated in the format).

Actually fixing this would require the plugin to read the types from the JSON schema itself (which is pretty difficult for complex schemas). This will happen some time in the future but it can take a while until that's implemented

[awgymer]

Ooft. This is a new edge-case I haven't seen before. Worryingly this breaks my previous solution of just allowing integer or string and converting anything that is an integer back to a string within the pipeline. It's also caused me to reassess the behaviour that allowed this and I discovered that whilst 0123 will be parsed as a string by default this doesn't work for floating points:

Object val = JSONObject.stringToValue("0123.2");
System.out.println(val);
System.out.println(val.getClass());

123.2
class java.math.BigDecimal

I think perhaps we need to revisit some form of forced casting for CSV/TSV?

[nvnieuwk]

I think perhaps we need to revisit some form of forced casting for CSV/TSV?

Yes that's the plan! It's just not super easy with the nested schemas containing Of blocks because how can we decide which type it needs to be :p. We will have to really think this through first

[awgymer]

Yeah I agree, I started down this road with trying to improve path-casting for #50, but I quickly discovered issues around collecting annotations which were not alongside the type designation, some of which I outline in #136.

[juyeongkim]

Thanks!

Yeah, it's a weird edge case. These are HIV antibodies: https://pubmed.ncbi.nlm.nih.gov/29386285/

For now, I am adding a prefix to the sample names, but in future, I might switch to json or yaml based samplesheets.

I don't see the json/yaml specification in the docs. Are there good examples?

• https://github.com/nextflow-io/nf-schema/blob/master/src/testResources/correct.yaml
• https://github.com/nextflow-io/nf-schema/blob/master/src/testResources/correct.json

[nvnieuwk]

Yes those are good examples of a JSON/YAML samplesheet :) Feel free to ping me if something is not working right