[SQL/MEG] Add physiological_file.DownloadPath field

[MaximeBICMTL]

As discussed on Slack.

Problem

For the EEG data types that LORIS currently handles, the acquisition files are "real" files, which is very nice. However, for the MEG CTF data types, the acquisition files are actually directories. The BIDS specification says that these MEG CTF acquisitions must be stored as directories, not as archives, and the neuroscience libraries like MNE only work on directories. As such, it makes sense to store the MEG CTF as directories in the LORIS BIDS dataset. However, the users must also be able to download these acquisition files, which is problematic if those are directories.

Solution

Split physiological_file.FilePath into two fields FilePath (stays the same) and DownloadPath (new field):

• For "real" files, like those we currently have with EEG, the values of these fields are identical, as they both point to the same file.
• For directories, the FilePath field points to the directory in the BIDS dataset, and the DownloadPath points to an archive of that directory.

Whenever computation needs to be done in the dataset, the FilePath field is used, whenever the user wants to download an acquisition file, DownloadPath is used.

Testing

• ✅ Ran the patch on Raisinread.
• ✅ Looked for all occurences of physiological_file.FilePath in the electrophysiology_modules.
• ✅ Downloaded a main EEG acquisition file and its associated files, and confirmed that changing the DownloadPath to an invalid value only breaks the main file download.

[jeffersoncasimir]

LGTM, but missing change to raisinbread/RB_files/RB_physiological_file.sql

[MaximeBICMTL]

@jeffersoncasimir Added the Raisinbread SQL, good catch!

[jeffersoncasimir]

LGTM!

[MaximeBICMTL]

Merged the main branch to resolve conflict, ready to go!

[driusan]

I am not crazy about this because of the possibility of data between the two getting de-synced and not having a single source of truth, and also because of the data duplication.

Is it not possible to either:

1. Only store the archive, and extract it on the fly when running tools or
2. Only store the directory, and package it on the fly to download?

[MaximeBICMTL]

Hhhhm, I see your point but I think it would make the code more complicated and less responsive. IMO storing an archive in the BIDS is a big no-no because that is against the BIDS specification and makes it a lot harder to run BIDS libraries on the dataset, especially dataset-wide analyses where we would have to extract/copy every CTF acquisition in the dataset for each analysis.

Regarding the opposite approach, I think tarring the directory on-request for download takes too much time and is not responsive enough. In fact, we already use the "copy multiple files into an archive for download" approach with the physiological_archive (all acquisition-related files) and physiological_event_archive (all acquisition event files) tables. Maybe we could have the on-demand download archiving but we don't have any infrastructure for that ATM... IMO CTF files are specific enough that this is acceptable.