Skip to content

Refine config filters#12

Open
lindaxiang wants to merge 7 commits intomainfrom
refine_config_filters
Open

Refine config filters#12
lindaxiang wants to merge 7 commits intomainfrom
refine_config_filters

Conversation

@lindaxiang
Copy link
Collaborator

@lindaxiang lindaxiang commented Jan 28, 2026

This is to address issue: #10
It also contains the code changes to support multi files input in PR: #4

You can test the PR using the provided test cases:

python3 prototype_mapper.py --study_id PR_12 --input_dir tests/data/source/PR_12 --output_dir tests/data/mapped/PR_12 --study_config_dir tests/studies

Test cases are under:

tests
├── data
│   └── source
│       └── PR_12
│           ├── clinical.csv
│           ├── clinical.tsv
│           └── demographics.csv
└── studies
    └── PR_12
        └── config
            ├── diagnosis.yaml
            └── participant.yaml

The test cases cover the following features:

1.  Support multi-file inputs with primary and secondary files in different formats (Gen3G use case)
2. Join the multiple input files (Gen3G use case)
3. Refine `filters` to support combined inclusion rules (BQC19 use case)
4. Add preprocess step: `calculate_field` (Gen3G use case)


# 3. Refined filter structure to support combined inclusion rules
participant:
filter:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand this correctly, this filter is supposed to only include participants who have consent_date and a diagnosis_code of U07.1 or J12.89. However, the resulting mapped files contain submitter_participant_id P006 which doesn't have a consent_date (https://github.com/Pan-Canadian-Genome-Library/pcgl-data-mapper/blob/refine_config_filters/tests/data/source/PR_12/demographics.csv). Could you clarify why this record is included?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants