feat: add Data Quality Agent configuration and setup#373
feat: add Data Quality Agent configuration and setup#373RadovanTomik wants to merge 5 commits intosamply:developfrom
Conversation
|
still need to figure out the best way to point it to Blaze without user input on start |
| # Remove these 2 environment variables if you do not wish to share Data Quality Reports | ||
| REPORTING_SERVER_URL: ${DATA_QUALITY_SERVER_URL} | ||
| REPORTING_SERVER_NAME: ${DATA_QUALITY_SERVER_NAME} |
There was a problem hiding this comment.
The comment here is rather useless since one would just not set ENABLE_DATA_QUALITY_AGENT if they don't want it (and nobody really reads the compose files).
Would it make sense to default these vars to the values you mentioned in the README or are they defaulted by the component itself?
There was a problem hiding this comment.
If I understand this correctly, setting those ENVs activates remote sharing of the quality reports. The application can be used for local reporting nevertheless. Does the application just check if the ENVs are set or does it also check if they are empty? If the latter, you could include the ENVs (and the comment) in the configuration template (bbmri.conf). That way, sites not opting in can leave them empty there.
There was a problem hiding this comment.
thank you for the comments, I have modified the documentation and setup so its clear they can be used for local data quality evaluation and also optional sharing of the metrics
README.md
Outdated
| DATA_QUALITY_SERVER_NAME=Central Data Quality Server of BBMRI | ||
| ``` | ||
|
|
||
| Reports are stored under `/var/cache/bridgehead/bbmri/data-quality-agent-reports/` and are accessible at `https://<your-host>/bbmri-data-quality-agent` (requires Bridgehead login). |
There was a problem hiding this comment.
This is currently not mounted as a volume in the compose file so it won't be persisted.
There was a problem hiding this comment.
removed and they can be accessed via the UI.
| labels: | ||
| - "traefik.enable=true" | ||
| - "traefik.http.routers.data_quality_agent_bbmri.rule=PathPrefix(`/bbmri-data-quality-agent`)" | ||
| - "traefik.http.services.data_quality_agent_bbmri.loadbalancer.server.port=8080" |
There was a problem hiding this comment.
From what I can see through docker inspecting the image this service runs on 8082 no?
There was a problem hiding this comment.
fixed. Is the path prefix correct/conventional for the BH?
You can just use point it to "http://bridgehead-bbmri-blaze:8080/fhir" (with or without the /fhir whatever you like) its a bbmri module so there should always be a blaze under that url without auth because its doing docker networking and not going via traefik |
added as an ENV |
|
Can we test the config on BBMRI-ERIC dev BH? |
|
Hi, @TKussel could you please take a look at the PR? |
| - /etc/timezone:/etc/timezone:ro | ||
| volumes: | ||
| agent-data: | ||
| driver: local No newline at end of file |
There was a problem hiding this comment.
| driver: local |
There was a problem hiding this comment.
better yet, remove the volume and mount a folder (see above)
| REPORTING_SERVER_URL: ${DATA_QUALITY_SERVER_URL} | ||
| REPORTING_SERVER_NAME: ${DATA_QUALITY_SERVER_NAME} |
There was a problem hiding this comment.
Does your app handle the case where these are set to "" gracefully? Just asking because we had issues with this for our components often times.
| depends_on: | ||
| - "blaze" | ||
| volumes: | ||
| - agent-data:/app/data |
There was a problem hiding this comment.
Please don't use a docker volume but use a folder mount in /var/cache/bridgehead/
| - /etc/timezone:/etc/timezone:ro | ||
| volumes: | ||
| agent-data: | ||
| driver: local No newline at end of file |
There was a problem hiding this comment.
better yet, remove the volume and mount a folder (see above)
|
|
||
| #### Sharing Data Quality Reports (recommended) | ||
|
|
||
| We strongly encourage sharing your data quality reports with the central BBMRI-ERIC quality dashboard. The reports contain only aggregated, non-patient-identifiable statistics and help the network to monitor and improve overall data quality. |
There was a problem hiding this comment.
| We strongly encourage sharing your data quality reports with the central BBMRI-ERIC quality dashboard. The reports contain only aggregated, non-patient-identifiable statistics and help the network to monitor and improve overall data quality. | |
| We encourage sharing your data quality reports with the central BBMRI-ERIC quality dashboard. The reports contain only aggregated, non-patient-identifiable statistics and help the network to monitor and improve overall data quality. However, quality reporting is completely optional and opt-in. |
Sorry, took me a while. Thank you for continually improving the PR. I only had some nitpicks regarding the volume remaining :) |
We currently don't have the ghcr in Bridgehead's required endpoint list so, I replicated the latest tag to our registry: https://docker.verbis.dkfz.de/cache/bbmri-cz/data-quality-server:latest |
3 participants
Adds an opt-in option to install the Data Quality Agent as part of the BH