[FEATURE] data collection and download feature

[ututono]

Is your feature request related to a problem? Please describe

CARE already provides a one-click full data download, but the raw export requires significant manual post-processing before it can be used for research. Administrators must write and run custom Python scripts to apply consent filtering and anonymization, there is no built-in mechanism to enforce privacy consent or anonymize user identities before sharing data with researchers.

Describe the solution you'd like

A job-based data export pipeline in the CARE admin UI that allows administrators to configure and trigger async export jobs scoped by instance profile and time range. The output is a downloadable ZIP containing a JSONL file (one package per paper or student), optional PDF documents, and a metadata file, with consent filtering and anonymization applied under the hood automatically according to each instance's policy.

Describe alternatives you've considered

Using the existing one-click full data download combined with manually running Python scripts for anonymization and consent filtering was considered, but this approach requires per-instance scripting effort, is not accessible to non-technical administrators, and provides no audit log. Standardizing this into a built-in pipeline removes the dependency on individual scripts and makes the process consistent across all instances.

Additional context

Details are list in the comments below

[ututono]

1. Goal

Enable admins of every CARE instance to export research data from their review workflows in a flexible, privacy-compliant, and analysis-ready format. The export functionality must support three distinct workflows: (1) Thesis workflow (student thesis review), (2) Internal Review workflow (peer review) and (3) EiwA course ("Einführung in wissenschaftliches Arbeiten") workflow, while enforcing consent-based filtering and privacy-preserving anonymization.

2. Highlevel Overview

We are going to implement a job-based export pipeline:

• An admin triggers an export with filters (instance/profile, time range).
• CARE enqueues an async export worker that extracts data from database, applies permission checks, consent policies, and anonymization.
• The worker writes a ZIP file to storage
• The ZIP contains looks like:

export_{job_id}.zip
├── export.jsonl              # one package per line
├── metadata.json             # job config, filter params, stats, timestamp, CARE version
└── documents/                # original pdf optional
    ├── anon_doc_001.pdf      # filename is the anonymized document ID
    ├── anon_doc_002.pdf
    └── ...

To support multiple workflows, exports are configured via three profile types:

Profile Type	Usage
Instance Profiles	Defines the export unit and included data domains per instance (e.g., internal_review, thesis_expose)
Consent Policies	Defines how author/reviewer consent affects inclusion or redaction of data
Schema Profiles	Defines the output JSONL structure for each instance

3. Must-haves

3.1 Frontend

3.1.1 Export Entry Point:

Requirements:

• Dedicated export section in the admin UI for every instance
• Entry point visible only to administrators.

Acceptance criteria:

• Export entry point is visible in the admin UI for every instance and hidden from non-admin users
• Instance/profile selector correctly updates visible scope options and split configuration to match the selected profile

3.1.2 Export Job Creation:

Requirements:

• Instance/profile selection (determines export unit, domains, consent policy)
• Scope filters:
  • Time range (created_at / updated_at)
• Train/test split configuration (e.g., 80/20 random split or manual assignment)

Acceptance criteria:

• Time range filter (created_at / updated_at) correctly scopes the preview record count before job submission
• Train/test split configuration accepts both ratio input (e.g. 80/20) and manual assignment; invalid inputs are rejected with a clear error message

3.1.3 External Consent Sheet Upload:

Requirements:

• Admin can upload a CSV file containing consent records from external sources (Moodle, Google Sheets, internal review website)
• System maps external identifiers to CARE user accounts (manual selection)
• Upload validation: report unmatched rows, duplicate entries, missing fields

Acceptance criteria:

• Admin can upload a consent CSV via drag-and-drop or file picker; the system displays a per-row import summary (matched / unmatched / duplicate / invalid) without aborting on partial failures
• Unmatched rows in the consent CSV are clearly identified with the original identifier value so the admin can correct and re-upload

3.1.4 Export Job List and Status View.:

Requirements:

• Table listing all export jobs with: job ID, instance profile, created date, created by, status, and download button
• Status lifecycle: queued → running → finished / failed
• Download button appears when status is finished
• Error message shown when status is failed

Acceptance criteria:

• Job list table displays job ID, instance profile, created date, created by, status badge, and download button for all jobs scoped to the current admin
• Job status badge updates in real-time (queued → running → finished / failed) without requiring a page refresh
• Download button is visible and functional only when job status is finished; clicking it triggers an authenticated file download of the ZIP file
• A failed job displays an informative error message (not a raw stack trace) inline in the job list row

3.2 Backend

3.2.1 Instance Profiles

Requirements
The data requirements for each instance as summarized as below table, the detail interview minute and requuirements have been recorded into this docuemnet (Avaibiable for internal developer).

	Thesis Workflow	Internal Review Workflow	EiwA course
Export unit	Per student (all document versions + annotations + comments)	Per paper (all annotations + comments)	Per student (all document versions + annotations + comments)
Data structure	Nested JSON aggregating all textual data, and pdf files are stored in another folder	Nested JSON aggregating all textual data	Nested JSON aggregating all textual data, and pdf files are stored in another folder
Consent	Single student consent (two types: data sharing & behavior data)	Dual consent (author AND reviewer must both agree)	Single student consent (two types: data sharing & behavior data)
Anonymization	Anonymize both students and reviewers	Anonymize reviewers only, maintain within-paper consistency	Anonymize both students and reviewers
Document content	Need	NOT exported (annotations/comments only)	Need

An exmample about profile for internl review instance:

{
  "profile_id": "internal_review",
  "export_unit": "paper",
  "included_domains": ["annotations", "comments"],
  "export_pdf": "false",
  "consent_policy": "dual_consent",
  "schema_profile": "internal_review_v1"
}

where:

• the field export_pdf controls how document content is exported:
  • false → Document content not exported (Internal Review)
  • true→ PDF bundled into documents/ directory in ZIP; JSONL references it via file_ref
• the field consent_policy controls what data should be exported based on user's consent. Details can be found in the below section Consent Policies.

Acceptance criteria:

• Each instance has exactly one active profile
• export_pdf: true causes the worker to bundle PDFs into documents/ and replace content fields in JSONL with file_ref objects pointing to the correct filenames
• Changing a profile's consent_policy field correctly routes the job to the corresponding consent logic without code changes to the worker
• Adding a new instance profile (e.g., for EiwA) requires only a new config entry and no changes to the pipeline code

3.2.2 Consent Policies

Requirements:
According to data requirements of each instances, there are two different consent strategies:

• dual_consent (Internal Review)

  • Both author and reviewer must have consent_given = true
  • If author consent is missing or false → exclude entire paper package
  • If reviewer consent is missing or false → redact that reviewer's annotations and comments (i.e., replace them with null); keep paper with remaining reviewers

• student_dual_type (Thesis Exposé)

  • Student must give two separate consents:
    • data_sharing → controls inclusion of document text, annotations, and comments
    • behavior_data → controls inclusion of interaction traces
  • If data_sharing = false → two options: (1) exclude all text content for that student (2) replace content with null.
  • If student has no consent record → treat as declined by default

Acceptance criteria:

• Consent filtering is applied before anonymization in the pipeline, so redacted fields never reach the anonymization step

3.2.3 Anonymization

Requirements:

• Replace all user-identifying fields (name, email, username) with anonymous string.
  • Admins can select which fields should be replaced with traceable pseudo-IDs; otherwise, a constant (e.g., "FAKE@EMAIL" for email field) will be used as the substitute.
• Anonymization applied to:
  • User metatdata
  • Document metadata
  • Author/reviewer references in annotations and comments

Acceptance criteria:

• All user-identifying fields (name, email, username) are replaced in user metadata, document metadata, and all author/reviewer references within annotations and comments
• When an admin selects a field for traceable pseudo-IDs, the same real identity maps to the same pseudo-ID consistently within the same export job (unique and tracable)
• When a field is not selected for pseudo-ID replacement, it is substituted with the designated constant (e.g., "FAKE@EMAIL" for email fields)
• Re-running the same export job produces identical anonymous IDs for the same source identities (deterministic mapping)

3.2.4 Async Export Worker

Requirements:
Load the relevant settings and rules, pull only the allowed data, and verify the requester is permitted to export it. Then enforce consent choices, anonymize sensitive information, group the results into the intended export units, and format them into the required output structure. Finally, package everything into a single export file and mark the job as successful or failed.

• Worker pipeline steps:
  1. Load instance profile and consent policy
  2. Query data from database with scope filters applied
  3. Validate admin permission scope
  4. Apply consent policy → include, redact, or exclude packages
     Apply anonymization
  5. Aggregate into export units (per paper / per student)
  6. Apply schema profile → shape into JSONL
  7. Write ZIP to /files/exports/{job_id}/export.zip
  8. Update job status to finished or failed

Acceptance criteria:

• A job picked up by the worker transitions from queued to running immediately, and the status change is visible to the frontend.
• If any pipeline step fails, the job status is set to failed with a descriptive error message; no partial ZIP file is written or made available for download
• The completed ZIP is written to /files/exports/{job_id}/export.zip and the job record is updated to finished with the ZIP file path in the same transaction

3.2.5 JSONL Output Schema

Requirements:
Each line in export.jsonl is one package. Example structures:

Example package for Internal Review:

{
  "package_type": "paper_review",
  "paper_id": "anon_paper_001",
  "study_metadata": { "study_id": "...", "start_date": "...", "end_date": "..." },
  "document_metadata": { "title": "...", "type": "...", "version": "..." },
  "reviewers": [
    {
      "reviewer_id": "Reviewer_A",
      "annotations": [{ "id": "...", "selector": "...", "text": "..." }],
      "comments": [{ "id": "...", "text": "...", "timestamp": "..." }]
    }
  ]
}

Example package for Thesis Exposé:

{
  "package_type": "student_expose",
  "user_id": "anon_student_001",
  "documents": [
    {
      "document_id": "anon_doc_001",
      "version": "v1",
      "content": {
        "type": "file_ref",
        "path": "documents/anon_doc_001.pdf"
      },
      "reviewers": [
        {
          "reviewer_id": "Reviewer_A",
          "annotations": [{"id": "...", "selector": "...", "text": "..."}],
          "comments": [{ "id": "...", "text": "...", "timestamp": "..." }]
        },
      ]
    },
  ]
}

Acceptance criteria:

• Every line in export.jsonl is valid JSON and independently parseable (no multi-line objects). The full export.jsonl file is parseable by pandas (pd.read_json(..., lines=True))
• The fields of row in the export.jsonl must strictly comply with the Schema.
• When export_pdf: true, the content field is a file_ref object whose path matches an actual file present in the documents/ directory of the same ZIP; otherwise, when export_pdf: false, the content field is set to null

3.2.5 Security & Access Control

Requirements:

• Only admins can create or view export jobs
• All export job creation and downloads are logged

Acceptance criteria:

• A non-admin user receives an authentication error when attempting to call any export-related socket event or the download endpoint
• Every export job creation and every ZIP download is written to the log with timestamp, admin user ID, and job ID

4. Nice-to-haves

• Data Quality Pre-check. Export only data created or modified since the last export
• Incremental Exports. Before running the job, show the admin a summary: how many packages would be included, how many excluded due to consent, how many have missing data.
• Reusable ZIP file. ZIP files stored server-side; download requires authenticated session (reuse the processed ZIP files to reduce working time for duplicate configuration). The ZIP files retained for a configurable period then deleted