IGH Data Transformation

Developer Getting Started

Prerequisites

• UV - Fast Python package manager (manages Python versions automatically)

Installation

1. Install UV (if not already installed):

   curl -LsSf https://astral.sh/uv/install.sh | sh

2. Clone the repository:

   git clone <repository-url>
   cd igh-data-transform

3. Install Python and dependencies:

   # UV will automatically install Python 3.12 if needed
   uv sync

Running the Application

You can run the CLI tool without activating the virtual environment using uv run:

# Run the CLI directly
uv run igh-data-transform

# Or run Python scripts directly
uv run python -m igh_data_transform

Alternatively, activate the virtual environment first:

source .venv/bin/activate  # On Linux/Mac
# or
.venv\Scripts\activate     # On Windows

# Then run normally
igh-data-transform

Pulling Data from Dataverse

This project uses igh-data-sync to pull data from Microsoft Dataverse before applying transformations.

Setup:

1. Configure environment variables - Create a .env file with your Dataverse credentials:

   CLIENT_ID=your-azure-client-id
   CLIENT_SECRET=your-azure-client-secret
   SCOPE=https://your-org.crm.dynamics.com/.default
   API_URL=https://your-org.api.crm.dynamics.com/api/data/v9.2/
   SQLITE_DB_PATH=./data/dataverse.db

2. Run the sync - Pull data from Dataverse to local SQLite:

   uv run sync-dataverse

3. Verify the data (optional) - Check foreign key integrity:

   uv run sync-dataverse --verify

The synced data will be stored in a SQLite database with SCD2 (Slowly Changing Dimension Type 2) versioning for historical tracking.

Development Workflow

The project uses UV for dependency management. Common commands:

• Add a dependency: uv add <package-name>
• Add a dev dependency: uv add --dev <package-name>
• Update dependencies: uv sync
• Run commands without activating venv: uv run <command>
• Run Python scripts: uv run python <script.py>