Remove SDK (#2657)

* Remove SDK

Signed-off-by: Eoin Fennessy <[email protected]>

* Fix e2e test

Signed-off-by: Eoin Fennessy <[email protected]>

* Remove unnecessary `pip install ./sdk` for unit tests

Signed-off-by: Eoin Fennessy <[email protected]>

* Remove sdk from pre-commit config

Signed-off-by: Eoin Fennessy <[email protected]>

* Update docstring for `generate` target

Signed-off-by: Eoin Fennessy <[email protected]>

* Update CONTRIBUTING.md

Signed-off-by: Eoin Fennessy <[email protected]>

* Remove .gitignore for OpenAPI Generator CLI JAR

Signed-off-by: Eoin Fennessy <[email protected]>

* README.md: Add link to `kubeflow/sdk` + some minor edits

Signed-off-by: Eoin Fennessy <[email protected]>

* Add TODO to repurpose SDK release info if needed

Signed-off-by: Eoin Fennessy <[email protected]>

* Add SDK installation step to all notebooks

Signed-off-by: Eoin Fennessy <[email protected]>

* Comment out SDK installation command in all notebooks

Signed-off-by: Eoin Fennessy <[email protected]>

---------

Signed-off-by: Eoin Fennessy <[email protected]>

Author: eoinfennessy

.github/workflows/test-e2e.yaml

@@ -42,7 +42,7 @@ jobs:
           pip install papermill==2.6.0 jupyter==1.1.1 ipykernel==6.29.5
 
           echo "Install Kubeflow SDK"
-          pip install ./sdk
+          pip install git+https://github.com/kubeflow/sdk.git@main#subdirectory=python
 
       - name: Setup cluster
         run: |

.gitignore

@@ -20,9 +20,6 @@ artifacts
 __pycache__/
 *.egg-info/
 
-# OpenAPI Generator CLI JAR file
-hack/python-sdk/openapi-generator-cli.jar
-
 # Coverage
 cover.out
 

.pre-commit-config.yaml

@@ -26,9 +26,5 @@ exclude: |
   (?x)^(
     docs/images/.*|
     pkg/client/.*|
-    sdk/kubeflow/trainer/__init__.py|
-    sdk/kubeflow/trainer/api/__init__.py|
-    sdk/kubeflow/trainer/models/.*|
     api/python_api/kubeflow_trainer_api/models/.*|
-    sdk/docs/.*
   )$

CONTRIBUTING.md

@@ -21,7 +21,7 @@ Note for Lima the link is to the Adopters, which supports several different cont
 The Kubeflow Trainer project includes a Makefile with several helpful commands to streamline your development workflow:
 
 ```sh
-# Generate manifests, APIs and SDK
+# Generate manifests and APIs.
 make generate
 ```
 
@@ -215,24 +215,6 @@ kubectl logs -n kubeflow -l training.kubeflow.org/job-name=pytorch-simple --foll
 
 ### SDK Development
 
-To generate Python SDK for the operator, run:
+Changes to the Kubeflow Trainer Python SDK can be made in the https://github.com/kubeflow/sdk repo.
 
-```sh
-./hack/python-sdk/gen-sdk.sh
-```
-
-This command will re-generate the api and model files together with the documentation and model tests.
-The following files/folders in `sdk/python` are auto-generated and should not be modified directly:
-
-```
-sdk/python/docs
-sdk/python/kubeflow/training/models
-sdk/python/kubeflow/training/*.py
-sdk/python/test/*.py
-```
-
-The Training Operator client and public APIs are located here:
-
-```
-sdk/python/kubeflow/training/api
-```
+The Trainer SDK can be found at https://github.com/kubeflow/sdk/tree/main/python/kubeflow/trainer.

Makefile

@@ -120,13 +120,11 @@ manifests: controller-gen ## Generate manifests.
 		output:rbac:artifacts:config=manifests/base/rbac \
 		output:webhook:artifacts:config=manifests/base/webhook
 
-## TODO (kramaranya): Remove gen-sdk.sh when moving SDK
 .PHONY: generate
-generate: go-mod-download manifests ## Generate APIs and SDK.
+generate: go-mod-download manifests ## Generate APIs.
 	$(CONTROLLER_GEN) object:headerFile="hack/boilerplate/boilerplate.go.txt" paths="./pkg/apis/..."
 	hack/update-codegen.sh
 	CONTAINER_RUNTIME=$(CONTAINER_RUNTIME) hack/python-api/gen-api.sh
-	CONTAINER_RUNTIME=$(CONTAINER_RUNTIME) hack/python-sdk/gen-sdk.sh
 
 .PHONY: go-mod-download
 go-mod-download: ## Run go mod download to download modules.
@@ -163,7 +161,6 @@ test-integration: ginkgo envtest jobset-operator-crd scheduler-plugins-crd ## Ru
 test-python: ## Run Python unit test.
 	pip install pytest
 	pip install -r ./cmd/initializers/dataset/requirements.txt
-	pip install ./sdk
 
 	PYTHONPATH=$(PROJECT_DIR) pytest ./pkg/initializers/dataset
 	PYTHONPATH=$(PROJECT_DIR) pytest ./pkg/initializers/model

README.md

@@ -20,8 +20,9 @@ You can integrate other ML libraries such as [HuggingFace](https://huggingface.c
 [DeepSpeed](https://github.com/microsoft/DeepSpeed), or [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)
 with Kubeflow Training to orchestrate their ML training on Kubernetes.
 
-Kubeflow Trainer allows you effortlessly develop your LLMs with the Kubeflow Python SDK and
-build Kubernetes-native Training Runtimes with Kubernetes Custom Resources APIs.
+Kubeflow Trainer enables you to effortlessly develop your LLMs with the
+[Kubeflow Python SDK](https://github.com/kubeflow/sdk/), and build Kubernetes-native Training
+Runtimes using Kubernetes Custom Resource APIs.
 
 <h1 align="center">
     <img src="./docs/images/trainer-tech-stack.drawio.svg" alt="logo" width="500">

docs/release/README.md

@@ -81,6 +81,9 @@ cherry pick your changes from the `master` branch and submit a PR.
 
 ### Release Training SDK
 
+TODO (eoinfennessy): This information no longer applies to Trainer V2 after moving the SDK to kubeflow/sdk.
+However, we should repurpose this information if we ever decide to publish the Trainer APIs.
+
 1. Update the the VERSION in the [gen-sdk.sh script](../../hack/python-sdk/gen-sdk.sh#L27),
    packageVersion in the [swagger_config.json file](../../hack/python-sdk/swagger_config.json#L4),
    and the version in the [setup.py file](../../sdk/python/setup.py#L36).

examples/deepspeed/text-summarization/T5-Fine-Tuning.ipynb

@@ -20,21 +20,40 @@
    ]
   },
   {
+   "metadata": {},
    "cell_type": "markdown",
-   "id": "b5ad0e01-d893-484c-8988-d25c2f322b80",
+   "source": [
+    "## Install the Kubeflow SDK\n",
+    "\n",
+    "You need to install the Kubeflow SDK to interact with Kubeflow Trainer APIs:"
+   ],
+   "id": "1c461b2984e77d99"
+  },
+  {
+   "metadata": {},
+   "cell_type": "code",
+   "outputs": [],
+   "execution_count": null,
+   "source": "# !pip install git+https://github.com/kubeflow/sdk.git@main#subdirectory=python",
+   "id": "4900404c5d532bdf"
+  },
+  {
    "metadata": {},
+   "cell_type": "markdown",
    "source": [
     "## Create Script to Fine-Tune T5 with DeepSpeed\n",
     "\n",
     "We need to wrap our fine-tuning script into a function to create Kubeflow TrainJob."
-   ]
+   ],
+   "id": "47534ee4955f3ff6"
   },
   {
+   "metadata": {
+    "jupyter": {
+     "is_executing": true
+    }
+   },
    "cell_type": "code",
-   "execution_count": null,
-   "id": "57952ce1-5752-4976-8a35-c71d25935b74",
-   "metadata": {},
-   "outputs": [],
    "source": [
     "def deepspeed_train_t5(args):\n",
     "    import os\n",
@@ -208,7 +227,10 @@
     "        file_path = os.path.join(HOME_PATH, \"global_step94/mp_rank_00_model_states.pt\")\n",
     "        bucket = boto3.resource(\"s3\").Bucket(args[\"BUCKET\"])\n",
     "        bucket.upload_file(file_path, f\"deepspeed/{file_path}\")"
-   ]
+   ],
+   "id": "35f06c45b614ecd0",
+   "outputs": [],
+   "execution_count": null
   },
   {
    "cell_type": "markdown",

examples/mlx/image-classification/MLX-Distributed-Mnist.ipynb

@@ -66,6 +66,24 @@
     "```"
    ]
   },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": [
+    "## Install the Kubeflow SDK\n",
+    "\n",
+    "You need to install the Kubeflow SDK to interact with Kubeflow Trainer APIs:"
+   ],
+   "id": "3cb9ab4c8e12221a"
+  },
+  {
+   "metadata": {},
+   "cell_type": "code",
+   "outputs": [],
+   "execution_count": null,
+   "source": "# !pip install git+https://github.com/kubeflow/sdk.git@main#subdirectory=python",
+   "id": "bd62189280760f42"
+  },
   {
    "cell_type": "markdown",
    "id": "b5ad0e01-d893-484c-8988-d25c2f322b80",

examples/pytorch/image-classification/mnist.ipynb

@@ -31,10 +31,7 @@
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": [
-    "# TODO (astefanutti): Change to the Kubeflow SDK when it's available.\n",
-    "# !pip install git+https://github.com/kubeflow/trainer.git@master#subdirectory=sdk"
-   ]
+   "source": "# !pip install git+https://github.com/kubeflow/sdk.git@main#subdirectory=python"
   },
   {
    "cell_type": "markdown",