|
| 1 | +# Provisioning PVCs on the File Storage with Lustre Service |
| 2 | + |
| 3 | +The Oracle Cloud Infrastructure File Storage with Lustre service is a fully managed storage service designed to meet the demands of AI/ML training and inference, and high performance computing needs. You use the Lustre CSI plugin to connect clusters to file systems in the File Storage with Lustre service. |
| 4 | + |
| 5 | +You can use the File Storage with Lustre service to provision persistent volume claims (PVCs) by manually creating a file system in the File Storage with Lustre service, then defining and creating a persistent volume (PV) backed by the new file system, and finally defining a new PVC. When you create the PVC, Kubernetes binds the PVC to the PV backed by the File Storage with Lustre service. |
| 6 | + |
| 7 | +The Lustre CSI driver is the overall software that enables Lustre file systems to be used with Kubernetes via the Container Storage Interface (CSI). The Lustre CSI plugin is a specific component within the driver, responsible for interacting with the Kubernetes API server and managing the lifecycle of Lustre volumes. |
| 8 | + |
| 9 | +Note the following: |
| 10 | + |
| 11 | +- The Lustre CSI driver is supported on Oracle Linux 8 x86 and on Ubuntu x86 22.04. |
| 12 | +- To use a Lustre file system with a Kubernetes cluster, the Lustre client package must be installed on worker nodes that have to mount the file system. For more information about Lustre clients, see [Mounting and Accessing a Lustre File System](https://docs.oracle.com/iaas/Content/lustre/file-system-connect.htm). |
| 13 | + |
| 14 | +## Provisioning a PVC on an Existing File System |
| 15 | + |
| 16 | +To create a PVC on an existing file system in the File Storage with Lustre service (using Oracle-managed encryption keys to encrypt data at rest): |
| 17 | + |
| 18 | +1. Create a file system in the File Storage with Lustre service, selecting the Encrypt using Oracle-managed keys encryption option. See [Creating a Lustre File System](https://docs.oracle.com/iaas/Content/lustre/file-system-create.htm). |
| 19 | + |
| 20 | +2. Create security rules in either a network security group (recommended) or a security list for both the Lustre file system, and for the cluster's worker nodes subnet. The security rules to create depend on the relative network locations of the Lustre file system and the worker nodes which act as the client, according to the following scenarios: |
| 21 | + |
| 22 | + These scenarios, the security rules to create, and where to create them, are fully described in the File Storage with Lustre service documentation (see [Required VCN Security Rules](https://docs.oracle.com/iaas/Content/lustre/security-rules.htm)). |
| 23 | + |
| 24 | +3. Create a PV backed by the file system in the File Storage with Lustre service as follows: |
| 25 | + |
| 26 | + a. Create a manifest file to define a PV and in the `csi:` section, set: |
| 27 | + |
| 28 | + - `driver` to `lustre.csi.oraclecloud.com` |
| 29 | + - `volumeHandle` to `<MGSAddress>@<LNetName>:/<MountName>` |
| 30 | + where: |
| 31 | + - `<MGSAddress>` is the Management service address for the file system in the File Storage with Lustre service |
| 32 | + - `<LNetName>` is the LNet network name for the file system in the File Storage with Lustre service |
| 33 | + - `<MountName>` is the mount name used while creating the file system in the File Storage with Lustre service |
| 34 | + |
| 35 | + For example: `10.0.2.6@tcp:/testlustrefs` |
| 36 | + |
| 37 | + - `fsType` to `lustre` |
| 38 | + - (optional, but recommended) `volumeAttributes.setupLnet` to `"true"` if you want the Lustre CSI driver to perform lnet (Lustre Network) setup before mounting the filesystem |
| 39 | + - (required) `volumeAttributes.lustreSubnetCidr` to the CIDR block of the subnet where the worker node's VNIC having access to lustre filesystem is located (typically worker node subnet in default setup) to ensure the worker node has network connectivity to the Lustre file system. For example, 10.0.2.0/24. |
| 40 | + - (optional) `volumeAttributes.lustrePostMountParameters` to set Lustre parameters. For example: |
| 41 | + ```yaml |
| 42 | + volumeAttributes: |
| 43 | + lustrePostMountParameters: '[{"*.*.*MDT*.lru_size": 11200},{"at_history" : 600}]' |
| 44 | + ``` |
| 45 | +
|
| 46 | + For example, the following manifest file (named `lustre-pv-example.yaml`) defines a PV called `lustre-pv-example` backed by a Lustre file system: |
| 47 | + |
| 48 | + ```yaml |
| 49 | + apiVersion: v1 |
| 50 | + kind: PersistentVolume |
| 51 | + metadata: |
| 52 | + name: lustre-pv-example |
| 53 | + spec: |
| 54 | + capacity: |
| 55 | + storage: 31Ti |
| 56 | + volumeMode: Filesystem |
| 57 | + accessModes: |
| 58 | + - ReadWriteMany |
| 59 | + persistentVolumeReclaimPolicy: Retain |
| 60 | + csi: |
| 61 | + driver: lustre.csi.oraclecloud.com |
| 62 | + volumeHandle: "10.0.2.6@tcp:/testlustrefs" |
| 63 | + fsType: lustre |
| 64 | + volumeAttributes: |
| 65 | + setupLnet: "true" |
| 66 | + ``` |
| 67 | + |
| 68 | + b. Create the PV from the manifest file by entering: |
| 69 | + ```bash |
| 70 | + kubectl apply -f <filename> |
| 71 | + ``` |
| 72 | + |
| 73 | + For example: |
| 74 | + ```bash |
| 75 | + kubectl apply -f lustre-pv-example.yaml |
| 76 | + ``` |
| 77 | + |
| 78 | + c. Verify that the PV has been created successfully by entering: |
| 79 | + ```bash |
| 80 | + kubectl get pv <pv-name> |
| 81 | + ``` |
| 82 | + |
| 83 | + For example: |
| 84 | + ```bash |
| 85 | + kubectl get pv lustre-pv-example |
| 86 | + ``` |
| 87 | + |
| 88 | + Example output: |
| 89 | + ``` |
| 90 | + NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE |
| 91 | + lustre-pv-example 31Ti RWX Retain Bound 56m |
| 92 | + ``` |
| 93 | + |
| 94 | +4. Create a PVC that is provisioned by the PV you have created, as follows: |
| 95 | + |
| 96 | + a. Create a manifest file to define the PVC and set: |
| 97 | + |
| 98 | + - `storageClassName` to `""` |
| 99 | + |
| 100 | + **Note:** You must specify an empty value for `storageClassName`, even though storage class is not applicable in the case of static provisioning of persistent storage. If you do not specify an empty value for `storageClassName`, the default storage class (`oci-bv`) is used, which causes an error. |
| 101 | + |
| 102 | + - `volumeName` to the name of the PV you created (for example, `lustre-pv-example`) |
| 103 | + |
| 104 | + For example, the following manifest file (named `lustre-pvc-example.yaml`) defines a PVC named `lustre-pvc-example` that will bind to a PV named `lustre-pv-example`: |
| 105 | + |
| 106 | + ```yaml |
| 107 | + apiVersion: v1 |
| 108 | + kind: PersistentVolumeClaim |
| 109 | + metadata: |
| 110 | + name: lustre-pvc-example |
| 111 | + spec: |
| 112 | + accessModes: |
| 113 | + - ReadWriteMany |
| 114 | + storageClassName: "" |
| 115 | + volumeName: lustre-pv-example |
| 116 | + resources: |
| 117 | + requests: |
| 118 | + storage: 31Ti |
| 119 | + ``` |
| 120 | + |
| 121 | + **Note:** The `requests: storage:` element must be present in the PVC's manifest file, and its value must match the value specified for the `capacity: storage:` element in the PV's manifest file. Apart from that, the value of the `requests: storage:` element is ignored. |
| 122 | + |
| 123 | + b. Create the PVC from the manifest file by entering: |
| 124 | + ```bash |
| 125 | + kubectl apply -f <filename> |
| 126 | + ``` |
| 127 | + |
| 128 | + For example: |
| 129 | + ```bash |
| 130 | + kubectl apply -f lustre-pvc-example.yaml |
| 131 | + ``` |
| 132 | + |
| 133 | + c. Verify that the PVC has been created and bound to the PV successfully by entering: |
| 134 | + ```bash |
| 135 | + kubectl get pvc <pvc-name> |
| 136 | + ``` |
| 137 | + |
| 138 | + For example: |
| 139 | + ```bash |
| 140 | + kubectl get pvc lustre-pvc-example |
| 141 | + ``` |
| 142 | + |
| 143 | + Example output: |
| 144 | + ``` |
| 145 | + NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE |
| 146 | + lustre-pvc-example Bound lustre-pv-example 31Ti RWX 57m |
| 147 | + ``` |
| 148 | + |
| 149 | + The PVC is bound to the PV backed by the File Storage with Lustre service file system. Data is encrypted at rest, using encryption keys managed by Oracle. |
| 150 | + |
| 151 | +5. Use the new PVC when creating other objects, such as deployments. For example: |
| 152 | + |
| 153 | + a. Create a manifest named `lustre-app-example-deployment.yaml` to define a deployment named `lustre-app-example-deployment` that uses the `lustre-pvc-example` PVC, as follows: |
| 154 | + |
| 155 | + ```yaml |
| 156 | + apiVersion: apps/v1 |
| 157 | + kind: Deployment |
| 158 | + metadata: |
| 159 | + name: lustre-app-example-deployment |
| 160 | + spec: |
| 161 | + selector: |
| 162 | + matchLabels: |
| 163 | + app: lustre-app-example |
| 164 | + replicas: 2 |
| 165 | + template: |
| 166 | + metadata: |
| 167 | + labels: |
| 168 | + app: lustre-app-example |
| 169 | + spec: |
| 170 | + containers: |
| 171 | + - args: |
| 172 | + - -c |
| 173 | + - while true; do echo $(date -u) >> /lustre/data/out.txt; sleep 60; done |
| 174 | + command: |
| 175 | + - /bin/sh |
| 176 | + image: busybox:latest |
| 177 | + imagePullPolicy: Always |
| 178 | + name: lustre-app-example |
| 179 | + volumeMounts: |
| 180 | + - mountPath: /lustre/data |
| 181 | + name: lustre-volume |
| 182 | + restartPolicy: Always |
| 183 | + volumes: |
| 184 | + - name: lustre-volume |
| 185 | + persistentVolumeClaim: |
| 186 | + claimName: lustre-pvc-example |
| 187 | + ``` |
| 188 | + |
| 189 | + b. Create the deployment from the manifest file by entering: |
| 190 | + ```bash |
| 191 | + kubectl apply -f lustre-app-example-deployment.yaml |
| 192 | + ``` |
| 193 | + |
| 194 | + c. Verify that the deployment pods have been created successfully and are running by entering: |
| 195 | + ```bash |
| 196 | + kubectl get pods |
| 197 | + ``` |
| 198 | + |
| 199 | + Example output: |
| 200 | + ``` |
| 201 | + NAME READY STATUS RESTARTS AGE |
| 202 | + lustre-app-example-deployment-7767fdff86-nd75n 1/1 Running 0 8h |
| 203 | + lustre-app-example-deployment-7767fdff86-wmxlh 1/1 Running 0 8h |
| 204 | + ``` |
| 205 | + |
| 206 | +## Provisioning a PVC on an Existing File System with Mount Options |
| 207 | + |
| 208 | +You can optimize the performance and control access to an existing Lustre file system by specifying mount options for the PV. Specifying mount options enables you to fine-tune how pods interact with the file system. |
| 209 | + |
| 210 | +To include mount options: |
| 211 | + |
| 212 | +1. Start by following the instructions in [Provisioning a PVC on an Existing File System](#provisioning-a-pvc-on-an-existing-file-system). |
| 213 | + |
| 214 | +2. In the PV manifest described in [Provisioning a PVC on an Existing File System](#provisioning-a-pvc-on-an-existing-file-system), add the `spec.mountOptions` field, which enables you to specify how the PV should be mounted by pods. |
| 215 | + |
| 216 | + For example, in the `lustre-pv-example.yaml` manifest file shown in [Provisioning a PVC on an Existing File System](#provisioning-a-pvc-on-an-existing-file-system), you can include the `mountOptions` field as follows: |
| 217 | + |
| 218 | + ```yaml |
| 219 | + apiVersion: v1 |
| 220 | + kind: PersistentVolume |
| 221 | + metadata: |
| 222 | + name: lustre-pv-example |
| 223 | + spec: |
| 224 | + capacity: |
| 225 | + storage: 31Ti |
| 226 | + volumeMode: Filesystem |
| 227 | + accessModes: |
| 228 | + - ReadWriteMany |
| 229 | + persistentVolumeReclaimPolicy: Retain |
| 230 | + mountOptions: |
| 231 | + - ro |
| 232 | + csi: |
| 233 | + driver: lustre.csi.oraclecloud.com |
| 234 | + volumeHandle: "10.0.2.6@tcp:/testlustrefs" |
| 235 | + fsType: lustre |
| 236 | + volumeAttributes: |
| 237 | + setupLnet: "true" |
| 238 | + ``` |
| 239 | + |
| 240 | + In this example, the `mountOptions` field is set to `ro`, indicating that pods are to have read-only access to the file system. For more information about PV mount options, see [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) in the Kubernetes documentation. |
| 241 | + |
| 242 | +## Encrypting Data At Rest on an Existing File System |
| 243 | + |
| 244 | +The File Storage with Lustre service always encrypts data at rest, using Oracle-managed encryption keys by default. However, you have the option to specify at-rest encryption using your own master encryption keys that you manage yourself in the Vault service. |
| 245 | + |
| 246 | +For more information about creating File Storage with Lustre file systems that use Oracle-managed encryption keys or your own master encryption keys that you manage yourself, see [Updating File System Encryption](https://docs.oracle.com/iaas/Content/lustre/file-system-encryption.htm). |
0 commit comments