Version: Next

Manage Workspace

Updated 2024.10.02

Topics

Manage Workspace

Detailed Steps

For detailed explanations of {variables}, refer to the Terminology page.

Workspace Description
- A Workspace is a logical space separated by project, accessible only to users affiliated with the project. (Admin can access all Workspaces)
  - Users are classified as system users (for Edge Conductor) and regular users.
- The Workspace matches the Namespace in Kubeflow, and only one User can access the Namespace.

1. Prerequisite

Environment setup for installation must be completed. (Refer to 1. Set Up Installation Environment)

export AWS_CLUSTER_NAME=
export AWS_DEFAULT_REGION=
export AWS_DEFAULT_REGION_ALIAS=
export INFRA_NAME=
export DEPLOY_ENV=
export PROJECT_NAME=
export AIC_FRONTEND_URL=
export WORKSPACE_NAME=${PROJECT_NAME}-ws
export PROJECT_DASHBOARD_HASH=
export PROJECT_KUBEFLOW_NAMESPACE_NAME=aic-ns-${WORKSPACE_NAME}
export PROJECT_KUBEFLOW_USER_NAME=aic-user-${WORKSPACE_NAME}
export PROJECT_KUBEFLOW_USER_PASSWD='$2a$12$A6GAI7xf1CjfPCF3MnycvuDcXUdP4O.Ruo7PvQUFUkmKGSYcCiieS'
export PROJECT_KUBEFLOW_USER_UNIQUE_ID=`$(echo date +"%Y%m%d%H%M%S")`
export PROJECT_S3_BUCKET_NAME=s3-${AWS_DEFAULT_REGION_ALIAS}-${INFRA_NAME}-${DEPLOY_ENV}-${PROJECT_NAME}
export PROJECT_NODEGROUP_SPEC=standard
export PROJECT_NODEGROUP_LABEL=${PROJECT_NAME}-ws-${PROJECT_NODEGROUP_SPEC}
export PROJECT_NODEGROUP_NAME=ng-${AWS_DEFAULT_REGION_ALIAS}-aicond-${PROJECT_NAME}-ws-${PROJECT_NODEGROUP_SPEC}
export PROJECT_NODEGROUP_DESIRED_SIZE=
export PROJECT_NODEGROUP_MIN=0
export PROJECT_NODEGROUP_MAX=
export PROJECT_NODEGROUP_EC2_NAME=
export PROJECT_NODEGROUP_EC2_VCPU=
export PROJECT_NODEGROUP_EC2_MEM=
export PROJECT_NODEGROUP_EC2_GPU=

2. Add Workspace

Adding a Workspace is installed in the following order:

Install Project infrastructure
- Kubernetes: Install Namespace and Profile
- Kubeflow: Create User
- Resource: Create S3 Bucket, add S3 Bucket permissions to Policy, create Nodegroup
Create Workspace in AI Conductor
Create Workspace system account in AI Conductor

2-1. Install Project Infrastructure

Create Namespace and Profile

NOTE: The Namespace to be used in the Project must not already exist.

Create a file called create-profile-namespace.yaml defining the creation of Namespace and Profile.

[Expand create-profile-namespace.yaml]

cat <<EOT > create-profile-namespace.yaml
apiVersion: kubeflow.org/v1beta1
kind: Profile
metadata:
  name: ${PROJECT_KUBEFLOW_NAMESPACE_NAME}
spec:
  owner:
    kind: User
    name: ${PROJECT_KUBEFLOW_USER_NAME}@aic.com
EOT

Create the Namespace and Profile with the following command.
```
kubectl apply -f create-profile-namespace.yaml
```

Verify the creation of Namespace and Profile

Check the owner in the Annotations

kubectl describe namespace ${PROJECT_KUBEFLOW_NAMESPACE_NAME}

# Output
Name:         ${PROJECT_KUBEFLOW_NAMESPACE_NAME}
Labels:       app.kubernetes.io/part-of=kubeflow-profile
              istio-injection=enabled
              katib.kubeflow.org/metrics-collector-injection=enabled
              kubernetes.io/metadata.name=${PROJECT_KUBEFLOW_NAMESPACE_NAME}
              pipelines.kubeflow.org/enabled=true
              serving.kubeflow.org/inferenceservice=enabled
Annotations:  owner: ${PROJECT_KUBEFLOW_USER_NAME}@aic.com
Status:       Active

No resource quota.

No LimitRange resource.

Create Kubeflow User Account

Obtain user information from dex

kubectl get configmap dex -n auth -o jsonpath='{.data.config\.yaml}' > dex-config.yaml

Add user account to dex

In dex-config.yaml, add the information of the Kubeflow user account to be created.

yq e -i '.staticPasswords += {"email" : env(PROJECT_KUBEFLOW_USER_NAME)+"@aic.com"}' dex-config.yaml
yq e -i '.staticPasswords[-1] += {"hash" : env(PROJECT_KUBEFLOW_USER_PASSWD)}' dex-config.yaml
yq e -i '.staticPasswords[-1] += {"username" : env(PROJECT_KUBEFLOW_USER_NAME)}' dex-config.yaml
yq e -i '.staticPasswords[-1] += {"userID" : "\"" + env(PROJECT_KUBEFLOW_USER_UNIQUE_ID) + "\""}' dex-config.yaml

Apply the user account to dex

kubectl create configmap dex --from-file=config.yaml=dex-config.yaml -n auth --dry-run -oyaml | kubectl apply -f -
kubectl rollout restart deployment dex -n auth

Create S3 Bucket
- Reference: Amazon Web Services Setup Guide | 3. Create S3 Bucket
- Navigate to the AWS S3 Console.
- Click Create bucket in the upper right corner.
- Step 1: General configuration
  - Bucket name: {PROJECT_S3_BUCKET_NAME}
- Step 2: Create bucket

Update Service Account Addition

Reference: Amazon Web Services Setup Guide | 5. Create Kubernetes Service Account

Add S3 Bucket control permissions to the Service Account used by the Kubeflow deploy user.

export POLICY_UPDATE_NAME=policy-s3-kubeflow-${AWS_DEFAULT_REGION_ALIAS}-${INFRA_NAME}-${DEPLOY_ENV}

Method for adding updates to {POLICY_UPDATE_NAME}

Save the currently applied Policy as a JSON format YAML file.

# Get the latest Policy Version
policy_version_id=$(aws iam get-policy --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/${POLICY_UPDATE_NAME} | jq -r '.Policy' | jq -r '.DefaultVersionId')

# Save the Policy JSON file (YAML)
aws iam get-policy-version --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/${POLICY_UPDATE_NAME} --version-id ${policy_version_id} | jq -r '.PolicyVersion' | jq -r '.Document' > ${POLICY_UPDATE_NAME}.yaml

Add {PROJECT_S3_BUCKET_NAME} to the Resource in {POLICY_UPDATE_NAME}.yaml as follows.

yq e -i '.Statement[0].Resource += "arn:aws:s3:::"+env(PROJECT_S3_BUCKET_NAME)' ${POLICY_UPDATE_NAME}.yaml
yq e -i '.Statement[0].Resource += "arn:aws:s3:::"+env(PROJECT_S3_BUCKET_NAME)+"/*"' ${POLICY_UPDATE_NAME}.yaml

Add the Policy Version for {POLICY_UPDATE_NAME}.

aws iam create-policy-version --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/${POLICY_UPDATE_NAME} --set-as-default --policy-document file://${POLICY_UPDATE_NAME}.yaml

If an error occurs while adding the Policy Version, resolve it as follows.

[Expand Policy Version Error]

Policy Version supports a maximum of 5 versions; thus, the oldest Policy Version must be deleted.

# Error Message
An error occurred (LimitExceeded) when calling the CreatePolicyVersion operation: A managed policy can have up to 5 versions. Before you create a new version, you must delete an existing version.

Example command to delete Policy Version #2

aws iam delete-policy-version --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/${POLICY_UPDATE_NAME} --version-id v2

Add S3 Bucket control permissions to the Service used in the AI Conductor Namespace.

export POLICY_UPDATE_NAME=policy-s3-aia-${AWS_DEFAULT_REGION_ALIAS}-${INFRA_NAME}-${DEPLOY_ENV}

Add updates to {POLICY_UPDATE_NAME} (same method as above)

Nodegroup Creation

Refer to the Nodegroup Creation page for details.

Create a create-nodegroup.yaml file defining the Nodegroup.
Put GPU instance name in {PROJECT_NODEGROUP_EC2_NAME} if you want to use one.

[Expand create-nodegroup.yaml]

NOTE: Setting "propagateASGTags: true" is mandatory.

NOTE: The availabilityZones are defined by {PROJECT_NODEGROUP_EC2_NAME}.

cat <<EOT > create-nodegroup.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
managedNodeGroups:
- amiFamily: AmazonLinux2
  desiredCapacity: ${PROJECT_NODEGROUP_DESIRED_SIZE}
  disableIMDSv1: false
  disablePodIMDS: false
  iam:
    withAddonPolicies:
      albIngress: false
      appMesh: false
      appMeshPreview: false
      autoScaler: true
      awsLoadBalancerController: false
      certManager: false
      cloudWatch: false
      ebs: false
      efs: false
      externalDNS: false
      fsx: false
      imageBuilder: false
      xRay: false
  instanceSelector: {}
  instanceType: ${PROJECT_NODEGROUP_EC2_NAME}
  labels:
    aic-role: ${PROJECT_NODEGROUP_LABEL}
    alpha.eksctl.io/cluster-name: ${AWS_CLUSTER_NAME}
    alpha.eksctl.io/nodegroup-name: ${PROJECT_NODEGROUP_NAME}
  maxSize: ${PROJECT_NODEGROUP_MAX}
  minSize: ${PROJECT_NODEGROUP_MIN}
  name: ${PROJECT_NODEGROUP_NAME}
  availabilityZones: ["${AWS_DEFAULT_REGION}a", "${AWS_DEFAULT_REGION}c"]
  privateNetworking: true
  releaseVersion: ""
  securityGroups:
    withLocal: null
    withShared: null
  ssh:
    allow: false
    publicKeyPath: ""
  tags:
    alpha.eksctl.io/nodegroup-name: ${PROJECT_NODEGROUP_NAME}
    alpha.eksctl.io/nodegroup-type: managed
  volumeIOPS: 3000
  volumeSize: 50
  volumeThroughput: 125
  volumeType: gp3
  propagateASGTags: true
metadata:
  name: ${AWS_CLUSTER_NAME}
  region: ${AWS_DEFAULT_REGION}
EOT

Use the following command to create the Nodegroup.

eksctl create nodegroup --config-file=create-nodegroup.yaml

[Expand Trouble Shooting: 'AccessConfig']

error getting cluster stack template: failed to parse GetStackTemplate response: json: unknown field "AccessConfig

Update eksctl following the instructions here and then run the command again.

2-2. Create Grafana Dashboard for Project

The monitoring infrastructure must be pre-installed. ([Install Resource Monitoring](.. /.. /installation/ai_conductor/resource_monitoring) For detailed instructions on dashboard settings, refer to the Manage Monitoring Dashboard page.

Check the hash value of the dashboard
- The URL of the imported dashboard contains the hash value, which is the separator of the dashboard. You can use the script below to get the dashboard hash value required to add a workspace.
```
url="URL of the imported dashboard"
PROJECT_DASHBOARD_HASH=$(echo "$url" | grep -o '[a-zA-Z0-9]\{13\}' | head -n 1)
```

2-3. Creating a Project Workspace

Access {AIC_FRONTEND_URL}
Log in with an admin account
- ID: {AIC_ADMIN_USER}
- Password: {AIC_ADMIN_PASSWD}
Click Settings at the top right
Click Workspace at the top left
Create a Workspace
- Click + New Workspace at the top right
  - Workspace Name: {PROJECT_NAME}
  - Bucket Name: {PROJECT_S3_BUCKET_NAME}
  - Dashboard Hash: {PROJECT_DASHBOARD_HASH}
  - Resources: (Input according to the created Nodegroup)
    - {PROJECT_NODEGROUP_SPEC}
    - vcpu: {PROJECT_NODEGROUP_EC2_VCPU}
    - ram_gb: {PROJECT_NODEGROUP_EC2_MEM}
    - gpu: {PROJECT_NODEGROUP_EC2_GPU}

2-4. Creating a System Account for the Project Workspace

For detailed explanations, refer to the Add Static Account and Add Role Account pages.

Access {AIC_FRONTEND_URL}
Log in with an admin account
- ID: {AIC_ADMIN_USER}
- Password: {AIC_ADMIN_PASSWD}
Select {WORKSPACE_NAME}
Click Settings at the top right
Create a system account
- Click Account at the top left
- Click + New Account at the top right
  - NOTE: The following three values are fixed and should not be changed.
  - ID: {PROJECT_NAME}-edge
  - Password: {PROJECT_NAME}-edge@com
  - Account Type: Developer
  - Click Save
Assign the created system account to {WORKSPACE_NAME}
- Click Workspace at the top left
- Click {WORKSPACE_NAME}
- Click Actions -> Manage Members at the top right
- Check {PROJECT_NAME}-edge
- Role in Workspace: Manager
- Click Save

3. Updating the Workspace (TBD)

Functionality updates are planned.

4. Deleting the Workspace

NOTE: All data related to the Workspace must be deleted.

The Workspace deletion process is as follows (the reverse of the 'Workspace addition' process):

Delete the Workspace system account in AI Conductor
Delete the Workspace in AI Conductor
Delete the Project infrastructure
- Resource: Delete S3 Bucket, Remove S3 Bucket permissions from Policy, Delete Nodegroup
- Kubeflow: Delete User
- Kubernetes: Delete Namespace and Profile

4-1. Deleting the Project Workspace System Account

For detailed explanations, refer to the Delete Role Account and Delete Account pages.

Access {AIC_FRONTEND_URL}
Log in with an admin account
- ID: {AIC_ADMIN_USER}
- Password: {AIC_ADMIN_PASSWD}
Select {WORKSPACE_NAME}
Click Settings at the top right
Unassign the created system account from {WORKSPACE_NAME}
- Click Workspace at the top left
- Click {WORKSPACE_NAME}
- Click Actions -> Manage Members at the top right
- Check {PROJECT_NAME}-edge
- Click - Delete at the top right
- Click OK
Delete the created system account
- Click Account at the top left
- Check {PROJECT_NAME}-edge
- Click Actions -> Delete Account at the top right
- Click OK

4-2. Deleting the Project Workspace

Access {AIC_FRONTEND_URL}
Log in with an admin account
- ID: {AIC_ADMIN_USER}
- Password: {AIC_ADMIN_PASSWD}
Click Settings at the top right
Click Workspace at the top left
Delete the Workspace
- Check {PROJECT_NAME}
- Click Actions -> Delete at the top right
- Click OK

4-3. Deleting Project Infrastructure

Deleting Nodegroup

Refer to the Nodegroup Deletion page.

Use the create-nodegroup.yaml to delete the nodegroup.

eksctl delete nodegroup --config-file=create-nodegroup.yaml --approve

Alternatively, you can delete the nodegroup using the eksctl command.

eksctl delete nodegroup --cluster ${AWS_CLUSTER_NAME} --region ${AWS_DEFAULT_REGION} --name ${PROJECT_NODEGROUP_NAME}

Service Account Deletion Update

Refer to the Amazon Web Services Configuration Guide | 5. Creating Kubernetes Service Accounts.

Remove the S3 Bucket control permissions from the Service Account used by the Kubeflow deploy user.

export POLICY_UPDATE_NAME=policy-s3-kubeflow-${AWS_DEFAULT_REGION_ALIAS}-${INFRA_NAME}-${DEPLOY_ENV}

Steps to update {POLICY_UPDATE_NAME} for deletion:

Save the currently applied policy as a JSON-formatted YAML file.

# Get the latest Policy Version
policy_version_id=$(aws iam get-policy --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/${POLICY_UPDATE_NAME} | jq -r '.Policy' | jq -r '.DefaultVersionId')

# Save Policy JSON to a YAML file
aws iam get-policy-version --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/${POLICY_UPDATE_NAME} --version-id ${policy_version_id} | jq -r '.PolicyVersion' | jq -r '.Document' > ${POLICY_UPDATE_NAME}.yaml

Remove {PROJECT_S3_BUCKET_NAME} from the {POLICY_UPDATE_NAME}.yaml Resource as follows:

yq e -i 'del(.Statement[0].Resource[] | select(. == "arn:aws:s3:::"+env(WORKSPACE_S3_BUCKET)))' ${POLICY_UPDATE_NAME}.yaml
yq e -i 'del(.Statement[0].Resource[] | select(. == "arn:aws:s3:::"+env(WORKSPACE_S3_BUCKET)+"/*"))' ${POLICY_UPDATE_NAME}.yaml

Add the updated policy version for {POLICY_UPDATE_NAME}.

aws iam create-policy-version --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/${POLICY_UPDATE_NAME} --set-as-default --policy-document file://${POLICY_UPDATE_NAME}.yaml

If an error occurs while adding the Policy Version, resolve it as follows:

Details

[Expand Policy Version Error]

The Policy Version supports a maximum of 5 versions, so you need to delete the oldest Policy Version.

# Error Message
An error occurred (LimitExceeded) when calling the CreatePolicyVersion operation: A managed policy can have up to 5 versions. Before you create a new version, you must delete an existing version.

Example command to delete Policy Version #2:

aws iam delete-policy-version --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/${POLICY_UPDATE_NAME} --version-id v2

Remove the S3 Bucket control permissions from the Service Account used in the AI Conductor Namespace.
```
export POLICY_UPDATE_NAME=policy-s3-aia-${AWS_DEFAULT_REGION_ALIAS}-${INFRA_NAME}-${DEPLOY_ENV}
```
Follow the same update steps for {POLICY_UPDATE_NAME} as described above.

Deleting the S3 Bucket
- Refer to the Amazon Web Services Configuration Guide | 3. Creating an S3 Bucket.
- Navigate to the AWS S3 Console.
- Click the radio button for {PROJECT_S3_BUCKET_NAME}.
- Click Delete in the top right corner.
- Enter {PROJECT_S3_BUCKET_NAME}.
- Click Delete bucket.

Deleting Kubeflow User Account

Obtain user information from dex.

kubectl get configmap dex -n auth -o jsonpath='{.data.config\.yaml}' > dex-config.yaml

Delete the user account from dex:

Edit dex-config.yaml to remove the Kubeflow user account information.

yq e -i 'del(.staticPasswords[] | select(.email == env(PROJECT_KUBEFLOW_USER_NAME)+"@aic.com"))' dex-config.yaml

Apply the changes to dex:

kubectl create configmap dex --from-file=config.yaml=dex-config.yaml -n auth --dry-run -oyaml | kubectl apply -f -
kubectl rollout restart deployment dex -n auth

Deleting Namespace and Profile

kubectl delete namespace ${PROJECT_KUBEFLOW_NAMESPACE_NAME}

Detailed Steps​

1. Prerequisite​

2. Add Workspace​

2-1. Install Project Infrastructure​

2-2. Create Grafana Dashboard for Project​

2-3. Creating a Project Workspace​

2-4. Creating a System Account for the Project Workspace​

3. Updating the Workspace (TBD)​

4. Deleting the Workspace​

4-1. Deleting the Project Workspace System Account​

4-2. Deleting the Project Workspace​

4-3. Deleting Project Infrastructure​