Skip to main content
Version: Next

Setup Environment and Create AWS Account


Table of Contents​

  1. Setup Deployment Environment
  2. Register AWS Account and Install AWS CLI
  3. Setup AWS Infra
  4. Create AWS Kubeflow User

Detailed Steps​

{For detailed explanations of variables, refer to the Terminology page}


1. Setup Deployment Environment​

 

  • Install Docker Environment
    • It is recommended to use a Docker Container (Ubuntu) to ensure the installation environment is configured independently. 
      export DOCKER_NAME=
      # Download Docker
      docker pull ubuntu:20.04

      # Run Docker
      docker container run -id -w /home/mellerikat --name ${DOCKER_NAME} ubuntu:20.04

      # Execute bash
      docker exec -it ${DOCKER_NAME} /bin/bash

      # Install necessary tools
      apt update
      apt install git curl unzip tar make sudo vim wget mysql-server jq -y

  • Install kubectl
    • kubectl is a command-line tool for controlling Kubernetes clusters.
    • The version should be the same as or up to one minor version earlier or later than the Kubernetes cluster version.
      • (If the cluster version is 1.28, the kubectl version should be 1.27, 1.28, or 1.29)
    • To install or upgrade kubectl, refer to the Install or Update kubectl section.
      • Configure it to communicate with the cluster from your installation environment. 
        export INFRA_NAME=
        export DEPLOY_ENV=
        export AWS_CLUSTER_VERSION=
        export AWS_CLUSTER_VERSION_STR=`echo ${AWS_CLUSTER_VERSION} | tr '.' '-'`
        export AWS_DEFAULT_REGION_ALIAS=
        export AWS_CLUSTER_NAME=eks-${AWS_DEFAULT_REGION_ALIAS}-${INFRA_NAME}-${DEPLOY_ENV}-${AWS_CLUSTER_VERSION_STR}-eks-master

        aws eks update-kubeconfig --region ${AWS_DEFAULT_REGION} --name ${AWS_CLUSTER_NAME} --alias ${INFRA_NAME}

  • Install eksctl
    • eksctl is a command-line tool for controlling EKS clusters.
    • You must have an IAM security principal with permissions to create and describe Amazon EKS clusters.
    • To install or upgrade eksctl, refer to the Install or Update eksctl section.

2. Register AWS Account and Install AWS CLI​

  • If you already have an AWS account with admin privileges, skip this step.
  • If you do not have an account, follow the instructions on the Sign Up for AWS page to create an account.
  • Verify AWS account information
    • Refer to the Find Your AWS Account ID page to find your 12-digit account identifier. 
      export AWS_ACCOUNT_ID=
    • Obtain the ACCESS KEY and SECRET ACCESS KEY for the AWS account. For the process, refer to Managing Access Keys for IAM Users.
    • Register the issued ACCESS KEY and SECRET ACCESS KEY, along with the account's region information, as variables. 
      export AWS_ACCESS_KEY_ID=
      export AWS_SECRET_ACCESS_KEY=
      export AWS_DEFAULT_REGION=
      export AWS_OUTPUT_FORMAT=json
  • Install AWS CLI
    • Configure the AWS CLI in your installation environment. For detailed installation instructions, refer to the Install the AWS CLI section.

3. Setup AWS Infra​

  • The configured name will be used for all resources, including the infrastructure. Multiple projects can be included in one infrastructure. It is recommended to avoid duplicating the infrastructure name ({INFRA_NAME}) and the project name ({PROJECT_NAME}). 
    export INFRA_NAME=
    # Choose between dev or prod for DEPLOY_ENV
    export DEPLOY_ENV=
  • Configure AWS PROFILE with the set INFRA_NAME. 
    {
    echo "${AWS_ACCESS_KEY_ID}"
    echo "${AWS_SECRET_ACCESS_KEY}"
    echo "${AWS_DEFAULT_REGION}"
    echo "${AWS_OUTPUT_FORMAT}"
    } | aws configure --profile=${INFRA_NAME}
    export AWS_PROFILE=${INFRA_NAME}

4. Create AWS Kubeflow User​

  • It is necessary to create a Kubeflow IAM User for the installation and operation of Kubeflow.

  • Create IAM Policy 

    • s3-{AWS_DEFAULT_REGION_ALIAS}-{INFRA_NAME}-{DEPLOY_ENV}-kubeflow: S3 bucket required for Kubeflow operation
    • s3-{AWS_DEFAULT_REGION_ALIAS}-{INFRA_NAME}-{DEPLOY_ENV}-aia: S3 bucket required for mellerikat operation
    [Expand s3-policy.json]
    cat <<EOT > s3-policy.json
    {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "s3:PutObject",
                   "s3:GetObject",
                   "s3:GetObjectAttributes",
                   "s3:GetObjectTagging",
                   "s3:DeleteObject",
                   "s3:ListBucket"
               ],
               "Resource": [
                   "arn:aws:s3:::s3-${AWS_DEFAULT_REGION_ALIAS}-${INFRA_NAME}-${DEPLOY_ENV}-kubeflow",
                   "arn:aws:s3:::s3-${AWS_DEFAULT_REGION_ALIAS}-${INFRA_NAME}-${DEPLOY_ENV}-kubeflow/*",
                   "arn:aws:s3:::s3-${AWS_DEFAULT_REGION_ALIAS}-${INFRA_NAME}-${DEPLOY_ENV}-aia",
                   "arn:aws:s3:::s3-${AWS_DEFAULT_REGION_ALIAS}-${INFRA_NAME}-${DEPLOY_ENV}-aia/*"
               ]
           }
       ]
    }
    EOT
    export KUBEFLOW_USER_POLICY_NAME=policy-${AWS_DEFAULT_REGION_ALIAS}-${INFRA_NAME}-${DEPLOY_ENV}-s3-rwd
    aws iam create-policy \
    --policy-name ${KUBEFLOW_USER_POLICY_NAME} \
    --policy-document file://s3-policy.json

  • Create IAM USER

    export KUBEFLOW_USER_NAME=user-kubeflow-${AWS_DEFAULT_REGION_ALIAS}-${INFRA_NAME}-${DEPLOY_ENV}
    aws iam create-user --user-name ${KUBEFLOW_USER_NAME}

  • Attach policy to USER

    aws iam attach-user-policy \
    --user-name user-kubeflow-${AWS_DEFAULT_REGION_ALIAS}-${INFRA_NAME}-${DEPLOY_ENV} \
    --policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/${KUBEFLOW_USER_POLICY_NAME}"

  • Create and save ACCESS_KEY and SECRET_ACCESS_KEY for USER

    access_key=$(aws iam create-access-key --user-name ${KUBEFLOW_USER_NAME} | jq -r '.AccessKey')
    export KUBEFLOW_USER_AWS_ACCESS_KEY_ID=`echo ${access_key} | jq -r '.AccessKeyId'`
    export KUBEFLOW_USER_AWS_SECRET_ACCESS_KEY=`echo ${access_key} | jq -r '.SecretAccessKey'`