Setting up the Pipeline

Updated 2024.05.05

Once you have a general understanding of the Titanic example, follow the guide below to set up a pipeline for AI Contents similar to the Titanic example using the experimental_plan.yaml.

Topics

• Creating the Pipeline

---

Creating the Pipeline

If you understand the contents of the experimental_plan.yaml below, you can create the pipeline as the process of creating a training pipeline and an inference pipeline is not significantly different.

Create the ./solution folder and experimental_plan.yaml

After installing ALO (Installing ALO), create a solution folder in the terminal and write the experimental_plan.yaml. Start by copying the experimental_plan.yaml template.

mkdir solution
cd solution
vi experimental_plan.yaml  ## You can use other file editors besides vim.
## Copy the experimental_plan_template below and save the yaml file.

Copy and use the experimental_plan.yaml template below. In this example, the load_train(inference)_data_path is set to a path under the solution folder, so place the necessary data for training and inference in that location. If the data is in a different location, change the path accordingly. Both absolute and relative local paths are supported, as well as S3 paths. Refer to the manual (Appendix: What is Experimental Plan) for details and write according to your situation.

Write the name & version of the new AI Contents to be created. This will be displayed in the UI in AI Conductor and used for history management. Note that accessing S3 requires AWS access rights, so write the profile name set with aws configure --profile {PROFILE NAME} in aws_key_profile of experimental_plan.yaml.

In the user_parameters section, write the step names of the train_pipeline and inference_pipeline and the args (arguments) of each step. During asset code development, the self.asset.load_args() API will return the args of each step as a dict. In the asset_source section, since we are developing new AI Contents, write local for the code part. After all development is complete, push each asset to a unique git repository and write the git address and branch name.

ui_args are parameters that users can adjust in the EdgeConductor UI after solution registration. For detailed instructions on writing ui_args, refer to the Write UI Parameter page.

## experimental_plan_template ....
name: My-Solution
version: 1.0.0

external_path:
    - load_train_data_path: ./solution/sample_data/train_data/
    - load_inference_data_path: ./solution/sample_data/inference_data/
    - save_train_artifacts_path:
    - save_inference_artifacts_path:
    - load_model_path:

external_path_permission:
    - aws_key_profile:

user_parameters:
    - train_pipeline:
      - step: input
        args:
          - x_columns:
            y_column:
		    ui_args:
      - step: train
        args:
		    ui_args:
## If you have written content in the ui_args_detail below and want to expose the argument name to allow users to change it in the EdgeConductor UI, write it below.
#			- arg1
#			- arg2
    - inference_pipeline:
      - step: input
        args:
          - x_columns:
            y_column:
	 	    ui_args:
      - step: inference
        args:
		    ui_args:
      - step: output
        args:
		    ui_args:

asset_source:
    - train_pipeline:
      - step: input
        source:
          code: local # or {input Asset git address}
          branch: # {git branch name}
          requirements:
            - pandas==1.5.3

      - step: train
        source:
          code: local
          branch: # {git branch name}
          requirements:
            - scikit-learn
			- requirements.txt

    - inference_pipeline:
      - step: input
        source:
          code: local # or {input Asset git address}          
		  branch: # {git branch name}
          requirements:
            - pandas==1.5.3 --force-reinstall

      - step: inference
        source:
          code: local
          branch: # {git branch name}
          requirements: []

      - step: output
        source:
          code: local
          branch: # {git branch name}
          requirements: []

control:
    - get_asset_source: once
    - backup_artifacts: True
    - backup_log: True
    - backup_size: 1000
    - interface_mode: memory
	- check_resource: False
    - save_inference_format: tar.gz

ui_args_detail:
## If you need users to change parameters in the EdgeConductor UI, write the following content for those parameters.
#    - train_pipeline:
#        - step:
#          args:
#	- inference_pipeline:
#        - step:
#          args: