Appendix : Log Analysis Guide
Updated 2024.05.05
When running a pipeline in ALO, process.log and pipeline.log files are created in the log folder within the train_artifact or inference_artifact path. process.log contains operational information of the ALO train & inference pipeline, generated by ALO, whereas pipeline.log records logs created using the ALO logger API within each Asset, containing information about the AI Solution.
Topics
process.log
ALO executes the training and inference pipeline in the following order:
- Load ALO code into memory
- Install ALO libraries
- Check ALO version
- Read Solution Metadata for system information and reflect it in ALO operations
- Read experimental_plan.yaml and reflect it in ALO operations
- Execute the training pipeline
- Execute the inference pipeline
The user's response method may vary depending on the type of error that occurs at each stage. (* Error Code response guide, TBD)
## Log Analysis Guide: Load ALO code into memory
[2024-04-02 09:35:57,347|ALO|INFO|utils.py(81)|_log_process()] -------------------- Start setting-up ALO source code into the memory..
[2024-04-02 09:35:57,351|ALO|INFO|utils.py(81)|_log_process()] -------------------- Finish setting-up ALO source code
## Log Analysis Guide: Install ALO libraries
[2024-04-02 09:35:57,356|ALO|INFO|utils.py(81)|_log_process()] -------------------- Start ALO library installation
[2024-04-02 09:35:57,360|ALO|MSG|alo.py(408)|_set_alolib()] alolib already exists in local path.
[2024-04-02 09:35:58,543|ALO|MSG|alo.py(418)|_set_alolib()] Success installing alolib requirements.txt
...
[2024-04-02 09:35:58,553|ALO|INFO|utils.py(81)|_log_process()] -------------------- Finish ALO library installation
## Log Analysis Guide: Check ALO version to ensure it is the latest.
[2024-04-02 09:35:58,557|ALO|INFO|utils.py(81)|_log_process()] -------------------- Check ALO version
[2024-04-02 09:35:58,562|ALO|MSG|alo.py(434)|_get_alo_version()] ALO version = develop
[2024-04-02 09:35:58,566|ALO|INFO|utils.py(81)|_log_process()] -------------------- Finish ALO version check
## Log Analysis Guide: Verify the entered Solution Metadata. Solution Metadata is AI-Advisor system meta information at the time of operation.
[2024-04-02 09:35:58,571|ALO|INFO|utils.py(81)|_log_process()] -------------------- Load solution-metadata
[2024-04-02 09:35:58,576|ALO|MSG|alo.py(522)|load_solution_metadata()] Solution metadata not entered. Skip updating solution metadata into experimental_plan.
[2024-04-02 09:35:58,580|ALO|INFO|utils.py(81)|_log_process()] -------------------- Finish loading solution-metadata
## Log Analysis Guide: Verify the entered experimental_plan.yaml.
[2024-04-02 09:35:58,585|ALO|INFO|utils.py(81)|_log_process()] -------------------- Load experimental_plan.yaml
[2024-04-02 09:35:58,590|ALO|MSG|yaml.py(125)|read_yaml()] Successfully loaded << experimental_plan.yaml >> (file: /home/sehyun.song/Project/alo/dev-240228/solution/experimental_plan.yaml)
[2024-04-02 09:35:58,643|ALO|INFO|utils.py(81)|_log_process()] -------------------- Finish loading experimental_plan.yaml
## Log Analysis Guide: Check environment variables that may affect ALO flow control.
[2024-04-02 09:35:58,647|ALO|INFO|utils.py(81)|_log_process()] -------------------- Setup ALO system environments
[2024-04-02 09:35:58,652|ALO|MSG|alo.py(497)|_set_system_envs()] system_envs['start_time']: 20240402T003558Z
[2024-04-02 09:35:58,656|ALO|MSG|alo.py(498)|_set_system_envs()] system_envs['run_status']: init
[2024-04-02 09:35:58,660|ALO|MSG|alo.py(499)|_set_system_envs()] system_envs['pipeline_list']: ['train_pipeline', 'inference_pipeline']
[2024-04-02 09:35:58,665|ALO|MSG|alo.py(501)|_set_system_envs()] system_envs['pipeline_mode (main config.)']: all
[2024-04-02 09:35:58,669|ALO|MSG|alo.py(502)|_set_system_envs()] system_envs['loop (main config.)']: False
[2024-04-02 09:35:58,674|ALO|MSG|alo.py(503)|_set_system_envs()] system_envs['boot_on (main config.)']: False
[2024-04-02 09:35:58,678|ALO|INFO|utils.py(81)|_log_process()] -------------------- Finish ALO system environments setup
## Log Analysis Guide: Create a list of Python packages recorded in experimental_plan.yaml. This will be used when registering the AI Solution later.
[2024-04-02 09:35:58,683|ALO|MSG|alo.py(152)|main()] Folder '/home/sehyun.song/Project/alo/dev-240228/.package_list/' has been removed & regenerated.
## Log Analysis Guide: Specify the pipeline to proceed.
[2024-04-02 09:35:58,687|ALO|INFO|utils.py(81)|_log_process()] -------------------- Current pipeline: train_pipeline
[2024-04-02 09:35:58,692|ALO|INFO|utils.py(79)|_log_process()]
----------------------------------------------------------------------------------------------------------------------
<< SETUP >> train_pipeline start
----------------------------------------------------------------------------------------------------------------------
## Log Analysis Guide: Reset the artifacts repository, which is the pipeline result.
[2024-04-02 09:35:58,697|ALO|MSG|pipeline.py(595)|_empty_artifacts()] Successfully emptied << /home/sehyun.song/Project/alo/dev-240228/train_artifacts/score >>
[2024-04-02 09:35:58,701|ALO|MSG|pipeline.py(595)|_empty_artifacts()] Successfully emptied << /home/sehyun.song/Project/alo/dev-240228/train_artifacts/models >>
[2024-04-02 09:35:58,706|ALO|MSG|pipeline.py(595)|_empty_artifacts()] Successfully emptied << /home/sehyun.song/Project/alo/dev-240228/train_artifacts/output >>
[2024-04-02 09:35:58,710|ALO|MSG|pipeline.py(595)|_empty_artifacts()] Successfully emptied << /home/sehyun.song/Project/alo/dev-240228/train_artifacts/report >>
## Log Analysis Guide: Git clone the Asset.
[2024-04-02 09:35:58,715|ALO|MSG|pipeline.py(667)|_install_asset()] Start setting-up << input >> asset @ << assets >> directory.
[2024-04-02 09:35:58,720|ALO|MSG|pipeline.py(699)|_install_asset()] << input >> asset had already been created at 2024-03-15 21:02:06.089211
[2024-04-02 09:35:58,725|ALO|MSG|pipeline.py(667)|_install_asset()] Start setting-up << train >> asset @ << assets >> directory.
[2024-04-02 09:35:58,729|ALO|MSG|pipeline.py(699)|_install_asset()] << train >> asset had already been created at 2024-03-15 21:02:06.589226
## Log Analysis Guide: Install the Python packages required by the Asset.
[2024-04-02 09:35:58,734|ALO|MSG|install.py(50)|_install_packages()] ======================================== Start dependency installation : << input >>
[2024-04-02 09:35:58,739|ALO|MSG|install.py(52)|_install_packages()] Start checking existence & installing package - pandas==1.5.3 | Progress: ( 1 / 2 total packages )
[2024-04-02 09:35:58,747|ALO|MSG|install.py(65)|_install_packages()] [OK] << pandas==1.5.3 >> already exists
[2024-04-02 09:35:58
,752|ALO|MSG|install.py(50)|_install_packages()] ======================================== Start dependency installation : << train >>
[2024-04-02 09:35:58,757|ALO|MSG|install.py(52)|_install_packages()] Start checking existence & installing package - scikit-learn | Progress: ( 2 / 2 total packages )
[2024-04-02 09:35:58,761|ALO|MSG|install.py(65)|_install_packages()] [OK] << scikit-learn >> already exists
[2024-04-02 09:35:58,766|ALO|MSG|install.py(50)|_install_packages()] ======================================== Start dependency installation : << force-reinstall >>
[2024-04-02 09:35:58,771|ALO|MSG|install.py(84)|_install_packages()] ======================================== Finish dependency installation
[2024-04-02 09:35:58,775|ALO|INFO|utils.py(79)|_log_process()]
----------------------------------------------------------------------------------------------------------------------
<< SETUP >> train_pipeline finish
----------------------------------------------------------------------------------------------------------------------
[2024-04-02 09:35:58,780|ALO|INFO|utils.py(79)|_log_process()]
----------------------------------------------------------------------------------------------------------------------
<< LOAD >> train_pipeline start
----------------------------------------------------------------------------------------------------------------------
## Log Analysis Guide: If accessing S3 is required to load data, acquire access & secret keys.
[2024-04-02 09:35:58,784|ALO|WARNING|external.py(188)|external_load_data()] Not allowed to access aws infra. You did not write any << aws_key_profile >> in the experimental_plan.yaml file.
[2024-04-02 09:35:58,789|ALO|MSG|external.py(243)|external_load_data()] Successfuly removed << /home/sehyun.song/Project/alo/dev-240228/input/train/ >> before loading external data.
[2024-04-02 09:35:58,793|ALO|MSG|external.py(564)|_get_ext_path_type()] << ./solution/sample_data/train_data/ >> may be relative path. The reference folder of relative path is << /home/sehyun.song/Project/alo/dev-240228/ >>.
If this is not appropriate relative path, Loading external data process would raise error.
## Log Analysis Guide: Copy external data to an area accessible by ALO.
[2024-04-02 09:35:58,799|ALO|MSG|external.py(554)|_load_data()] Successfully done loading external data:
./solution/sample_data/train_data/ --> /home/sehyun.song/Project/alo/dev-240228/input/train/
[2024-04-02 09:35:58,803|ALO|MSG|external.py(250)|external_load_data()] Successfuly finish loading << ./solution/sample_data/train_data/ >> into << /home/sehyun.song/Project/alo/dev-240228/input/ >>
[2024-04-02 09:35:58,808|ALO|INFO|utils.py(79)|_log_process()]
----------------------------------------------------------------------------------------------------------------------
<< LOAD >> train_pipeline finish
----------------------------------------------------------------------------------------------------------------------
[2024-04-02 09:35:58,812|ALO|INFO|utils.py(79)|_log_process()]
----------------------------------------------------------------------------------------------------------------------
<< RUN >> train_pipeline start
----------------------------------------------------------------------------------------------------------------------
## Log Analysis Guide: Check the progress of each Asset(=step) in the pipeline. The progress within the Asset code can be checked in pipeline.log.
[2024-04-02 09:35:58,816|ALO|INFO|utils.py(81)|_log_process()] -------------------- current step: input
[2024-04-02 09:35:59,402|ALO|INFO|utils.py(81)|_log_process()] -------------------- current step: train
[2024-04-02 09:36:00,841|ALO|INFO|utils.py(79)|_log_process()]
----------------------------------------------------------------------------------------------------------------------
<< RUN >> train_pipeline finish
----------------------------------------------------------------------------------------------------------------------
[2024-04-02 09:36:00,848|ALO|INFO|utils.py(79)|_log_process()]
----------------------------------------------------------------------------------------------------------------------
<< SAVE >> train_pipeline start
----------------------------------------------------------------------------------------------------------------------
## Log Analysis Guide: If the result needs to be saved externally, transfer the data externally.
[2024-04-02 09:36:00,854|ALO|MSG|external.py(352)|external_save_artifacts()] None of external path is written in your experimental_plan.yaml. Skip saving artifacts into external path.
[2024-04-02 09:36:00,861|ALO|MSG|pipeline.py(200)|save()] Process finish-time: 2024-04-02 09:36:00
[2024-04-02 09:36:04,146|ALO|MSG|artifacts.py(129)|_delete_old_folders()] [backup_history] Current total size is within the limit. No deletion required.
[2024-04-02 09:36:04,421|ALO|INFO|utils.py(79)|_log_process()]
----------------------------------------------------------------------------------------------------------------------
<< SAVE >> train_pipeline finish
----------------------------------------------------------------------------------------------------------------------
[2024-04-02 09:36:04,427|ALO|INFO|utils.py(81)|_log_process()] -------------------- Current pipeline: inference_pipeline
[2024-04-02 09:36:04,437|ALO|INFO|utils.py(79)|_log_process()]
## The following logs for the inference pipeline are omitted
...
pipeline.log
The content of this log varies depending on the executed AI Contents or AI Solution. The example below is the result of running the sample AI solution titanic.
## Log Analysis Guide: Execution result of the input Asset in the train pipeline. "ASSET START" is printed at the start.
## Log Analysis Guide: Asset (step denotes the role of the Asset's execution) receives config, data, args inputs and outputs config (out) and data (out). Note that args are parameters written in experimental_plan.yaml, and data and config are information that this Asset passes to the next Asset.
[INFO][ASSET][2024-04-02 09:35:59,313][train_pipeline][input]: [36m
=========================================================== ASSET START ===========================================================
- time (UTC) : 2024-04-02 00:35:59
- current step : input
- asset branch. : release_1.0
- alolib ver. : develop
- alo ver. : develop
- load config. keys : dict_keys(['meta'])
- load data keys : dict_keys([])
- load args. : {'input_data_folder': 'train_data', 'x_columns': ['Pclass', 'Sex', 'SibSp', 'Parch'], 'y_column': 'Survived'}
====================================================================================================================================
## Log Analysis Guide: Output result of the message saved using save_info() from the ALO logger API.
[INFO][ASSET][2024-04-02 09:35:59,327][train_pipeline][input]: Loaded dataframe0
[INFO][ASSET][2024-04-02 09:35:59,328][train_pipeline][input]: [36m
=========================================================== ASSET FINISH ===========================================================
- time (UTC) : 2024-04-02 00:35:59
- current step : input
- save config. keys : dict_keys(['meta', 'x_columns', 'y_column'])
- save data keys : dict_keys(['dataframe0'])
====================================================================================================================================
[0m
[INFO][ASSET][2024-04-02 09:36:00,385][train_pipeline][train]: [36m
=========================================================== ASSET START ===========================================================
- time (UTC) : 2024-04-02 00:36:00
- current step : train
- asset branch. : release_1.0
- alolib ver. : develop
- alo ver. : develop
- load config. keys : dict_keys(['meta', 'x_columns', 'y_column'])
- load data keys : dict_keys(['dataframe0'])
- load args. : {'n_estimators': 100}
====================================================================================================================================
[0m
[INFO][ASSET][2024-04-02 09:36:00,697][train_pipeline][train]: Successfully got model path for saving or loading your AI model:
/home/sehyun.song/Project/alo/dev-240228/train_artifacts/models/train/
## Log Analysis Guide: Output result of the message saved using save_warning() from the ALO logger API.
[WARNING][ASSET][2024-04-02 09:36:00,704][train_pipeline][train]: Please enter the << external_path - save_train_artifacts_path >> in the experimental_plan.yaml.
## Log Analysis Guide: Output result of save_summary() from the ALO API. This is a mandatory API for AI Solution creation.
[INFO][ASSET][2024-04-02 09:36:00,709][train_pipeline][train]: Successfully saved inference summary yaml.
>> /home/sehyun.song/Project/alo/dev-240228/train_artifacts/score/train_summary.yaml
[INFO][ASSET][2024-04-02 09:36:00,710][train_pipeline][train]: Save summary :
{'result': 'precision: 0.796594261196031', '
score': 0.27, 'date': '2024-04-02 00:36:00', 'note': 'Test Titanic-demo (date: 2024-04-02 00:36:00)', 'probability': {}, 'file_path': '', 'version': ''}
[INFO][ASSET][2024-04-02 09:36:00,711][train_pipeline][train]:
=========================================================== ASSET FINISH ===========================================================
- time (UTC) : 2024-04-02 00:36:00
- current step : train
- save config. keys : dict_keys(['meta', 'x_columns', 'y_column', 'model_path'])
- save data keys : dict_keys(['dataframe0'])
====================================================================================================================================