Appendix : Log 분석 가이드
Updated 2024.05.05
ALO 에서 파이프라인을 실행하면 process.log, pipeline.log 파일이 train_artifact 또는 inference_artifact 경로 내의 log 폴더에 생성됩니다. process.log 는 ALOtrain & inference pipeline 의 동작 정보로써, ALO가 생성하는 로그이며, pipeline.log 는 각 Asset에서 ALO logger API를 활용해 생성해낸 로그로써, AI Solution의 정보를 기록한 것 입니다.
Topics
process.log
ALO 는 아래와 같은 순서로 학습 및 추론 파이프라인을 실행 합니다.
- ALO 코드를 메모리에 올리기
- ALO 라이브러리 설치하기
- ALO 버전 체크하기
- Mellerikat 시스템 정보인 Solutoin Metadata 읽고 ALO 동작 제어 반영하기
- experimental_plan.yaml 읽고 ALO 동작 제어에 반영하기
- 학습 파이프라인 실행하기
- 추론 파이프라인 실행하기
각 과정에서 발생하는 에러 타입에 따라 사용자 대응 방법이 달라질 수 있습니다. (* Error Code 별 대응 가이드, TBD)
## 로그 분석 가이드: ALO 코드를 메모리에 올리기
[2024-04-02 09:35:57,347|ALO|INFO|utils.py(81)|_log_process()] -------------------- Start setting-up ALO source code into the memory..
[2024-04-02 09:35:57,351|ALO|INFO|utils.py(81)|_log_process()] -------------------- Finish setting-up ALO source code
## 로그 분석 가이드: ALO 라이브러리 설치하기
[2024-04-02 09:35:57,356|ALO|INFO|utils.py(81)|_log_process()] -------------------- Start ALO library installation
[2024-04-02 09:35:57,360|ALO|MSG|alo.py(408)|_set_alolib()] alolib already exists in local path.
[2024-04-02 09:35:58,543|ALO|MSG|alo.py(418)|_set_alolib()] Success installing alolib requirements.txt
...
[2024-04-02 09:35:58,553|ALO|INFO|utils.py(81)|_log_process()] -------------------- Finish ALO library installation
## 로그 분석 가이드: ALO 버전을 확인 합니다. 최신 버전임을 확인하기 위함 입니다.
[2024-04-02 09:35:58,557|ALO|INFO|utils.py(81)|_log_process()] -------------------- Check ALO version
[2024-04-02 09:35:58,562|ALO|MSG|alo.py(434)|_get_alo_version()] ALO version = develop
[2024-04-02 09:35:58,566|ALO|INFO|utils.py(81)|_log_process()] -------------------- Finish ALO version check
## 로그 분석 가이드: 입력 된 Solution Metadata 를 확인합니다. Solution Metadata 는 운영 시점에 AI-Advisor 시스템 메타 정보 입니다.
[2024-04-02 09:35:58,571|ALO|INFO|utils.py(81)|_log_process()] -------------------- Load solution-metadata
[2024-04-02 09:35:58,576|ALO|MSG|alo.py(522)|load_solution_metadata()] Solution metadata not entered. Skip updating solution metadata into experimental_plan.
[2024-04-02 09:35:58,580|ALO|INFO|utils.py(81)|_log_process()] -------------------- Finish loading solution-metadata
## 로그 분석 가이드: 입력된 experimental_plan.yaml 를 확인 합니다.
[2024-04-02 09:35:58,585|ALO|INFO|utils.py(81)|_log_process()] -------------------- Load experimental_plan.yaml
[2024-04-02 09:35:58,590|ALO|MSG|yaml.py(125)|read_yaml()] Successfully loaded << experimental_plan.yaml >> (file: /home/sehyun.song/Project/alo/dev-240228/solution/experimental_plan.yaml)
[2024-04-02 09:35:58,643|ALO|INFO|utils.py(81)|_log_process()] -------------------- Finish loading experimental_plan.yaml
## 로그 분석 가이드: ALO 흐름 제어에 영향을 줄 수 있는 환경 변수를 확인 합니다.
[2024-04-02 09:35:58,647|ALO|INFO|utils.py(81)|_log_process()] -------------------- Setup ALO system environments
[2024-04-02 09:35:58,652|ALO|MSG|alo.py(497)|_set_system_envs()] system_envs['start_time']: 20240402T003558Z
[2024-04-02 09:35:58,656|ALO|MSG|alo.py(498)|_set_system_envs()] system_envs['run_status']: init
[2024-04-02 09:35:58,660|ALO|MSG|alo.py(499)|_set_system_envs()] system_envs['pipeline_list']: ['train_pipeline', 'inference_pipeline']
[2024-04-02 09:35:58,665|ALO|MSG|alo.py(501)|_set_system_envs()] system_envs['pipeline_mode (main config.)']: all
[2024-04-02 09:35:58,669|ALO|MSG|alo.py(502)|_set_system_envs()] system_envs['loop (main config.)']: False
[2024-04-02 09:35:58,674|ALO|MSG|alo.py(503)|_set_system_envs()] system_envs['boot_on (main config.)']: False
[2024-04-02 09:35:58,678|ALO|INFO|utils.py(81)|_log_process()] -------------------- Finish ALO system environments setup
## 로그 분석 가이드: experimental_plan.yaml 에 기록된 python 패키지 리스트를 생성합니다. 추후 AI Solution 등록 시 사용 됩니다.
[2024-04-02 09:35:58,683|ALO|MSG|alo.py(152)|main()] Folder '/home/sehyun.song/Project/alo/dev-240228/.package_list/' has been removed & regenerated.
## 로그 분석 가이드: 진행 할 파이프라인을 명시 합니다.
[2024-04-02 09:35:58,687|ALO|INFO|utils.py(81)|_log_process()] -------------------- Current pipeline: train_pipeline
[2024-04-02 09:35:58,692|ALO|INFO|utils.py(79)|_log_process()]
----------------------------------------------------------------------------------------------------------------------
<< SETUP >> train_pipeline start
----------------------------------------------------------------------------------------------------------------------
## 로그 분석 가이드: 파이프라인 결과물인 artifacts 저장 소를 reset 합니다.
[2024-04-02 09:35:58,697|ALO|MSG|pipeline.py(595)|_empty_artifacts()] Successfully emptied << /home/sehyun.song/Project/alo/dev-240228/train_artifacts/score >>
[2024-04-02 09:35:58,701|ALO|MSG|pipeline.py(595)|_empty_artifacts()] Successfully emptied << /home/sehyun.song/Project/alo/dev-240228/train_artifacts/models >>
[2024-04-02 09:35:58,706|ALO|MSG|pipeline.py(595)|_empty_artifacts()] Successfully emptied << /home/sehyun.song/Project/alo/dev-240228/train_artifacts/output >>
[2024-04-02 09:35:58,710|ALO|MSG|pipeline.py(595)|_empty_artifacts()] Successfully emptied << /home/sehyun.song/Project/alo/dev-240228/train_artifacts/report >>
## 로그 분석 가이드: Asset 을 git clone 합니다.
[2024-04-02 09:35:58,715|ALO|MSG|pipeline.py(667)|_install_asset()] Start setting-up << input >> asset @ << assets >> directory.
[2024-04-02 09:35:58,720|ALO|MSG|pipeline.py(699)|_install_asset()] << input >> asset had already been created at 2024-03-15 21:02:06.089211
[2024-04-02 09:35:58,725|ALO|MSG|pipeline.py(667)|_install_asset()] Start setting-up << train >> asset @ << assets >> directory.
[2024-04-02 09:35:58,729|ALO|MSG|pipeline.py(699)|_install_asset()] << train >> asset had already been created at 2024-03-15 21:02:06.589226
## 로그 분석 가이드: Asset 에서 필요로 하는 python 패키지를 설치 합니다.
[2024-04-02 09:35:58,734|ALO|MSG|install.py(50)|_install_packages()] ======================================== Start dependency installation : << input >>
[2024-04-02 09:35:58,739|ALO|MSG|install.py(52)|_install_packages()] Start checking existence & installing package - pandas==1.5.3 | Progress: ( 1 / 2 total packages )
[2024-04-02 09:35:58,747|ALO|MSG|install.py(65)|_install_packages()] [OK] << pandas==1.5.3 >> already exists
[2024-04-02 09:35:58,752|ALO|MSG|install.py(50)|_install_packages()] ======================================== Start dependency installation : << train >>
[2024-04-02 09:35:58,757|ALO|MSG|install.py(52)|_install_packages()] Start checking existence & installing package - scikit-learn | Progress: ( 2 / 2 total packages )
[2024-04-02 09:35:58,761|ALO|MSG|install.py(65)|_install_packages()] [OK] << scikit-learn >> already exists
[2024-04-02 09:35:58,766|ALO|MSG|install.py(50)|_install_packages()] ======================================== Start dependency installation : << force-reinstall >>
[2024-04-02 09:35:58,771|ALO|MSG|install.py(84)|_install_packages()] ======================================== Finish dependency installation
[2024-04-02 09:35:58,775|ALO|INFO|utils.py(79)|_log_process()]
----------------------------------------------------------------------------------------------------------------------
<< SETUP >> train_pipeline finish
----------------------------------------------------------------------------------------------------------------------
[2024-04-02 09:35:58,780|ALO|INFO|utils.py(79)|_log_process()]
----------------------------------------------------------------------------------------------------------------------
<< LOAD >> train_pipeline start
----------------------------------------------------------------------------------------------------------------------
## 로그 분석 가이드: 데이터 로드를 위해 s3 접근이 필요한 경우 access & secret key 를 획득 합니다.
[2024-04-02 09:35:58,784|ALO|WARNING|external.py(188)|external_load_data()] Not allowed to access aws infra. You did not write any << aws_key_profile >> in the experimental_plan.yaml file.
[2024-04-02 09:35:58,789|ALO|MSG|external.py(243)|external_load_data()] Successfuly removed << /home/sehyun.song/Project/alo/dev-240228/input/train/ >> before loading external data.
[2024-04-02 09:35:58,793|ALO|MSG|external.py(564)|_get_ext_path_type()] << ./solution/sample_data/train_data/ >> may be relative path. The reference folder of relative path is << /home/sehyun.song/Project/alo/dev-240228/ >>.
If this is not appropriate relative path, Loading external data process would raise error.
## 로그 분석 가이드: 외부 데이터를 ALO 가 접근 가능한 영역으로 copy 합니다.
[2024-04-02 09:35:58,799|ALO|MSG|external.py(554)|_load_data()] Successfully done loading external data:
./solution/sample_data/train_data/ --> /home/sehyun.song/Project/alo/dev-240228/input/train/
[2024-04-02 09:35:58,803|ALO|MSG|external.py(250)|external_load_data()] Successfuly finish loading << ./solution/sample_data/train_data/ >> into << /home/sehyun.song/Project/alo/dev-240228/input/ >>
[2024-04-02 09:35:58,808|ALO|INFO|utils.py(79)|_log_process()]
----------------------------------------------------------------------------------------------------------------------
<< LOAD >> train_pipeline finish
----------------------------------------------------------------------------------------------------------------------
[2024-04-02 09:35:58,812|ALO|INFO|utils.py(79)|_log_process()]
----------------------------------------------------------------------------------------------------------------------
<< RUN >> train_pipeline start
----------------------------------------------------------------------------------------------------------------------
## 로그 분석 가이드: 파이프라인 각 Asset(=step) 진행 상황을 확인 합니다. Asset 코드 내의 진행은 pipeline.log 에서 확인 가능합니다.
[2024-04-02 09:35:58,816|ALO|INFO|utils.py(81)|_log_process()] -------------------- current step: input
[2024-04-02 09:35:59,402|ALO|INFO|utils.py(81)|_log_process()] -------------------- current step: train
[2024-04-02 09:36:00,841|ALO|INFO|utils.py(79)|_log_process()]
----------------------------------------------------------------------------------------------------------------------
<< RUN >> train_pipeline finish
----------------------------------------------------------------------------------------------------------------------
[2024-04-02 09:36:00,848|ALO|INFO|utils.py(79)|_log_process()]
----------------------------------------------------------------------------------------------------------------------
<< SAVE >> train_pipeline start
----------------------------------------------------------------------------------------------------------------------
## 로그 분석 가이드: 결과값을 외부에 저장해야 하는 경우, 데이터를 외부로 전송 합니다.
[2024-04-02 09:36:00,854|ALO|MSG|external.py(352)|external_save_artifacts()] None of external path is written in your experimental_plan.yaml. Skip saving artifacts into external path.
[2024-04-02 09:36:00,861|ALO|MSG|pipeline.py(200)|save()] Process finish-time: 2024-04-02 09:36:00
[2024-04-02 09:36:04,146|ALO|MSG|artifacts.py(129)|_delete_old_folders()] [backup_history] Current total size is within the limit. No deletion required.
[2024-04-02 09:36:04,421|ALO|INFO|utils.py(79)|_log_process()]
----------------------------------------------------------------------------------------------------------------------
<< SAVE >> train_pipeline finish
----------------------------------------------------------------------------------------------------------------------
[2024-04-02 09:36:04,427|ALO|INFO|utils.py(81)|_log_process()] -------------------- Current pipeline: inference_pipeline
[2024-04-02 09:36:04,437|ALO|INFO|utils.py(79)|_log_process()]
## 이하 inference 파이프라인 로그 생략
...
pipeline.log
해당 로그는 실행되는 AI Contents 혹은 AI Solution 에 따라 내용이 달라집니다. 아래 예시는 샘플 AI solution 인 titanic의 실행 결과 입니다.
## 로그 분석 가이드: train 파이프라인의 input Asset의 실행 결과 입니다. 시작 시점에 "ASSET START" 가 출력 됩니다.
## 로그 분석 가이드: Asset (step 은 Asset 의 실행 역할을 의미) 은 config, data, args 입력을 받고, config (out), data (out) 를 출력 합니다. 참고로 args는 experimental_plan.yaml에 작성된 parameter이며, data와 config는 이번 Asset이 다음 Asset으로 넘겨주는 정보입니다.
[INFO][ASSET][2024-04-02 09:35:59,313][train_pipeline][input]: [36m
=========================================================== ASSET START ===========================================================
- time (UTC) : 2024-04-02 00:35:59
- current step : input
- asset branch. : release_1.0
- alolib ver. : develop
- alo ver. : develop
- load config. keys : dict_keys(['meta'])
- load data keys : dict_keys([])
- load args. : {'input_data_folder': 'train_data', 'x_columns': ['Pclass', 'Sex', 'SibSp', 'Parch'], 'y_column': 'Survived'}
====================================================================================================================================
## 로그 분석 가이드: ALO logger API 중 save_info() 를 통해 저장된 message 출력 결과 입니다.
[INFO][ASSET][2024-04-02 09:35:59,327][train_pipeline][input]: Loaded dataframe0
[INFO][ASSET][2024-04-02 09:35:59,328][train_pipeline][input]: [36m
=========================================================== ASSET FINISH ===========================================================
- time (UTC) : 2024-04-02 00:35:59
- current step : input
- save config. keys : dict_keys(['meta', 'x_columns', 'y_column'])
- save data keys : dict_keys(['dataframe0'])
====================================================================================================================================
[0m
[INFO][ASSET][2024-04-02 09:36:00,385][train_pipeline][train]: [36m
=========================================================== ASSET START ===========================================================
- time (UTC) : 2024-04-02 00:36:00
- current step : train
- asset branch. : release_1.0
- alolib ver. : develop
- alo ver. : develop
- load config. keys : dict_keys(['meta', 'x_columns', 'y_column'])
- load data keys : dict_keys(['dataframe0'])
- load args. : {'n_estimators': 100}
====================================================================================================================================
[0m
[INFO][ASSET][2024-04-02 09:36:00,697][train_pipeline][train]: Successfully got model path for saving or loading your AI model:
/home/sehyun.song/Project/alo/dev-240228/train_artifacts/models/train/
## 로그 분석 가이드: ALO logger API 중 save_warning() 를 통해 저장된 message 출력 결과 입니다.
[WARNING][ASSET][2024-04-02 09:36:00,704][train_pipeline][train]: Please enter the << external_path - save_train_artifacts_path >> in the experimental_plan.yaml.
## 로그 분석 가이드: ALO API 중 save_summary() 결과를 출력 합니다. AI Solution 제작 시, 필수로 적용해야 하는 API 입니다.
[INFO][ASSET][2024-04-02 09:36:00,709][train_pipeline][train]: Successfully saved inference summary yaml.
>> /home/sehyun.song/Project/alo/dev-240228/train_artifacts/score/train_summary.yaml
[INFO][ASSET][2024-04-02 09:36:00,710][train_pipeline][train]: Save summary :
{'result': 'precision: 0.796594261196031', 'score': 0.27, 'date': '2024-04-02 00:36:00', 'note': 'Test Titanic-demo (date: 2024-04-02 00:36:00)', 'probability': {}, 'file_path': '', 'version': ''}
[INFO][ASSET][2024-04-02 09:36:00,711][train_pipeline][train]:
=========================================================== ASSET FINISH ===========================================================
- time (UTC) : 2024-04-02 00:36:00
- current step : train
- save config. keys : dict_keys(['meta', 'x_columns', 'y_column', 'model_path'])
- save data keys : dict_keys(['dataframe0'])
====================================================================================================================================