쿠버네티스의 ML Toolkit인 kubeflow로 ML workflow인 모델개발 -> 튜닝 -> 서빙 까지 해보는 것이 이번 핸즈온의 Goal입니다.
local 진행이기 때문에 kubeflow설치를 위해 minikube를 사용합니다.
기본OS는 macOS로 진행합니다. 그 외 OS는 관련 설치 문서 참조 부탁드립니다.
(arrikto miniKF 사용시 kubeflow가 설치되어 있기때문에 별도의 kubeflow 설치가 필요 없습니다.)
- https://minikube.sigs.k8s.io/docs/start/macos/ 참조
$ minikube start --kubernetes-version v1.14.1 --cpus 4 --memory 8096 --disk-size=40g --insecure-registry handson-registry:15000
- Deploying Kubeflow on Existing Clusters
- Kubeflow Deployment with kfctl_k8s_istio
$ wget https://github.com/kubeflow/kubeflow/releases/download/v0.6.2/kfctl_v0.6.2_darwin.tar.gz $ tar -xvf kfctl_v0.6.2_darwin.tar.gz # Add kfctl to PATH, to make the kfctl binary easier to use. # Use only alphanumeric characters or - in the directory name. $ export PATH=$PATH:"<path-to-kfctl>" $ export KFAPP="<your-choice-of-application-directory-name>" # Installs Istio by default. Comment out Istio components in the config file to skip Istio installation. See https://github.com/kubeflow/kubeflow/pull/3663 $ export CONFIG="https://raw.githubusercontent.com/kubeflow/kubeflow/v0.6-branch/bootstrap/config/kfctl_k8s_istio.0.6.2.yaml" $ kubectl create ns kubeflow-anonymous $ kfctl init ${KFAPP} --config=${CONFIG} -V $ cd ${KFAPP} $ kfctl generate all -V $ kfctl apply all -V $ k get po -n kubeflow # https://github.com/istio/istio/issues/10795 port issue
- vagrant 필요
- miniKF from arrikto (https://www.arrikto.com/minikf/)
$ vagrant init arrikto/minikf $ vagrant up
/fairing/mnist.py 참조
3. Fairing - gcloud 설정 (https://cloud.google.com/sdk/docs/quickstart-linux)
private registry가 https를 지원해준다면, 그걸로 진행하지만 그렇지 않다면 gcp를 사용해서 registry 생성
$ curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-245.0.0-linux-x86_64.tar.gz
$ tar zxvf google-cloud-sdk-245.0.0-linux-x86_64.tar.gz google-cloud-sdk
$ ./google-cloud-sdk/install.sh
$ gcloud auth login
$ gcloud config set project handon-kubeflow
# container-registry 관련 auth가 필요합니다.
# https://console.cloud.google.com/apis/api/containerregistry.googleapis.com/landing?project=handon-kubeflow
$ gcloud auth configure-docker
# GOOGLE_APPLICATION_CREDENTIALS service account auth.json 생성 필요
# https://console.cloud.google.com/apis/credentials?project=handon-kubeflow
# gke 생성 - https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-cluster?hl=ko
# 네임스페이스의 imagePullSecrets 설정 필요
$ kubectl create secret docker-registry gcr-json-key \
--docker-server=gcr.io \
--docker-username=_json_key \
--docker-password="$(cat ~/auth.json)" \
--docker-email=any@valid.email
# for private registry
kubectl create secret generic regcred \
--from-file=.dockerconfigjson=/home/vagrant/.docker/config.json \
--type=kubernetes.io/dockerconfigjson -n kubeflow-test
# 서비스 어카운트에 imagePullSecrets 설정
$ kubectl patch serviceaccount default \
-p '{"imagePullSecrets": [{"name": "gcr-json-key"}, {"name": "regcred"}]}'
# 설정 확인
$ kubectl get serviceaccount default -o yaml
- pip install
pip3 install kubeflow-fairing - install from repo.
git clone https://github.com/kubeflow/fairing cd fairing pip3 install -r examples/prediction/requirements.txt pip3 setup.py install
/fairing/fairing_mnist.py 참조
아직 namespace scope을 지원하지 않기 때문에 katib experiment job은 kubeflow namespace로 던져야 metric collector가 생성된다
그래서 ServiceAccount에 kubeflow namespace의 pods, experiment resource를 사용할 수 있게 role binding 시켜줘야함.
/fairing/kubeflow_role_binding.yaml 참조
$ kubectl apply -f mnist_experiment_random.yaml
https://github.com/kubeflow/examples/tree/master/pipelines/mnist-pipelines 기준 변경
# pipeline sdk install
$ !pip install https://storage.googleapis.com/ml-pipeline/release/latest/kfp.tar.gz --upgrade
# /pipeline/mnist_pipeline.py
$ python3 mnist_pipeline.py
# mnist_pipeline.tar.gz 생성 되면 파이프라인에 등록,
# onprem용 pvc 등록
$ kubectl apply -f pvc_for_pipeline.yaml
# model-export-dir을 minio 경로로 변경하기 위해,
# 파이프 라인용 minio ui에 접속하여(NodePort or ingress) minio/ minio123, katib-model bucket 생성
# Add kfctl to PATH, to make the kfctl binary easier to use.
# Use only alphanumeric characters or - in the directory name.
PATH=$PATH:/Users/leemyounghwan/kubeflow_for_kind
KFAPP=/Users/leemyounghwan/kubeflow_for_kind/handson
# Installs Istio by default. Comment out Istio components in the config file to skip Istio installation. See https://github.com/kubeflow/kubeflow/pull/3663
CONFIG="https://raw.githubusercontent.com/kubeflow/kubeflow/v0.6-branch/bootstrap/config/kfctl_k8s_istio.0.6.2.yaml"
kfctl init ${KFAPP} --config=${CONFIG} -V
cd ${KFAPP}
kfctl generate all -V
kfctl apply all -V
cd ${KFAPP}
kfctl delete all -V