Rancher K8s 구성 - 8(Jenkins & ArgoCD CI/CD 구성)

2024.02.15 - [Rancher k8s] - Rancher K8s 구성 - 1(DNS)

Rancher K8s 구성 - 1(DNS)

2024.02.19 - [DB] - Mongodb Shard Cluster 설치 - 2(연동 작업)

Mongodb Shard Cluster 설치 - 2(연동 작업)

2024.02.20 - [Rancher k8s] - Rancher K8s 구성 - 3(Master, Worker)

Rancher K8s 구성 - 3(Master, Worker)

2024.02.21 - [Rancher k8s] - Rancher K8s 구성 - 4(MetalLB & Nginx Ingress 구성)

Rancher K8s 구성 - 4(MetalLB & Nginx Ingress 구성)

2024.02.21 - [Rancher k8s] - Rancher K8s 구성 - 5(Rancher UI 구성)

Rancher K8s 구성 - 5(Rancher UI 구성)

2024.02.21 - [Rancher k8s] - Rancher K8s 구성 - 6(Ceph Storage 구성)

Rancher K8s 구성 - 6(Ceph Storage 구성)

2024.02.21 - [Rancher k8s] - Rancher K8s 구성 - 7(Private Registry 구성 - Harbor)

Rancher K8s 구성 - 7(Private Registry 구성 - Harbor)

---

Helm 활용한 ArgoCD 구성

# argocd 네임스페이스 생성
kubectl create namespace argocd

# helm argocd 레포지토리 추가
helm repo add argo https://argoproj.github.io/argo-helm
helm repo ls
NAME          	URL
jetstack      	https://charts.jetstack.io
rancher-stable	https://releases.rancher.com/server-charts/stable
harbor        	https://helm.goharbor.io
argo          	https://argoproj.github.io/argo-helm

# argocd 디렉토리로 이동하여 helm 파일 다운로드
cd ~/k8s-opensource/argo-cd
helm fetch argo/argo-ci --untar
cd argo-ci

# argocd 설치
helm install argocd argo/argo-cd -f ./values.yaml -n argocd

# pod 확인(3~5분 후 확인)
kubectl get pods -n argocd -o wide
NAME                                                READY   STATUS    RESTARTS      AGE    IP           NODE             NOMINATED NODE   READINESS GATES
argocd-application-controller-0                     1/1     Running   0             2m1s   10.42.5.33   r1-k8s-workre3   <none>           <none>
argocd-applicationset-controller-6c6c8cbd96-zxgl7   1/1     Running   0             2m1s   10.42.5.31   r1-k8s-workre3   <none>           <none>
argocd-dex-server-65f876bd6-ksvgq                   1/1     Running   4 (57s ago)   2m1s   10.42.4.28   r1-k8s-workre2   <none>           <none>
argocd-notifications-controller-6ccff9f5c6-6jzk9    1/1     Running   0             2m1s   10.42.5.32   r1-k8s-workre3   <none>           <none>
argocd-redis-7b45c4599c-jm4g6                       1/1     Running   0             2m1s   10.42.3.30   r1-k8s-workre1   <none>           <none>
argocd-repo-server-64fc45bb8c-rxpxb                 1/1     Running   0             2m1s   10.42.3.32   r1-k8s-workre1   <none>           <none>
argocd-server-74f84fd779-74jdn                      1/1     Running   3 (71s ago)   2m1s   10.42.3.31   r1-k8s-workre1   <none>           <none>

# argocd 패스워드 확인
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
jNWjUP8erbZFtxGw

# argocd ingress에 설정할 tls 생성 (argocd server는 기본적으로 https 통신이 활성화 되어 있음)
cd ..
mkdir tls-secret
cd tls-secret
openssl req -x509 -nodes -days 99999 -newkey rsa:2048 -keyout ingresstls.key -out ingresstls.cert -subj "/CN=argocd.internal.xg.com" -addext "subjectAltName = DNS:argocd.internal.xg.com"

# 생성된 tls 파일 확인
ls -l 
-rw-rw-r-- 1 ubuntu ubuntu 1200 Feb 15 06:34 ingresstls.cert
-rw------- 1 ubuntu ubuntu 1704 Feb 15 06:34 ingresstls.key

# tls 파일을 활용하여 tls secret 생성
kubectl create secret tls ingress-tls --key ingresstls.key --cert ingresstls.cert -n argocd

# argocd ingress 생성
cd ..
cat <<EOT>> argocd-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-argocd
  namespace: argocd
  annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  rules:
  - host: argocd.internal.xg.com
    http:
      paths:
      - path: "/"
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port:
             name: https
  tls:
  - hosts:
    - argocd.internal.xg.com
    secretName: ingress-tls
EOT

# argocd ingress 배포
kubectl apply -f argocd-ingress.yaml

# ingress 확인
kubectl get ingress -n argocd
NAME             CLASS    HOSTS                    ADDRESS                                  PORTS     AGE
ingress-argocd   <none>   argocd.internal.xg.com   10.10.X.204,10.10.X.205,10.10.X.206   80, 443   27s

# Nginx Ingress Controller 로그 확인
kubectl get pods -n kube-system | grep rke2-ingress
helm-install-rke2-ingress-nginx-w5t5s                  0/1     Completed          0               3h33m
rke2-ingress-nginx-controller-28k7z                    1/1     Running            0               3h13m
rke2-ingress-nginx-controller-6fpvl                    1/1     Running            0               3h14m
rke2-ingress-nginx-controller-dk8np                    1/1     Running            0               3h13m

## logs
kubectl logs -n kube-system [nginx ingress pod]
kubectl logs -n kube-system rke2-ingress-nginx-controller-28k7z

# 도메인 설정
    - 도메인 : argocd.internal.xg.com
    - 대상 IP : Nginx Ingress Controller LoadBalancer IP
    - 
# 도메인 접속 확인 - https://argocd.internal.xg.com / (admin / xgk8stest1234!@)
    - ID : admin
    - PW : jNWjUP8erbZFtxGw → (admin / xgk8stest1234!@)
    - ArgoCD 버전 : v2.10.1+a79e0ea

---

Jenkins 파이프라인 구성 (Github, ArgoCD 연동)

# 이미지 빌드를 위해 Jenkins 서버에 도커 설치
# 기존 Docker, Docker-Compose, Containerd 등 설치된 패키지 삭제
for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done

# 도커 설치 및 경로 변경
curl -fsSL https://get.docker.com | sh

# 도커 정보 확인 : client - server, Docker Root Dir, Registry
docker info

# 도커 정보 확인 : Docker Engine - Community
docker version

# 도커 프로세스 내리기
sudo systemctl stop docker.service
sudo systemctl stop docker.socket
sudo systemctl status docker.service
sudo systemctl status docker.socket

# 폴더 복사
cd /var/lib
sudo cp -av docker/ /data/

# 도커 폴더 변경, docker login 경우 기본 https로 요청이 가기 때문에 http로 가도록 허용
sudo su -
cat <<EOT>> /etc/docker/daemon.json
{
        "data-root": "/data/docker"
}
{
  "insecure-registries" : ["harbor-ui.internal.xg.com"]
}
EOT

# 재시작 후 확인
sudo systemctl start docker
sudo systemctl status docker.service
sudo systemctl status docker.socket

# 변경된 디렉터리 확인
sudo docker info | grep "Docker Root Dir"
Docker Root Dir: /data/docker

# jenkins 유저에게 docker 권한 허용
sudo usermod -aG docker jenkins

# jenkins 재시작
sudo systemctl restart jenkins
sudo systemctl status jenkins

• Jenkins Harbor Credential 생성
  • 생성 절차 : Jenkins 관리 → Credentials → System → Add Credentials
  • Kind : Username with password
  • Scope : Global
  • Username : admin
  • Password : xgk8stest1234!@
  • ID : harbor_credential
  • Description: K8s_Rke2_Harbor_Private_Registry

• Jenkins Docker Pipeline 플러그인 설치
  • Jenkins 관리 → Plugins → Available plugins → Docker Pipeline 설치 → 설치 완료 후 재기동