카이도스의 Tech Blog

EKS CloudFormation 본문

EKS

EKS CloudFormation

카이도스 2024. 3. 3. 02:38
728x90
반응형

EKS CloudFormation

코드배포에 대해서 공부 및 테스트 후 겨우 완성했다..(EKS cloudformation 배포)
완성후에는 간단해보이지만 그과정에서는 엄청난 삽질과 고통을 겪었다..ㅠㅠ
다른사람들은 삽질을 하지않기를 바란다..

CloudFormation 환경은 아래와 같다.(총3개리전에 배포하여 가용성 확보)

pjh-eks-onclick.yaml 내용

AWSTemplateFormatVersion: '2010-09-09'

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:

      - Label:
          default: "<<<<< EKS Config >>>>>"
        Parameters:
          - KeyName
          - ClusterBaseName
          - KubernetesVersion
          - WorkerNodeInstanceType
          - WorkerNodeCount
          - WorkerNodeVolumesize
          
      - Label:
          default: "<<<<< Region AZ >>>>>"
        Parameters:
          - TargetRegion
          - AvailabilityZone1
          - AvailabilityZone2
          - AvailabilityZone3

      - Label:
          default: "<<<<< VPC Subnet >>>>>"
        Parameters:
          - VpcBlock
          - PublicSubnet1Block
          - PublicSubnet2Block
          - PublicSubnet3Block
          - PrivateSubnet1Block
          - PrivateSubnet2Block
          - PrivateSubnet3Block

Parameters:
  KeyName:
    Description: Name of an existing EC2 KeyPair to enable SSH access to the instances. Linked to AWS Parameter
    Type: AWS::EC2::KeyPair::KeyName
    ConstraintDescription: must be the name of an existing EC2 KeyPair.
  ClusterBaseName:
    Type: String
    Default: eksdev
    AllowedPattern: "[a-zA-Z][-a-zA-Z0-9]*"
    Description: must be a valid Allowed Pattern '[a-zA-Z][-a-zA-Z0-9]*'
    ConstraintDescription: ClusterBaseName - must be a valid Allowed Pattern
  KubernetesVersion:
    Description: Enter Kubernetes Version, 1.23 ~ 1.26
    Type: String
    Default: 1.24
  WorkerNodeInstanceType:
    Description: Instance Type. Default is t3.medium.
    Type: String
    Default: t3.medium
  WorkerNodeCount:
    Description: Number of worker nodes to create
    Type: Number
    Default: 3
  WorkerNodeVolumesize:
    Description: Worker Node Volumes size
    Type: Number
    Default: 30

  TargetRegion:
    Type: String
    Default: ap-northeast-2
  AvailabilityZone1:
    Type: String
    Default: ap-northeast-2a
  AvailabilityZone2:
    Type: String
    Default: ap-northeast-2b
  AvailabilityZone3:
    Type: String
    Default: ap-northeast-2c

  VpcBlock:
    Type: String
    Default: 192.168.0.0/16
  PublicSubnet1Block:
    Type: String
    Default: 192.168.1.0/24
  PublicSubnet2Block:
    Type: String
    Default: 192.168.2.0/24
  PublicSubnet3Block:
    Type: String
    Default: 192.168.3.0/24
  PrivateSubnet1Block:
    Type: String
    Default: 192.168.11.0/24
  PrivateSubnet2Block:
    Type: String
    Default: 192.168.12.0/24
  PrivateSubnet3Block:
    Type: String
    Default: 192.168.13.0/24

Resources:
# VPC
  EksVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VpcBlock
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags:
        - Key: Name
          Value: !Sub ${ClusterBaseName}-VPC

# PublicSubnets
  PublicSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Ref AvailabilityZone1
      CidrBlock: !Ref PublicSubnet1Block
      VpcId: !Ref EksVPC
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: !Sub ${ClusterBaseName}-PublicSubnet1
        - Key: kubernetes.io/role/elb
          Value: 1
  PublicSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Ref AvailabilityZone2
      CidrBlock: !Ref PublicSubnet2Block
      VpcId: !Ref EksVPC
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: !Sub ${ClusterBaseName}-PublicSubnet2
        - Key: kubernetes.io/role/elb
          Value: 1
  PublicSubnet3:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Ref AvailabilityZone3
      CidrBlock: !Ref PublicSubnet3Block
      VpcId: !Ref EksVPC
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: !Sub ${ClusterBaseName}-PublicSubnet3
        - Key: kubernetes.io/role/elb
          Value: 1

  InternetGateway:
    Type: AWS::EC2::InternetGateway
  VPCGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      InternetGatewayId: !Ref InternetGateway
      VpcId: !Ref EksVPC

  PublicSubnetRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref EksVPC
      Tags:
        - Key: Name
          Value: !Sub ${ClusterBaseName}-PublicSubnetRouteTable
  PublicSubnetRoute:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PublicSubnetRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway

  PublicSubnet1RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet1
      RouteTableId: !Ref PublicSubnetRouteTable
  PublicSubnet2RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet2
      RouteTableId: !Ref PublicSubnetRouteTable
  PublicSubnet3RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet3
      RouteTableId: !Ref PublicSubnetRouteTable

# PrivateSubnets
  PrivateSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Ref AvailabilityZone1
      CidrBlock: !Ref PrivateSubnet1Block
      VpcId: !Ref EksVPC
      Tags:
        - Key: Name
          Value: !Sub ${ClusterBaseName}-PrivateSubnet1
        - Key: kubernetes.io/role/internal-elb
          Value: 1
  PrivateSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Ref AvailabilityZone2
      CidrBlock: !Ref PrivateSubnet2Block
      VpcId: !Ref EksVPC
      Tags:
        - Key: Name
          Value: !Sub ${ClusterBaseName}-PrivateSubnet2
        - Key: kubernetes.io/role/internal-elb
          Value: 1
  PrivateSubnet3:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Ref AvailabilityZone3
      CidrBlock: !Ref PrivateSubnet3Block
      VpcId: !Ref EksVPC
      Tags:
        - Key: Name
          Value: !Sub ${ClusterBaseName}-PrivateSubnet3
        - Key: kubernetes.io/role/internal-elb
          Value: 1

  PrivateSubnetRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref EksVPC
      Tags:
        - Key: Name
          Value: !Sub ${ClusterBaseName}-PrivateSubnetRouteTable

  PrivateSubnet1RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PrivateSubnet1
      RouteTableId: !Ref PrivateSubnetRouteTable
  PrivateSubnet2RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PrivateSubnet2
      RouteTableId: !Ref PrivateSubnetRouteTable
  PrivateSubnet3RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PrivateSubnet3
      RouteTableId: !Ref PrivateSubnetRouteTable

# EFS
  EFSSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      VpcId: !Ref EksVPC
      GroupDescription: EFS Security Group
      Tags:
      - Key : Name
        Value : !Sub ${ClusterBaseName}-EFS
      SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: '2049'
        ToPort: '2049'
        CidrIp: !Ref VpcBlock

  ElasticFileSystem:
    Type: AWS::EFS::FileSystem
    Properties:
      FileSystemTags:
        - Key: Name
          Value: !Sub ${ClusterBaseName}-EFS
  ElasticFileSystemMountTarget0:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref ElasticFileSystem
      SecurityGroups:
      - !Ref EFSSG
      SubnetId: !Ref PublicSubnet1
  ElasticFileSystemMountTarget1:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref ElasticFileSystem
      SecurityGroups:
      - !Ref EFSSG
      SubnetId: !Ref PublicSubnet2
  ElasticFileSystemMountTarget2:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref ElasticFileSystem
      SecurityGroups:
      - !Ref EFSSG
      SubnetId: !Ref PublicSubnet3

# NAT Gateway
  EKSVPCEIP:
    Type: AWS::EC2::EIP
    Properties:
      Domain: vpc
  EKSVPCNATGW:
    Type: AWS::EC2::NatGateway
    Properties:
      AllocationId: !GetAtt EKSVPCEIP.AllocationId
      SubnetId: !Ref PublicSubnet1
      Tags:
        - Key: Name
          Value: !Sub ${ClusterBaseName}--NATGW

# Add Route - PrivateSubnet RouteTable
  EKSVPCNATGWRoute:
    Type: AWS::EC2::Route
    DependsOn: EKSVPCNATGW
    Properties:
      RouteTableId: !Ref PrivateSubnetRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref EKSVPCNATGW

# EKS Cluster
  EKSDEV:
    Type: AWS::EKS::Cluster
    Properties:
      Name: !Ref ClusterBaseName
      Version: !Ref KubernetesVersion
      RoleArn: !GetAtt EKSClusterRole.Arn
      ResourcesVpcConfig:
        SubnetIds:
          - !Ref PublicSubnet1
          - !Ref PublicSubnet2
          - !Ref PublicSubnet3
  EKSClusterRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - eks.amazonaws.com
          Action: sts:AssumeRole
      ManagedPolicyArns:
      - arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
      - arn:aws:iam::aws:policy/AmazonEKSServicePolicy  

# EKS NodeGroup
  NodeGroup:
    Type: AWS::EKS::Nodegroup
    Properties:
      ClusterName: !Ref EKSDEV
      NodegroupName: ngdev
      NodeRole: !GetAtt WorkerNodesRole.Arn
      DiskSize: !Ref WorkerNodeVolumesize
      RemoteAccess: 
        Ec2SshKey: !Ref KeyName
      Subnets:
        - !Ref PublicSubnet1
        - !Ref PublicSubnet2
        - !Ref PublicSubnet3
      InstanceTypes:
        - !Ref WorkerNodeInstanceType
      ScalingConfig:
        MinSize:
          Ref: WorkerNodeCount
        DesiredSize:
          Ref: WorkerNodeCount
        MaxSize:
          Ref: WorkerNodeCount
  WorkerNodesRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          Effect: Allow
          Principal:
            Service:
              - ec2.amazonaws.com
          Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

Outputs:
  ClusterName:
    Value: !Ref EKSDEV
    Description: Name of the Amazon EKS cluster.
  NodeGroup:
    Value: !Ref NodeGroup
    Description: Name of the Amazon EKS node group.

파라미터 : 아래 빨간색 부분은 설정해주어야 할 것, 그외 부분은 기본값 사용을 권장

  1. <<<<< EKS Config >>>>>
    1. KeyName : kops-ec2에 SSH 접속을 위한 SSH 키페어 선택 ← 미리 SSH 키 생성 해두자!
    2. ClusterBaseName : EKS 클러스터 이름
    3. KubernetesVersion : EKS 호환, 쿠버네티스 버전 (기본 v1.24) ⇒ 변경 가능
    4. WorkerNodeInstanceType: 워커 노드 EC2 인스턴스의 타입 (기본 t3.medium) ⇒ 변경 가능
    5. WorkerNodeCount : 워커노드의 갯수를 입력 (기본 3대) ⇒ 변경 가능
    6. WorkerNodeVolumesize : 워커노드의 EBS 볼륨 크기 (기본 30GiB) ⇒ 변경 가능
  2. <<<<< Region AZ >>>>> : 리전과 가용영역을 지정, 기본값 그대로 사용
    a. Region : ap-northeast-2(서울)
    b. VPC 대역 : 172.20.0.0/16
    Public: 192.168.1.0, 192.168.2.0, 192.168.3.0 / Private: 192.168.11.0, 192.168.12.0, 192.168.13.0

스택 이름, Clisterbasename 은 eksdev로 설정 후 진행함.

배포 후 확인

# Mac 에서 shell통해 확인. 윈도우는 putty를 사용하거나 bastionhost ec2를 추가 생성하여 관리
# iam 확인
eksctl get iamidentitymapping --cluster eksdev

# iam 업데이트 (아래 2개중 하나 사용)
aws eks update-kubeconfig --name eksdev
aws eks --region ap-northeast-2 update-kubeconfig --name eksdev

# 배포 확인
eksctl get nodegroups --cluster eksdev
CLUSTER NODEGROUP       STATUS  CREATED                 MIN SIZE        MAX SIZE        DESIRED CAPACITY        INSTANCE TYPE   IMAGE ID        ASG NAME                                        TYPE
eksdev  ngdev           ACTIVE  2023-05-22T06:09:57Z    3               3               3                       t3.medium       AL2_x86_64      eks-ngdev-28c4210a-0f44-86fc-5185-a0fa12d979a0  managed

kubectl get nodes
NAME                                               STATUS   ROLES    AGE     VERSION
ip-192-168-1-130.ap-northeast-2.compute.internal   Ready    <none>   7m45s   v1.24.13-eks-0a21954
ip-192-168-2-58.ap-northeast-2.compute.internal    Ready    <none>   7m53s   v1.24.13-eks-0a21954
ip-192-168-3-202.ap-northeast-2.compute.internal   Ready    <none>   7m53s   v1.24.13-eks-0a21954

kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   12m

# 노드 정보 확인
kubectl get node --label-columns=node.kubernetes.io/instance-type,eks.amazonaws.com/capacityType,topology.kubernetes.io/zone
NAME                                               STATUS   ROLES    AGE     VERSION                INSTANCE-TYPE   CAPACITYTYPE   ZONE
ip-192-168-1-130.ap-northeast-2.compute.internal   Ready    <none>   9m50s   v1.24.13-eks-0a21954   t3.medium       ON_DEMAND      ap-northeast-2a
ip-192-168-2-58.ap-northeast-2.compute.internal    Ready    <none>   9m58s   v1.24.13-eks-0a21954   t3.medium       ON_DEMAND      ap-northeast-2b
ip-192-168-3-202.ap-northeast-2.compute.internal   Ready    <none>   9m58s   v1.24.13-eks-0a21954   t3.medium       ON_DEMAND      ap-northeast-2c

eksctl get iamidentitymapping --cluster myeks

# 노드 IP 확인
aws ec2 describe-instances --query "Reservations[*].Instances[*].{PublicIPAdd:PublicIpAddress,PrivateIPAdd:PrivateIpAddress,InstanceName:Tags[?Key=='Name']|[0].Value,Status:State.Name}" --filters Name=instance-state-name,Values=running --output table
---------------------------------------------------------------
|                      DescribeInstances                      |
+--------------+-----------------+-----------------+----------+
| InstanceName |  PrivateIPAdd   |   PublicIPAdd   | Status   |
+--------------+-----------------+-----------------+----------+
|  None        |  192.168.3.202  |  13.209.7.100   |  running |
|  None        |  192.168.2.58   |  43.201.43.190  |  running |
|  None        |  192.168.1.130  |  13.125.18.135  |  running |
+--------------+-----------------+-----------------+----------+
728x90
반응형
저작자표시 비영리 변경금지

'EKS' 카테고리의 다른 글

5주차 - EKS Autoscaling  (0) 2023.05.30
EKS 4주차 - EKS Observability  (0) 2023.05.16
EKS 3주차 - EKS Storage & Node 관리  (0) 2023.05.12
EKS 2주차 - EKS Networking  (0) 2023.05.09
EKS 1주차 - Amzaon EKS 설치 및 기본 사용  (0) 2023.04.26
'EKS' Related Articles more
Comments