Terraform EKS - 01(VPC)

2023.04.26 - [EKS] - EKS 1주차 - Amzaon EKS 설치 및 기본 사용

EKS 1주차 - Amzaon EKS 설치 및 기본 사용

2023.05.23 - [Terraform] - [CHAPTER 1&2] 기초

[CHAPTER 1&2] 기초

2025.02.14 - [EKS] - Terraform EKS - 02(Network, Storage)

Terraform EKS - 02(Network, Storage)

---

기존에 CloudFormation으로만 EKS 배포하는걸 테스트했었지만 진정한 IAC를 위해 네트워크 영역만 CloudFormation로 배포 후 인프라 배포 자동화는 테라폼으로 하려고 한다.

대략적인 구성은 아래와 같다.

---

테스트 VPC 환경

• VPC: service → 172.20.0.0/16
• Subnet
  • eks-public-01 → 172.20.1.0/24
  • eks-public-02 → 172.20.2.0/24
  • eks-public-03 → 172.20.3.0/24
  • eks-private-01 → 172.20.11.0/24
  • eks-private-02 → 172.20.12.0/24
  • eks-private-03 → 172.20.13.0/24
• IGW: service-igw
• NATGW: service-nat → eks-public-01
• Routing Table
  • service-public → 0.0.0.0/0 ⇒ service-igw
  • service-private → 0.0.0.0/0 ⇒ service-nat

---

1. 기본 환경 구성

• 기본 환경 CloudFormation 배포

# 폴더생성
mkdir -p pjh-dev-eks/cf && cd pjh-dev-eks/cf

# CloudFormation 스택 배포
aws cloudformation create-stack \
  --stack-name my-basic-infra \
  --template-body file://basic_infra.yaml
{
    "StackId": "arn:aws:cloudformation:ap-northeast-2:20....:stack/my-basic-infra/d0ee1bf0-e76a-11ef-9f3e-02470a0bdccd"
}

# [모니터링] CloudFormation 스택 상태
while true; do 
  date
  AWS_PAGER="" aws cloudformation list-stacks \
    --stack-status-filter CREATE_IN_PROGRESS CREATE_COMPLETE CREATE_FAILED DELETE_IN_PROGRESS DELETE_FAILED \
    --query "StackSummaries[*].{StackName:StackName, StackStatus:StackStatus}" \
    --output table
  sleep 1
done
2025년 2월 10일 월요일 13시 52분 45초 KST
------------------------------------------
|               ListStacks               |
+-----------------+----------------------+
|    StackName    |     StackStatus      |
+-----------------+----------------------+
|  my-basic-infra |  CREATE_IN_PROGRESS  |
+-----------------+----------------------+

---

Terraform EKS 구성 정보

• Cluster Name
  • pjh-dev-eks
• Kubernetes Version
  • 1.30
• Cluster Type: Fully Private Cluster
  • API Endpoint Public Access: False
  • API Endpoint Private Access: True
• Cluster Add-On
  • coredns
  • kube-proxy
  • vpc-cni
  • eks-pod-identity-agent
  • aws-ebs-csi-driver
• VPC
  • service
• Subnets
  • eks-private-01, eks-private-02, eks-private-03
• Node Group: Managed Node Group
  • Name: pjh-dev-eks-node-group
  • Instance Type: t3.medium
  • Scaling
    • Desired: 3
    • Min: 2
    • Max: 5
• Cluster Access
  • AmazonEKSClusterAdminPolicy
• VPC Endpoints
  • Gateway: s3
  • Interface: ec2, elasticloadbalancing, ecr_api, ecr_dkr, sts
• EKS Bastion EC2
  • kubectl, helm, eksctl, awscli, docker
  • kube-ps1, kubecolor, kubectx, kubens

---

2. Terraform EKS on VPC 배포

Terraform Code

# 폴더 생성 후 아래 파일 생성
cd
mkdir -p pjh-dev-eks/aws && cd pjh-dev-eks/aw

• variable.tf

variable "KeyName" {
  description = "Name of an existing EC2 KeyPair to enable SSH access to the instances."
  type        = string
}

variable "MyIamUserAccessKeyID" {
  description = "IAM User - AWS Access Key ID."
  type        = string
  sensitive   = true
}

variable "MyIamUserSecretAccessKey" {
  description = "IAM User - AWS Secret Access Key."
  type        = string
  sensitive   = true
}

variable "SgIngressSshCidr" {
  description = "The IP address range that can be used to SSH to the EC2 instances."
  type        = string
  validation {
    condition     = can(regex("^(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})$", var.SgIngressSshCidr))
    error_message = "The SgIngressSshCidr value must be a valid IP CIDR range of the form x.x.x.x/x."
  }
}

variable "MyInstanceType" {
  description = "EC2 instance type."
  type        = string
  default     = "t3.medium"
  validation {
    condition     = contains(["t2.micro", "t2.small", "t2.medium", "t3.micro", "t3.small", "t3.medium"], var.MyInstanceType)
    error_message = "Invalid instance type. Valid options are t2.micro, t2.small, t2.medium, t3.micro, t3.small, t3.medium."
  }
}

variable "ClusterBaseName" {
  description = "Base name of the cluster."
  type        = string
  default     = "pjh-dev-eks"
}

variable "KubernetesVersion" {
  description = "Kubernetes version for the EKS cluster."
  type        = string
  default     = "1.30"
}

variable "WorkerNodeInstanceType" {
  description = "EC2 instance type for the worker nodes."
  type        = string
  default     = "t3.medium"
}

variable "WorkerNodeCount" {
  description = "Number of worker nodes."
  type        = number
  default     = 3
}

variable "WorkerNodeVolumesize" {
  description = "Volume size for worker nodes (in GiB)."
  type        = number
  default     = 30
}

variable "TargetRegion" {
  description = "AWS region where the resources will be created."
  type        = string
  default     = "ap-northeast-2"
}

variable "availability_zones" {
  description = "List of availability zones."
  type        = list(string)
  default     = ["ap-northeast-2a", "ap-northeast-2b", "ap-northeast-2c", "ap-northeast-2d"]
}

• ec2.tf

data "aws_ssm_parameter" "ami" {
  name = "/aws/service/canonical/ubuntu/server/22.04/stable/current/amd64/hvm/ebs-gp2/ami-id"
}

resource "aws_security_group" "eks_sec_group" {
  vpc_id = data.aws_vpc.service_vpc.id

  name        = "${var.ClusterBaseName}-bastion-sg"
  description = "Security group for ${var.ClusterBaseName} Host"

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = [var.SgIngressSshCidr]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${var.ClusterBaseName}-HOST-SG"
  }
}

resource "aws_instance" "eks_bastion" {
  ami                         = data.aws_ssm_parameter.ami.value
  instance_type               = var.MyInstanceType
  key_name                    = var.KeyName
  subnet_id                   = data.aws_subnet.eks_public_1.id
  associate_public_ip_address = true
  private_ip                  = "172.20.1.100"
  vpc_security_group_ids      = [aws_security_group.eks_sec_group.id]

  tags = {
    Name = "${var.ClusterBaseName}-bastion-EC2"
  }

  root_block_device {
    volume_type           = "gp3"
    volume_size           = 30
    delete_on_termination = true
  }

  user_data = <<-EOF
    #!/bin/bash
    hostnamectl --static set-hostname "${var.ClusterBaseName}-bastion-EC2"

    # Config convenience
    echo 'alias vi=vim' >> /etc/profile
    echo "sudo su -" >> /home/ubuntu/.bashrc
    timedatectl set-timezone Asia/Seoul

    # Install Packages
    apt update
    apt install -y tree jq git htop unzip

    # Install kubectl & helm
    curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/1.30.0/2024-05-12/bin/linux/amd64/kubectl
    install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
    curl -s https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

    # Install eksctl
    curl -sL "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_Linux_amd64.tar.gz" | tar xz -C /tmp
    mv /tmp/eksctl /usr/local/bin

    # Install aws cli v2
    curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
    unzip awscliv2.zip >/dev/null 2>&1
    ./aws/install
    complete -C '/usr/local/bin/aws_completer' aws
    echo 'export AWS_PAGER=""' >> /etc/profile
    echo "export AWS_DEFAULT_REGION=${var.TargetRegion}" >> /etc/profile

    # Install YAML Highlighter
    wget https://github.com/andreazorzetto/yh/releases/download/v0.4.0/yh-linux-amd64.zip
    unzip yh-linux-amd64.zip
    mv yh /usr/local/bin/

    # Install kube-ps1
    echo 'source <(kubectl completion bash)' >> /root/.bashrc
    echo 'alias k=kubectl' >> /root/.bashrc
    echo 'complete -F __start_kubectl k' >> /root/.bashrc
            
    git clone https://github.com/jonmosco/kube-ps1.git /root/kube-ps1
    cat <<"EOT" >> /root/.bashrc
    source /root/kube-ps1/kube-ps1.sh
    KUBE_PS1_SYMBOL_ENABLE=false
    function get_cluster_short() {
      echo "$1" | grep -o '${var.ClusterBaseName}[^/]*' | cut -c 1-14 
    }
    KUBE_PS1_CLUSTER_FUNCTION=get_cluster_short
    KUBE_PS1_SUFFIX=') '
    PS1='$(kube_ps1)'$PS1
    EOT

    # kubecolor
    apt install kubecolor
    echo 'alias kubectl=kubecolor' >> /root/.bashrc

    # Install kubectx & kubens
    git clone https://github.com/ahmetb/kubectx /opt/kubectx >/dev/null 2>&1
    ln -s /opt/kubectx/kubens /usr/local/bin/kubens
    ln -s /opt/kubectx/kubectx /usr/local/bin/kubectx

    # Install Docker
    curl -fsSL https://get.docker.com -o get-docker.sh
    sh get-docker.sh
    systemctl enable docker

    # Create SSH Keypair
    ssh-keygen -t rsa -N "" -f /root/.ssh/id_rsa

    # IAM User Credentials
    export AWS_ACCESS_KEY_ID="${var.MyIamUserAccessKeyID}"
    export AWS_SECRET_ACCESS_KEY="${var.MyIamUserSecretAccessKey}"
    export ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)
    echo "export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID" >> /etc/profile
    echo "export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY" >> /etc/profile
    echo "export ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)" >> /etc/profile

    # CLUSTER_NAME
    export CLUSTER_NAME="${var.ClusterBaseName}"
    echo "export CLUSTER_NAME=$CLUSTER_NAME" >> /etc/profile

    # VPC & Subnet
    export VPCID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=service" | jq -r .Vpcs[].VpcId)
    echo "export VPCID=$VPCID" >> /etc/profile
    export PublicSubnet1=$(aws ec2 describe-subnets --filters "Name=tag:Name,Values=service-public-01" | jq -r '.Subnets[] | select(.CidrBlock | startswith("172.31")).SubnetId')
    echo "export PublicSubnet1=$PublicSubnet1" >> /etc/profile
    export PrivateSubnet1=$(aws ec2 describe-subnets --filters "Name=tag:Name,Values=eks-private-01" | jq -r '.Subnets[] | select(.CidrBlock | startswith("172.31")).SubnetId')
    export PrivateSubnet2=$(aws ec2 describe-subnets --filters "Name=tag:Name,Values=eks-private-02" | jq -r '.Subnets[] | select(.CidrBlock | startswith("172.31")).SubnetId')
    export PrivateSubnet3=$(aws ec2 describe-subnets --filters "Name=tag:Name,Values=eks-private-03" | jq -r '.Subnets[] | select(.CidrBlock | startswith("172.31")).SubnetId')
    echo "export PrivateSubnet1=$PrivateSubnet1" >> /etc/profile
    echo "export PrivateSubnet2=$PrivateSubnet2" >> /etc/profile
    echo "export PrivateSubnet3=$PrivateSubnet3" >> /etc/profile
    
    # ssh key-pair
    aws ec2 delete-key-pair --key-name kp_node
    aws ec2 create-key-pair --key-name kp_node --query 'KeyMaterial' --output text > ~/.ssh/kp_node.pem
    chmod 400 ~/.ssh/kp_node.pem

  EOF
  
  user_data_replace_on_change = true
  
}

 

• main.tf

provider "aws" {
  region = var.TargetRegion
}

# caller_identity data
data "aws_caller_identity" "current" {}

# vpc data
data "aws_vpc" "service_vpc" {
  tags = {
    Name = "service"
  }
}

# subnet data
data "aws_subnet" "eks_public_1" {
  vpc_id = data.aws_vpc.service_vpc.id
  tags = {
    Name = "eks-public-01"
  }
}
data "aws_subnet" "eks_public_2" {
  vpc_id = data.aws_vpc.service_vpc.id
  tags = {
    Name = "eks-public-02"
  }
}
data "aws_subnet" "eks_public_3" {
  vpc_id = data.aws_vpc.service_vpc.id
  tags = {
    Name = "eks-public-03"
  }
}
data "aws_subnet" "eks_private_1" {
  vpc_id = data.aws_vpc.service_vpc.id
  tags = {
    Name = "eks-private-01"
  }
}
data "aws_subnet" "eks_private_2" {
  vpc_id = data.aws_vpc.service_vpc.id
  tags = {
    Name = "eks-private-02"
  }
}
data "aws_subnet" "eks_private_3" {
  vpc_id = data.aws_vpc.service_vpc.id
  tags = {
    Name = "eks-private-03"
  }
}

# route table data
data "aws_route_table" "service_public" {
  vpc_id = data.aws_vpc.service_vpc.id
  tags = {
    Name = "service-public"
  }
}
data "aws_route_table" "service_private" {
  vpc_id = data.aws_vpc.service_vpc.id
  tags = {
    Name = "service-private"
  }
}

# node_group sg
resource "aws_security_group" "node_group_sg" {
  name        = "${var.ClusterBaseName}-node-group-sg"
  description = "Security group for EKS Node Group"
  vpc_id      = data.aws_vpc.service_vpc.id

  tags = {
    Name = "${var.ClusterBaseName}-node-group-sg"
  }
}
resource "aws_security_group_rule" "allow_ssh" {
  type        = "ingress"
  from_port   = 22
  to_port     = 22
  protocol    = "tcp"
  cidr_blocks = ["172.20.1.100/32"]

  security_group_id = aws_security_group.node_group_sg.id
}

# eks module
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~>20.0"

  cluster_name    = var.ClusterBaseName
  cluster_version = var.KubernetesVersion

  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = false

  cluster_addons = {
    coredns = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
    vpc-cni = {
      most_recent = true
    }
    eks-pod-identity-agent = {
      most_recent = true
    }
    aws-ebs-csi-driver = {
      most_recent = true
      service_account_role_arn = module.irsa-ebs-csi.iam_role_arn
    }
  }

  vpc_id     = data.aws_vpc.service_vpc.id
  subnet_ids = [data.aws_subnet.eks_private_1.id, data.aws_subnet.eks_private_2.id, data.aws_subnet.eks_private_3.id]

  eks_managed_node_group_defaults = {
    ami_type = "AL2_x86_64"

  }

  # eks managed node group
  eks_managed_node_groups = {
    default = {
      name             = "${var.ClusterBaseName}-node-group"
      use_name_prefix  = false
      instance_type    = var.WorkerNodeInstanceType
      desired_size     = var.WorkerNodeCount
      max_size         = var.WorkerNodeCount + 2
      min_size         = var.WorkerNodeCount - 1
      disk_size        = var.WorkerNodeVolumesize
      subnets          = [data.aws_subnet.eks_private_1.id, data.aws_subnet.eks_private_2.id, data.aws_subnet.eks_private_3.id]
      key_name         = "kp_node"
      vpc_security_group_ids = [aws_security_group.node_group_sg.id]
      iam_role_name    = "${var.ClusterBaseName}-node-group-eks-node-group"
      iam_role_use_name_prefix = false
   }
  }

  # Cluster access entry
  enable_cluster_creator_admin_permissions = false
  access_entries = {
    admin = {
      kubernetes_groups = []
      principal_arn     = "${data.aws_caller_identity.current.arn}" 

      policy_associations = {
        myeks = {
          policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
          access_scope = {
            namespaces = []
            type       = "cluster"
          }
        }
      }
    }
  }
}

# cluster sg - ingress rule add
resource "aws_security_group_rule" "cluster_sg_add" {
  type              = "ingress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  cidr_blocks       = ["172.20.1.100/32"]
  security_group_id = module.eks.cluster_security_group_id
  depends_on        = [module.eks]
}

# shared node sg - ingress rule add
resource "aws_security_group_rule" "node_sg_add" {
  type              = "ingress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  cidr_blocks       = ["172.20.1.100/32"]
  security_group_id = module.eks.node_security_group_id
  depends_on        = [module.eks]
}

# vpc endpoint
module "vpc_vpc-endpoints" {
  source  = "terraform-aws-modules/vpc/aws//modules/vpc-endpoints"
  version = "5.8.1"

  vpc_id             = data.aws_vpc.service_vpc.id
  security_group_ids = [module.eks.node_security_group_id]

  endpoints = {
    # gateway endpoints
    s3 = {
      service = "s3"
      route_table_ids = [data.aws_route_table.service_private.id]
      tags = { Name = "s3-vpc-endpoint" }
    }
    
    # interface endpoints
    ec2 = {
      service = "ec2"
      subnet_ids = [data.aws_subnet.eks_private_1.id, data.aws_subnet.eks_private_2.id, data.aws_subnet.eks_private_3.id]
      tags = { Name = "ec2-vpc-endpoint" }
    }
    elasticloadbalancing = {
      service = "elasticloadbalancing"
      subnet_ids = [data.aws_subnet.eks_private_1.id, data.aws_subnet.eks_private_2.id, data.aws_subnet.eks_private_3.id]
      tags = { Name = "elasticloadbalancing-vpc-endpoint" }
    }
    ecr_api = {
      service = "ecr.api"
      subnet_ids = [data.aws_subnet.eks_private_1.id, data.aws_subnet.eks_private_2.id, data.aws_subnet.eks_private_3.id]
      tags = { Name = "ecr-api-vpc-endpoint" }
    }
    ecr_dkr = {
      service = "ecr.dkr"
      subnet_ids = [data.aws_subnet.eks_private_1.id, data.aws_subnet.eks_private_2.id, data.aws_subnet.eks_private_3.id]
      tags = { Name = "ecr-api-vpc-endpoint" }
    }
    sts = {
      service = "sts"
      subnet_ids = [data.aws_subnet.eks_private_1.id, data.aws_subnet.eks_private_2.id, data.aws_subnet.eks_private_3.id]
      tags = { Name = "sts-vpc-endpoint" }
    }
  }
}

# ebs-csi policy & irsa
data "aws_iam_policy" "ebs_csi_policy" {
  arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
}

module "irsa-ebs-csi" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
  version = "5.39.0"

  create_role                   = true
  role_name                     = "AmazonEKSTFEBSCSIRole-${module.eks.cluster_name}"
  provider_url                  = module.eks.oidc_provider
  role_policy_arns              = [data.aws_iam_policy.ebs_csi_policy.arn]
  oidc_fully_qualified_subjects = ["system:serviceaccount:kube-system:ebs-csi-controller-sa"]
}

 

• out.tf

output "public_ip" {
  value       = aws_instance.eks_bastion.public_ip
  description = "The public IP of the myeks-host EC2 instance."
}

• Terraform 변수 선언

# Terraform 환경 변수 저장
export TF_VAR_KeyName=[ssh keypair]
export TF_VAR_MyIamUserAccessKeyID=[iam 사용자의 access key id]
export TF_VAR_MyIamUserSecretAccessKey=[iam 사용자의 secret access key]
export TF_VAR_SgIngressSshCidr=$(curl -s ipinfo.io/ip)/32

• Terraform 배포

# Terraform 배포
terraform init

terraform plan

terraform apply -auto-approve
....
Apply complete! Resources: 54 added, 0 changed, 0 destroyed.

Outputs:

public_ip = "13.125......."

• 디렉터리 구조

tree
.
├── aws
│   ├── ec2.tf
│   ├── main.tf
│   ├── out.tf
│   ├── terraform.tfstate
│   ├── terraform.tfstate.backup
│   └── variable.tf
└── cf
    └── basic_infra.yaml

3 directories, 7 files

---

3. 테스트 구성 확인

• EKS Bastion EC2에 접근해서 작업한다.

# ec2 접속
ssh -i /Users/jeongheepark/.ssh/PJH-aws-test.pem ubuntu@13.125.66.112

# kubeconfig 정보 확인 (클러스터 인증 전이기 때문에 파일이 존재하지 않음)
cat ~/.kube/config | yh

# EKS 클러스터 인증 정보 업데이트
aws eks update-kubeconfig --region $AWS_DEFAULT_REGION --name $CLUSTER_NAME
Added new context arn:aws:eks:ap-northeast-2:20....:cluster/pjh-dev-eks to /root/.kube/config

# kubeconfig 정보 확인 (OK)
cat ~/.kube/config | yh
apiVersion: v1
clusters: 
....
kind: Config
preferences: {}
users: 
- name: arn:aws:eks:ap-northeast-2:20....:cluster/pjh-dev-eks
  user: 
    exec: 
      apiVersion: client.authentication.k8s.io/v1beta1
      args: 
      - --region
      - ap-northeast-2
      - eks
      - get-token
      - --cluster-name
      - pjh-dev-eks
      - --output
      - json
      command: aws

 

• kubens 설정

# kubectl 명령을 수행할 네임스페이스 지정
kubens default
Context "arn:aws:eks:ap-northeast-2:20....:cluster/pjh-dev-eks" modified.
Active namespace is "default".

• 변수 호출 종합

# 배포된 EC2에 선언된 변수 호출
echo $AWS_DEFAULT_REGION
echo $CLUSTER_NAME
echo $VPCID
echo $PublicSubnet1
echo $PrivateSubnet1,$PrivateSubnet2,$PrivateSubnet3
ap-northeast-2
pjh-dev-eks
vpc-0d5e51d3d21c2b265

• EKS Cluster 정보 확인

# kubectl을 통한 eks cluster 정보 확인
kubectl cluster-info
Kubernetes control plane is running at https://6A3BE8222BA90FF1DC69671123031614.yl4.ap-northeast-2.eks.amazonaws.com
CoreDNS is running at https://6A3BE8222BA90FF1DC69671123031614.yl4.ap-northeast-2.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

# eksctl을 통한 eks cluster 정보 확인
eksctl get cluster
--
NAME		REGION		EKSCTL CREATED
pjh-dev-eks	ap-northeast-2	False

# awscli를 통한 eks cluster 정보 확인 (상세)
aws eks describe-cluster --name $CLUSTER_NAME | jq

# awscli를 통한 eks cluster 정보 확인 (endpoint 주소와 endpoint access 확인)
aws eks describe-cluster --name $CLUSTER_NAME | grep endpoint
        "endpoint": "https://6A3BE8222BA90FF1DC69671123031614.yl4.ap-northeast-2.eks.amazonaws.com",
            "endpointPublicAccess": false,
            "endpointPrivateAccess": true,

• API Server Endpoint 주소 확인 및 조회

# API Server Endpoint 도메인 변수 선언
APIDNS=$(aws eks describe-cluster --name ${CLUSTER_NAME} | jq -r .cluster.endpoint | cut -d '/' -f 3)
echo $APIDNS
echo "export APIDNS=$APIDNS" >> /etc/profile
6A3BE8222BA90FF1DC69671123031614.yl4.ap-northeast-2.eks.amazonaws.com

# API Server Endpoint 도메인의 IP 주소 확인
dig +short $APIDNS
--
172.20.11.133
172.20.13.154

API Endpoint Private Access만 가능해서 API Server Endpoint 주소를 dig로 조회해 보면 Private IP 출력

• Cluster Add-On 확인

# 1.30 지원하는 Add-On 확인
eksctl utils describe-addon-versions --kubernetes-version 1.30 | grep AddonName
--
			"AddonName": "netapp_trident-operator",
			"AddonName": "uptycs_uptycs-collector",
			"AddonName": "grafana-labs_kubernetes-monitoring",
			"AddonName": "cloudsoft_amp-add-on",
			"AddonName": "uptycs_uptycs-runtime-sensor",
			"AddonName": "aws-guardduty-agent",
			"AddonName": "appviewx_appviewx-mim-eks",
			"AddonName": "snyk_runtime-sensor",
			"AddonName": "tetrate-io_istio-distro",
			"AddonName": "solo-io_gloo-gateway",
			"AddonName": "aws-ebs-csi-driver",
			"AddonName": "solarwinds_swo-k8s-collector-addon",
			"AddonName": "guance_datakit",
			"AddonName": "kubecost_kubecost",
			"AddonName": "aws-efs-csi-driver",
			"AddonName": "cribl_cribledge",
			"AddonName": "stormforge_optimize-live",
			"AddonName": "solo-io_istio-distro",
			"AddonName": "amazon-cloudwatch-observability",
			"AddonName": "kube-proxy",
			"AddonName": "upwind-security_upwind-operator",
			"AddonName": "aws-network-flow-monitoring-agent",
			"AddonName": "coredns",
			"AddonName": "catalogic-software_cloudcasa",
			"AddonName": "aws-mountpoint-s3-csi-driver",
			"AddonName": "vpc-cni",
			"AddonName": "datadog_operator",
			"AddonName": "akuity_agent",
			"AddonName": "eks-pod-identity-agent",
			"AddonName": "adot",
			"AddonName": "eks-node-monitoring-agent",
			"AddonName": "snapshot-controller",
			"AddonName": "spacelift_workerpool-controller",
			"AddonName": "metrics-server",
			"AddonName": "new-relic_kubernetes-operator",
			"AddonName": "dynatrace_dynatrace-operator",

# Cluster에 설치된 Add-On 확인
eksctl get addon --cluster $CLUSTER_NAME
NAME			VERSION			STATUS	ISSUES	IAMROLE									UPDATE AVAILABLE	CONFIGURATION VALUES	POD IDENTITY ASSOCIATION ROLES
aws-ebs-csi-driver	v1.39.0-eksbuild.1	ACTIVE	0	arn:aws:iam::20....:role/AmazonEKSTFEBSCSIRole-pjh-dev-eks				
coredns			v1.11.4-eksbuild.2	ACTIVE	0													
eks-pod-identity-agent	v1.3.4-eksbuild.1	ACTIVE	0													
kube-proxy		v1.30.7-eksbuild.2	ACTIVE	0													
vpc-cni			v1.19.2-eksbuild.1	ACTIVE	0

• EKS Node Group 확인

# eksctl을 통한 노드 그룹 정보 확인
eksctl get nodegroup --cluster $CLUSTER_NAME --name ${CLUSTER_NAME}-node-group
CLUSTER		NODEGROUP		STATUS	CREATED			MIN SIZE	MAX SIZE	DESIRED CAPACITY	INSTANCE TYPE	IMAGE ID	ASG NAME							TYPE
pjh-dev-eks	pjh-dev-eks-node-group	ACTIVE	2025-02-10T05:57:08Z	2		5		3			t3.medium	AL2_x86_64	eks-pjh-dev-eks-node-group-a0ca7737-640d-1fd6-c1cc-b631dfda5e08	managed

# awscli를 통한 노드 그룹 정보 확인 (상세)
aws eks describe-nodegroup --cluster-name $CLUSTER_NAME --nodegroup-name ${CLUSTER_NAME}-node-group | jq
{
  "nodegroup": {
    "nodegroupName": "pjh-dev-eks-node-group",
....
    "labels": {},
    "resources": {
      "autoScalingGroups": [
        {
          "name": "eks-pjh-dev-eks-node-group-a0ca7737-640d-1fd6-c1cc-b631dfda5e08"
        }
      ]
    },
    "health": {
      "issues": []
    },
    "updateConfig": {
      "maxUnavailablePercentage": 33
    },
    "launchTemplate": {
      "name": "default-20250210055659442700000015",
      "version": "1",
      "id": "lt-00115475768291bbc"
    },
    "tags": {
      "Name": "pjh-dev-eks-node-group"
    }
  }
}

# kubectl을 통한 노드 정보 확인
kubectl get node
NAME                                               STATUS   ROLES    AGE   VERSION
ip-172-20-11-117.ap-northeast-2.compute.internal   Ready    <none>   10m   v1.30.8-eks-aeac579
ip-172-20-12-121.ap-northeast-2.compute.internal   Ready    <none>   10m   v1.30.8-eks-aeac579
ip-172-20-13-116.ap-northeast-2.compute.internal   Ready    <none>   10m   v1.30.8-eks-aeac579

kubectl get node -owide
NAME                                               STATUS   ROLES    AGE   VERSION               INTERNAL-IP     EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION                  CONTAINER-RUNTIME
ip-172-20-11-117.ap-northeast-2.compute.internal   Ready    <none>   10m   v1.30.8-eks-aeac579   172.20.11.117   <none>        Amazon Linux 2   5.10.233-223.887.amzn2.x86_64   containerd://1.7.25
ip-172-20-12-121.ap-northeast-2.compute.internal   Ready    <none>   10m   v1.30.8-eks-aeac579   172.20.12.121   <none>        Amazon Linux 2   5.10.233-223.887.amzn2.x86_64   containerd://1.7.25
ip-172-20-13-116.ap-northeast-2.compute.internal   Ready    <none>   10m   v1.30.8-eks-aeac579   172.20.13.116   <none>        Amazon Linux 2   5.10.233-223.887.amzn2.x86_64   containerd://1.7.25

kubectl get node -v=6
I0210 15:10:19.380654    4625 loader.go:395] Config loaded from file:  /root/.kube/config
I0210 15:10:20.493210    4625 round_trippers.go:553] GET https://6A3BE8222BA90FF1DC69671123031614.yl4.ap-northeast-2.eks.amazonaws.com/api/v1/nodes?limit=500 200 OK in 1097 milliseconds
NAME                                               STATUS   ROLES    AGE   VERSION
ip-172-20-11-117.ap-northeast-2.compute.internal   Ready    <none>   10m   v1.30.8-eks-aeac579
ip-172-20-12-121.ap-northeast-2.compute.internal   Ready    <none>   11m   v1.30.8-eks-aeac579
ip-172-20-13-116.ap-northeast-2.compute.internal   Ready    <none>   11m   v1.30.8-eks-aeac579

• 노드 IP 변수 선언 및 SSH 접근

# 노드 IP 변수 저장
N1=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2a -o jsonpath={.items[0].status.addresses[0].address})
N2=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2b -o jsonpath={.items[0].status.addresses[0].address})
N3=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2c -o jsonpath={.items[0].status.addresses[0].address})
echo "export N1=$N1" >> /etc/profile
echo "export N2=$N2" >> /etc/profile
echo "export N3=$N3" >> /etc/profile
echo $N1, $N2, $N3
172.20.11.117, 172.20.12.121, 172.20.13.116

# 노드에 ssh 접근 확인
for node in $N1 $N2 $N3; do ssh -i ~/.ssh/kp_node.pem -o StrictHostKeyChecking=no ec2-user@$node hostname; done
Warning: Permanently added '172.20.11.117' (ED25519) to the list of known hosts.
ip-172-20-11-117.ap-northeast-2.compute.internal
Warning: Permanently added '172.20.12.121' (ED25519) to the list of known hosts.
ip-172-20-12-121.ap-northeast-2.compute.internal
Warning: Permanently added '172.20.13.116' (ED25519) to the list of known hosts.
ip-172-20-13-116.ap-northeast-2.compute.internal

• VPC Endpoint 확인

# 구성된 VPC Endpoints 확인
aws ec2 describe-vpc-endpoints --query 'VpcEndpoints[*].[VpcEndpointId, ServiceName, State]' --output text
--
vpce-0e698ba7b4a8a123f	com.amazonaws.ap-northeast-2.ecr.dkr	available
vpce-0d4a986e7568128c8	com.amazonaws.ap-northeast-2.ecr.api	available
vpce-05e4f41dd8f681b8e	com.amazonaws.ap-northeast-2.ec2	available
vpce-01dfe460d76ae08d5	com.amazonaws.ap-northeast-2.sts	available
vpce-0b92c0f15ebb1a794	com.amazonaws.ap-northeast-2.elasticloadbalancing	available
vpce-045cfd793bc3f3cef	com.amazonaws.ap-northeast-2.s3	available

---

4. 통신 흐름 확인 (Fully Private Cluster)

• API Server에서 kubelet으로 통신

# [터미널1] 테스트용 nginx 파드 1대 배포
kubectl run nginx --image=nginx --restart=Never
pod/nginx created

# [터미널1] nginx 파드에 bash 실행
kubectl exec pod/nginx -it -- bash

# [터미널2] exec 실행으로 tcp 세션 확인 시 Peer Address 확인
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh -i ~/.ssh/kp_node.pem ec2-user@$i sudo ss -tnp; echo; done
--
>> node 172.20.11.117 <<
State Recv-Q Send-Q Local Address:Port   Peer Address:Port Process                                                 
ESTAB 0      0      172.20.11.117:22     172.20.1.100:57468 users:(("sshd",pid=18833,fd=3),("sshd",pid=18816,fd=3))
ESTAB 0      0      172.20.11.117:35748 172.20.11.133:443   users:(("kube-proxy",pid=4278,fd=7))                   
ESTAB 0      0      172.20.11.117:48866 172.20.11.133:443   users:(("kubelet",pid=2886,fd=25))                     
ESTAB 0      0      172.20.11.117:54244 172.20.13.154:443   users:(("aws-k8s-agent",pid=4426,fd=7))                
ESTAB 0      0      172.20.11.117:39790    10.100.0.1:443   users:(("controller",pid=4540,fd=12))                  

>> node 172.20.12.121 <<
State Recv-Q Send-Q Local Address:Port   Peer Address:Port Process                                               
ESTAB 0      0      172.20.12.121:22     172.20.1.100:46770 users:(("sshd",pid=9688,fd=3),("sshd",pid=9671,fd=3))
ESTAB 0      0      172.20.12.121:57336 172.20.11.133:443   users:(("kube-proxy",pid=4593,fd=7))                 
ESTAB 0      0      172.20.12.121:40972 172.20.13.154:443   users:(("kubelet",pid=2882,fd=27))                   
ESTAB 0      0      172.20.12.121:42000    10.100.0.1:443   users:(("controller",pid=5088,fd=12))                
ESTAB 0      0      172.20.12.121:48098 172.20.13.154:443   users:(("aws-k8s-agent",pid=4976,fd=7))              

>> node 172.20.13.116 <<
State Recv-Q Send-Q          Local Address:Port            Peer Address:Port Process                                                 
ESTAB 0      0               172.20.13.116:22              172.20.1.100:34266 users:(("sshd",pid=10133,fd=3),("sshd",pid=10116,fd=3))
ESTAB 0      0               172.20.13.116:56260             10.100.0.1:443   users:(("controller",pid=5341,fd=12))                  
ESTAB 0      0               172.20.13.116:59718          172.20.11.133:443   users:(("kubelet",pid=2888,fd=27))                     
ESTAB 0      0               172.20.13.116:51564          172.20.11.133:443   users:(("aws-k8s-agent",pid=5228,fd=7))                
ESTAB 0      0                   127.0.0.1:35895              127.0.0.1:36056 users:(("containerd",pid=2759,fd=54))                  
ESTAB 0      0               172.20.13.116:54720          172.20.11.133:443   users:(("kube-proxy",pid=4215,fd=7))                   
ESTAB 0      0                   127.0.0.1:36056              127.0.0.1:35895 users:(("kubelet",pid=2888,fd=12))                     
ESTAB 0      0      [::ffff:172.20.13.116]:10250 [::ffff:172.20.11.133]:37808 users:(("kubelet",pid=2888,fd=9))  

# [터미널1] 테스트용 nginx 파드 1대 삭제
kubectl delete pod nginx

• eks-bastion에서 API Server로 통신

# [터미널1] tcpdump: tcp 443 (3Way Handshake) - 대기 후 터미널2에서 명령 수행 후 결과 확인
tcpdump -n -i ens5 tcp port 443 -c 3
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens5, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:21:40.042765 IP 172.20.1.100.46610 > 172.20.11.133.443: Flags [S], seq 1502385545, win 62727, options [mss 8961,sackOK,TS val 2144353184 ecr 0,nop,wscale 7], length 0
15:21:40.043561 IP 172.20.11.133.443 > 172.20.1.100.46610: Flags [S.], seq 2861023300, ack 1502385546, win 65160, options [mss 1460,sackOK,TS val 3024288488 ecr 2144353184,nop,wscale 7], length 0
15:21:40.044323 IP 172.20.1.100.46610 > 172.20.11.133.443: Flags [.], ack 1, win 491, options [nop,nop,TS val 2144353185 ecr 3024288488], length 0
3 packets captured
30 packets received by filter
0 packets dropped by kernel

# [터미널2] kubectl 명령 수행
kubectl get node

# [터미널2] dig 명령으로 api-server endpoint 주소 확인
dig +short $APIDNS
--
172.20.13.154
172.20.11.133

• kubelet과 kube-proxy에서 API Server로 통신

# 노드마다 tcp 세션 정보 확인
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh -i ~/.ssh/kp_node.pem ec2-user@$i sudo ss -tnp; echo; done
>> node 172.20.11.117 <<
State Recv-Q Send-Q Local Address:Port   Peer Address:Port Process                                                 
ESTAB 0      0      172.20.11.117:35748 172.20.11.133:443   users:(("kube-proxy",pid=4278,fd=7))                   
ESTAB 0      0      172.20.11.117:48866 172.20.11.133:443   users:(("kubelet",pid=2886,fd=25))                     
ESTAB 0      0      172.20.11.117:22     172.20.1.100:55982 users:(("sshd",pid=20542,fd=3),("sshd",pid=20487,fd=3))
ESTAB 0      0      172.20.11.117:54244 172.20.13.154:443   users:(("aws-k8s-agent",pid=4426,fd=7))                
ESTAB 0      0      172.20.11.117:39790    10.100.0.1:443   users:(("controller",pid=4540,fd=12))                  

>> node 172.20.12.121 <<
State Recv-Q Send-Q Local Address:Port   Peer Address:Port Process                                                 
ESTAB 0      0      172.20.12.121:22     172.20.1.100:57260 users:(("sshd",pid=11398,fd=3),("sshd",pid=11381,fd=3))
ESTAB 0      0      172.20.12.121:57336 172.20.11.133:443   users:(("kube-proxy",pid=4593,fd=7))                   
ESTAB 0      0      172.20.12.121:40972 172.20.13.154:443   users:(("kubelet",pid=2882,fd=27))                     
ESTAB 0      0      172.20.12.121:42000    10.100.0.1:443   users:(("controller",pid=5088,fd=12))                  
ESTAB 0      0      172.20.12.121:48098 172.20.13.154:443   users:(("aws-k8s-agent",pid=4976,fd=7))                

>> node 172.20.13.116 <<
State Recv-Q Send-Q Local Address:Port   Peer Address:Port Process                                                 
ESTAB 0      0      172.20.13.116:22     172.20.1.100:47880 users:(("sshd",pid=20923,fd=3),("sshd",pid=20906,fd=3))
ESTAB 0      0      172.20.13.116:56260    10.100.0.1:443   users:(("controller",pid=5341,fd=12))                  
ESTAB 0      0      172.20.13.116:59718 172.20.11.133:443   users:(("kubelet",pid=2888,fd=27))                     
ESTAB 0      0      172.20.13.116:51564 172.20.11.133:443   users:(("aws-k8s-agent",pid=5228,fd=7))                
ESTAB 0      0      172.20.13.116:54720 172.20.11.133:443   users:(("kube-proxy",pid=4215,fd=7))

---

5. 자원 삭제

• 자원 삭제

# terraform 배포 인프라 삭제
terraform destroy -auto-approve

# CloudFormation 스택 삭제
aws cloudformation delete-stack \
  --stack-name my-basic-infra


# [모니터링] CloudFormation 스택 상태
while true; do 
  date
  AWS_PAGER="" aws cloudformation list-stacks \
    --stack-status-filter CREATE_IN_PROGRESS CREATE_COMPLETE CREATE_FAILED DELETE_IN_PROGRESS DELETE_FAILED \
    --query "StackSummaries[*].{StackName:StackName, StackStatus:StackStatus}" \
    --output table
  sleep 1
done