안녕하세요. CloudNet@ K8S Study를 진행하며 해당 내용을 이해하고 공유하기 위해 작성한 글입니다. 해당 내용은 EKS docs와 workshop을 기본으로 정리하였습니다.
실습 환경
# YAML 파일 다운로드
$ curl -O https://s3.ap-northeast-2.amazonaws.com/cloudformation.cloudneta.net/K8S/eks-oneclick5.yaml
# CloudFormation 스택 배포
예시) aws cloudformation deploy --template-file eks-oneclick5.yaml --stack-name myeks --parameter-overrides KeyName=kp-hayley3 SgIngressSshCidr=$(curl -s ipinfo.io/ip)/32 MyIamUserAccessKeyID=AKIA5... MyIamUserSecretAccessKey='CVNa2...' ClusterBaseName=myeks --region ap-northeast-2
# CloudFormation 스택 배포 완료 후 작업용 EC2 IP 출력
$vaws cloudformation describe-stacks --stack-name myeks --query 'Stacks[*].Outputs[0].OutputValue' --output text
# 작업용 EC2 SSH 접속
$vssh -i ~/.ssh/kp-hayley3.pem ec2-user@$(aws cloudformation describe-stacks --stack-name myeks --query 'Stacks[*].Outputs[0].OutputValue' --output text)기본 설정
# default 네임스페이스 적용
$ kubectl ns default
# (옵션) context 이름 변경
# NICK=<각자 자신의 닉네임>
$ NICK=hayley
$ kubectl ctx
iam-root-account@eks-hayley.ap-northeast-2.eksctl.io
$ kubectl config rename-context admin@myeks.ap-northeast-2.eksctl.io $NICK
# ExternalDNS
# MyDomain=<자신의 도메인>
# echo "export MyDomain=<자신의 도메인>" >> /etc/profile
$ MyDomain=wellbeconnected.com
$ echo "export MyDomain=wellbeconnected.com" >> /etc/profile
$ MyDnzHostedZoneId=$(aws route53 list-hosted-zones-by-name --dns-name "${MyDomain}." --query "HostedZones[0].Id" --output text)
$ echo $MyDomain, $MyDnzHostedZoneId
wellbeconnected.com, /hostedzone/Z06204681HPKV2R55I4OR
$ curl -s -O https://raw.githubusercontent.com/gasida/PKOS/main/aews/externaldns.yaml
$ MyDomain=$MyDomain MyDnzHostedZoneId=$MyDnzHostedZoneId envsubst < externaldns.yaml | kubectl apply -f -
serviceaccount/external-dns created
clusterrole.rbac.authorization.k8s.io/external-dns created
clusterrolebinding.rbac.authorization.k8s.io/external-dns-viewer created
deployment.apps/external-dns created
# AWS LB Controller
$ helm repo add eks https://aws.github.io/eks-charts
$ helm repo update
$ helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=$CLUSTER_NAME \
--set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller
# 노드 IP 확인 및 PrivateIP 변수 지정
$ N1=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2a -o jsonpath={.items[0].status.addresses[0].address})
$ N2=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2b -o jsonpath={.items[0].status.addresses[0].address})
$ N3=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2c -o jsonpath={.items[0].status.addresses[0].address})
$ echo "export N1=$N1" >> /etc/profile
$ echo "export N2=$N2" >> /etc/profile
$ echo "export N3=$N3" >> /etc/profile
$ echo $N1, $N2, $N3
192.168.1.83, 192.168.2.145, 192.168.3.57
# 노드 보안그룹 ID 확인
$ NGSGID=$(aws ec2 describe-security-groups --filters Name=group-name,Values='*ng1*' --query "SecurityGroups[*].[GroupId]" --output text)
$ aws ec2 authorize-security-group-ingress --group-id $NGSGID --protocol '-1' --cidr 192.168.1.0/24
{
"Return": true,
"SecurityGroupRules": [
{
"SecurityGroupRuleId": "sgr-079740a4d01fb8d55",
"GroupId": "sg-05b26e567964086be",
"GroupOwnerId": "90XXXXXXXX",
"IsEgress": false,
"IpProtocol": "-1",
"FromPort": -1,
"ToPort": -1,
"CidrIpv4": "192.168.1.0/24"
}
]
}
# 워커 노드 SSH 접속
$ for node in $N1 $N2 $N3; do ssh ec2-user@$node hostname; done프로메테우스 & 그라파나(admin / prom-operator) 설치
# 사용 리전의 인증서 ARN 확인
$ CERT_ARN=`aws acm list-certificates --query 'CertificateSummaryList[].CertificateArn[]' --output text`
$ echo $CERT_ARN
arn:aws:acm:ap-northeast-2:9XXXXXXXX:certificate/3f025fab-XXXXXX
# repo 추가
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
# 파라미터 파일 생성
$ cat <<EOT > monitor-values.yaml
prometheus:
prometheusSpec:
podMonitorSelectorNilUsesHelmValues: false
serviceMonitorSelectorNilUsesHelmValues: false
retention: 5d
retentionSize: "10GiB"
ingress:
enabled: true
ingressClassName: alb
hosts:
- prometheus.$MyDomain
paths:
- /*
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN
alb.ingress.kubernetes.io/success-codes: 200-399
alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb
alb.ingress.kubernetes.io/group.name: study
alb.ingress.kubernetes.io/ssl-redirect: '443'
grafana:
defaultDashboardsTimezone: Asia/Seoul
adminPassword: prom-operator
ingress:
enabled: true
ingressClassName: alb
hosts:
- grafana.$MyDomain
paths:
- /*
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN
alb.ingress.kubernetes.io/success-codes: 200-399
alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb
alb.ingress.kubernetes.io/group.name: study
alb.ingress.kubernetes.io/ssl-redirect: '443'
defaultRules:
create: false
kubeControllerManager:
enabled: false
kubeEtcd:
enabled: false
kubeScheduler:
enabled: false
alertmanager:
enabled: false
EOT
# 배포
$ kubectl create ns monitoring
$ helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 45.27.2 \
--set prometheus.prometheusSpec.scrapeInterval='15s' --set prometheus.prometheusSpec.evaluationInterval='15s' \
-f monitor-values.yaml --namespace monitoring
# Metrics-server 배포
$ kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
$ k get po -n monitoring
NAME READY STATUS RESTARTS AGE
kube-prometheus-stack-grafana-778f55f4cd-4hfq9 3/3 Running 0 40s
kube-prometheus-stack-kube-state-metrics-5d6578867c-j9729 1/1 Running 0 40s
kube-prometheus-stack-operator-74d474b47b-w6q95 1/1 Running 0 40s
kube-prometheus-stack-prometheus-node-exporter-2gw2g 1/1 Running 0 40s
kube-prometheus-stack-prometheus-node-exporter-ckwb7 1/1 Running 0 40s
kube-prometheus-stack-prometheus-node-exporter-pwxhz 1/1 Running 0 40s
prometheus-kube-prometheus-stack-prometheus-0 2/2 Running 0 35s사전 지식
1. AWS IAM
- AWS 서비스와 리소스에 대한 액세스를 안전하게 관리하는 기능
- AWS 사용자 및 그룹을 만들고 관리하거나 권한을 통해 AWS 리소스에 대한 액세스를 허용하거나 거부
- 이전 블로그 참고 : IAM&Organization , EKS IAM
2. K8S 인증/인가
- K8S의 API 서버를 사용해서 인증/인가 절차가 이뤄집니다.
- 인증 : X.509 Client Certs, kubectl, Service Account 방식을 통해 인증
- 인가 : RBAC(Role, RoleBinding) 등의 방식을 통해 인가
.kube/config 파일 내용
$ cat .kube/config
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1EVXlPVEExTWpFME5Wb1hEVE16TURVeU5qQTFNakUwTlZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTDBYCjIyazFxNCthSTFodkpWYWRjMlhpK0pudHVZSlZ5b1E4TlZuelZWbWd0MVNISXAyaUw4WVFVRVdEa3FCT09yaXkKL01UVmhTbG5Zd1JBbEpNU3BkMkNOT0ZBUWpLQ3RhWnNWeTF3YldSK3NyclJDNCs0Yjg0a3hrTzRXY29xZkNHZgphUlEwcjhhT0g4MHdJSHFXSC9Qd0ozb096U0prcXU2Qk5lUlpPMHNuQkxPWVpGc05xamhGNEJLTVpyZC9ZdjBxCjlYSXc1ZjZSSkxxWnR1YzlaNVUwYlkxOTNYWUcrQWEyNWFDSTRtWEtuREtyU01kbzZVYkh2ZHlQNXB5RVhxNm8KNkxqcVNLbEJoSW9uZm8xTEZkbng4SFR3QlVBMk52ckQ5aGZROENSYjVWNWxFdFlxVVE5Z3F1cmNUcHZSdmpPUwpCeVI2NGgvOFRla3ZNYythZk1VQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZKaUxVOVZibFRGZG1TWFpvbDQ0WWVUeG1UWmFNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBRVp1RmN4N2xuR0dyMEZQMUpTYQpYSlJUMnBiMTN0Ty9zMGRaSHdIL3VodTB1dUZCZUVuZ05zeDBlaU9jQy8vNHRLUCtnTFhEYm50NkZxK3kvM0tOCmtEcGplZ3UvcGdsaVFDUWtCVDlUNGZ3dFpGMGNOeERJeUZZcnNvaVBWODBwc2xuWG5UcmZubTJoc2hOZnNwNTUKTSs1LzJ2M3BtYlN5K3RXenJWN1RYb0xMS0NJVll3R3R3c0hwV0ZiQ0NTQ0VQOTZ6cStyOU5VM25FdVpmNDdOUwozRkVmNWE3eTdQa1kwdTBIck1GL3BXWFdwektwL1F4UDE1MU9zUE11aDV1ZjRzbzkyekJyVFN5b2xiZnFvWEZpCjdQZXg3N1pQNEVVOTE5MEVsa3NKalcrWG95Vk56a3MwRHU1TXRBRmM4TXdPM1ZTY0t3MkdVeHUvZ2UrdEpVTnUKMDYwPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
server: https://442308BCEF7A13DE72CFA354A0708CB7.yl4.ap-northeast-2.eks.amazonaws.com
name: eks-hayley.ap-northeast-2.eksctl.io
contexts:
- context:
cluster: eks-hayley.ap-northeast-2.eksctl.io
namespace: default
user: iam-root-account@eks-hayley.ap-northeast-2.eksctl.io
name: iam-root-account@eks-hayley.ap-northeast-2.eksctl.io
current-context: iam-root-account@eks-hayley.ap-northeast-2.eksctl.io
kind: Config
preferences: {}
users:
- name: iam-root-account@eks-hayley.ap-northeast-2.eksctl.io
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- eks
- get-token
- --output
- json
- --cluster-name
- eks-hayley
- --region
- ap-northeast-2
command: aws
env:
- name: AWS_STS_REGIONAL_ENDPOINTS
value: regional
interactiveMode: IfAvailable
provideClusterInfo: false- clusters : kubectl 이 사용할 쿠버네티스 API 서버의 접속 정보 목록
- users : 쿠버네티스의 API 서버에 접속하기 위한 사용자 인증 정보 목록
- contexts : cluster 항목과 users 항목에 정의된 값을 조합해 최종적으로 사용할 쿠버네티스 클러스터의 정보(컨텍스트)를 설정
실습 — K8S 인증/인가
- 각각의 서비스 어카운트(dev-k8s, infra-k8s)는 각기 다른 권한(Role)을 가짐
- dev-k8s는 dev-team 네임스페이스 내 동작, infra-k8s는 dev-team 네임스페이스 내 동작
- 각각 별도 kubectl 파드를 생성하고 해당 파드에 SA를 지정하여 권한에 대한 테스트
네임스페이스와 서비스 어카운트 생성 후 확인
- 서비스 어카운트에 자동 생성된 시크릿에 저장된 토큰으로 쿠버네티스 API에 대한 인증 정보로 사용 할 수 있습니다. (1.23 이전 버전의 경우에만 해당)
# 네임스페이스(Namespace, NS) 생성 및 확인
$ kubectl create namespace dev-team
$ kubectl create ns infra-team
# 네임스페이스 확인
$ kubectl get ns
NAME STATUS AGE
default Active 104m
dev-team Active 15s
infra-team Active 8s
kube-node-lease Active 104m
kube-public Active 104m
kube-system Active 104m
monitoring Active 35m
# 네임스페이스에 각각 서비스 어카운트 생성 : serviceaccounts 약자(=sa)
$ kubectl create sa dev-k8s -n dev-team
$ kubectl create sa infra-k8s -n infra-team
# 서비스 어카운트 정보 확인
$ kubectl get sa -n dev-team
NAME SECRETS AGE
default 0 53s
dev-k8s 0 20s
$ kubectl get sa dev-k8s -n dev-team -o yaml | yh
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: "2023-05-29T07:07:27Z"
name: dev-k8s
namespace: dev-team
resourceVersion: "23567"
uid: c8f62a5c-e679-4e03-9192-10f4b9ce17b2
$ kubectl get sa -n infra-team
NAME SECRETS AGE
default 0 86s
infra-k8s 0 55s
$ kubectl get sa infra-k8s -n infra-team -o yaml | yh
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: "2023-05-29T07:07:32Z"
name: infra-k8s
namespace: infra-team
resourceVersion: "23589"
uid: c7570e5e-3647-4240-89c2-84629c5a1181dev-k8s 서비스어카운트의 토큰 정보 확인
# dev-k8s 서비스어카운트의 토큰 정보 확인
$ DevTokenName=$(kubectl get sa dev-k8s -n dev-team -o jsonpath="{.secrets[0].name}")
$ DevToken=$(kubectl get secret -n dev-team $DevTokenName -o jsonpath="{.data.token}" | base64 -d)
$ echo $DevToken- 쿠버네티스에서 Bearer 토큰을 전송할 때 주로 JWT (JSON Web Token) 토큰을 사용
- jwt(JSON Web Token)는 쿠버네티스에서 뿐만 아니라 다양한 웹 사이트에서 인증, 권한 허가, 세션관리 등의 목적으로 사용
서비스 어카운트를 지정하여 파드 생성 후 권한 테스트
# 각각 네임스피이스에 kubectl 파드 생성 - 컨테이너이미지
# docker run --rm --name kubectl -v /path/to/your/kube/config:/.kube/config bitnami/kubectl:latest
$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: dev-kubectl
namespace: dev-team
spec:
serviceAccountName: dev-k8s
containers:
- name: kubectl-pod
image: bitnami/kubectl:1.24.10
command: ["tail"]
args: ["-f", "/dev/null"]
terminationGracePeriodSeconds: 0
EOF
$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: infra-kubectl
namespace: infra-team
spec:
serviceAccountName: infra-k8s
containers:
- name: kubectl-pod
image: bitnami/kubectl:1.24.10
command: ["tail"]
args: ["-f", "/dev/null"]
terminationGracePeriodSeconds: 0
EOF
# 확인
$ kubectl get pod -A
$ kubectl get pod -o dev-kubectl -n dev-team -o yaml
serviceAccount: dev-k8s
...
$ kubectl get pod -o infra-kubectl -n infra-team -o yaml
serviceAccount: infra-k8s
...
# 파드에 기본 적용되는 서비스 어카운트(토큰) 정보 확인
$ kubectl exec -it dev-kubectl -n dev-team -- ls /run/secrets/kubernetes.io/serviceaccount
ca.crt namespace token
$ kubectl exec -it dev-kubectl -n dev-team -- cat /run/secrets/kubernetes.io/serviceaccount/token
eyJhbGciOiJSUzI1NiIsImtpZCI6IjQ0YTg0NjczZjQwYWNmYmRjNzE2MzZjMzhkYjU2MTJhMjlmOTQ3MDIifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTcxNjg4MDQ2MSwiaWF0IjoxNjg1MzQ0NDYxLCJpc3MiOiJodHRwczovL29pZGMuZWtzLmFwLW5vcnRoZWFzdC0yLmFtYXpvbmF3cy5jb20vaWQvNDQyMzA4QkNFRjdBMTNERTcyQ0ZBMzU0QTA3MDhDQjciLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6ImRldi10ZWFtIiwicG9kIjp7Im5hbWUiOiJkZXYta3ViZWN0bCIsInVpZCI6IjdkMDI0MmRiLTcwZTUtNDEwNC1hOGU5LWRmMTNmZDUzMzhkMiJ9LCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoiZGV2LWs4cyIsInVpZCI6ImM4ZjYyYTVjLWU2NzktNGUwMy05MTkyLTEwZjRiOWNlMTdiMiJ9LCJ3YXJuYWZ0ZXIiOjE2ODUzNDgwNjh9LCJuYmYiOjE2ODUzNDQ0NjEsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZXYtdGVhbTpkZXYtazhzIn0.NxEEzUbkwddUT78bP1xrkDEJNpzmpnCQk4ejW3KoQtrtToM7TO1R18APnSV1VZOShHSTFjJwZNUy-UNkCipv2TIFAweax2UjWXsELLhwaX7K-eZL3ycp3gZB13613ky46z9cTAM3GXZJ1lgdDCngUPVm_Pyk061NT6bMFvdYIY7MotJAK3Tj2uSfas0CB2QokC6LZI_Mu1DPT2VMNfYcG1A8cLXwkB98CF_5NyDN8LYJJc_UvRcLfATmewTaIMYNuRyT0Bm-1y07_gDi1DKe-H1hyCaiIHG15mnssC4BVP539CzIRYUm4di_TG_FYUuuGaH4CPsRIqq4H9iLu3IbjA
$ kubectl exec -it dev-kubectl -n dev-team -- cat /run/secrets/kubernetes.io/serviceaccount/namespace
dev-team
$ kubectl exec -it dev-kubectl -n dev-team -- cat /run/secrets/kubernetes.io/serviceaccount/ca.crt
-----BEGIN CERTIFICATE-----
MIIC/jCCAeagAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTIzMDUyOTA1MjE0NVoXDTMzMDUyNjA1MjE0NVowFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL0X
...
-----END CERTIFICATE-----
# 각각 파드로 Shell 접속하여 정보 확인 : 단축 명령어(alias) 사용
$ alias k1='kubectl exec -it dev-kubectl -n dev-team -- kubectl'
$ alias k2='kubectl exec -it infra-kubectl -n infra-team -- kubectl'
# 권한 테스트
$ k1 get pods # kubectl exec -it dev-kubectl -n dev-team -- kubectl get pods 와 동일한 실행 명령이다!
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:dev-team:dev-k8s" cannot list resource "pods" in API group "" in the namespace "dev-team"
command terminated with exit code 1
$ k1 run nginx --image nginx:1.20-alpine
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:dev-team:dev-k8s" cannot create resource "pods" in API group "" in the namespace "dev-team"
command terminated with exit code 1
$ k1 get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:dev-team:dev-k8s" cannot list resource "pods" in API group "" in the namespace "kube-system"
command terminated with exit code 1
$ k2 get pods # kubectl exec -it infra-kubectl -n infra-team -- kubectl # get pods 와 동일한 실행 명령이다!
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:infra-team:infra-k8s" cannot create resource "pods" in API group "" in the namespace "infra-team"
command terminated with exit code 1
$ k2 run nginx --image nginx:1.20-alpine
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:infra-team:infra-k8s" cannot create resource "pods" in API group "" in the namespace "infra-team"
command terminated with exit code 1
$ k2 get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:infra-team:infra-k8s" cannot create resource "pods" in API group "" in the namespace "infra-team"
command terminated with exit code 1
# (옵션) kubectl auth can-i 로 kubectl 실행 사용자가 특정 권한을 가졌는지 확인
$ k1 auth can-i get pods
no각각 네임스페이스에 롤(Role)를 생성 후 서비스 어카운트 바인딩
- 롤(Role) : apiGroups 와 resources 로 지정된 리소스에 대해 verbs 권한을 인가
- 실행 가능한 조작(verbs) : *(모두 처리), create(생성), delete(삭제), get(조회), list(목록조회), patch(일부업데이트), update(업데이트), watch(변경감시)
# 각각 네임스페이스내의 모든 권한에 대한 롤 생성
$ cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: role-dev-team
namespace: dev-team
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
EOF
$ cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: role-infra-team
namespace: infra-team
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
EOF
# 롤 확인
$ kubectl get roles -n dev-team
NAME CREATED AT
role-dev-team 2023-05-29T07:35:40Z
$ kubectl get roles -n infra-team
NAME CREATED AT
role-infra-team 2023-05-29T07:36:07Z
$ kubectl get roles -n dev-team -o yaml
$ kubectl describe roles role-dev-team -n dev-team
...
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
*.* [] [] [*]
# 롤바인딩 생성 : '서비스어카운트 <-> 롤' 간 서로 연동
$ cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: roleB-dev-team
namespace: dev-team
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: role-dev-team
subjects:
- kind: ServiceAccount
name: dev-k8s
namespace: dev-team
EOF
$ cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: roleB-infra-team
namespace: infra-team
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: role-infra-team
subjects:
- kind: ServiceAccount
name: infra-k8s
namespace: infra-team
EOF
# 롤바인딩 확인
$ kubectl get rolebindings -n dev-team
NAME ROLE AGE
roleB-dev-team Role/role-dev-team 15s
$ kubectl get rolebindings -n infra-team
NAME ROLE AGE
roleB-infra-team Role/role-infra-team 8s
$ kubectl get rolebindings -n dev-team -o yaml
$ kubectl describe rolebindings roleB-dev-team -n dev-team
...
Role:
Kind: Role
Name: role-dev-team
Subjects:
Kind Name Namespace
---- ---- ---------
ServiceAccount dev-k8s dev-team서비스 어카운트를 지정하여 생성한 파드에서 다시 권한 테스트
# 각각 파드로 Shell 접속하여 정보 확인 : 단축 명령어(alias) 사용
$ alias k1='kubectl exec -it dev-kubectl -n dev-team -- kubectl'
$ alias k2='kubectl exec -it infra-kubectl -n infra-team -- kubectl'
# 권한 테스트
$ k1 get pods
NAME READY STATUS RESTARTS AGE
dev-kubectl 1/1 Running 0 30m
$ k1 run nginx --image nginx:1.20-alpine
pod/nginx created
$ k1 get pods
NAME READY STATUS RESTARTS AGE
dev-kubectl 1/1 Running 0 30m
nginx 1/1 Running 0 23s
$ k1 delete pods nginx
pod "nginx" deleted
$ k1 get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:dev-team:dev-k8s" cannot list resource "pods" in API group "" in the namespace "kube-system"
command terminated with exit code 1
$ k1 get nodes
Error from server (Forbidden): nodes is forbidden: User "system:serviceaccount:dev-team:dev-k8s" cannot list resource "nodes" in API group "" at the cluster scope
command terminated with exit code 1
$ k2 get pods
NAME READY STATUS RESTARTS AGE
infra-kubectl 1/1 Running 0 31m
$ k2 run nginx --image nginx:1.20-alpine
pod/nginx created
$ k2 get pods
NAME READY STATUS RESTARTS AGE
infra-kubectl 1/1 Running 0 31m
nginx 1/1 Running 0 12s
$ k2 delete pods nginx
pod "nginx" deleted
$ k2 get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:infra-team:infra-k8s" cannot list resource "pods" in API group "" in the namespace "kube-system"
command terminated with exit code 1
$ k2 get nodes
Error from server (Forbidden): nodes is forbidden: User "system:serviceaccount:infra-team:infra-k8s" cannot list resource "nodes" in API group "" at the cluster scope
command terminated with exit code 1
# (옵션) kubectl auth can-i 로 kubectl 실행 사용자가 특정 권한을 가졌는지 확인
$ k1 auth can-i get pods
yes리소스 삭제
$ kubectl delete ns dev-team infra-teamEKS 인증/인가
- 사용자/애플리케이션 → k8s 사용 시 ⇒ 인증은 AWS IAM, 인가는 K8S RBAC
RBAC 관련 krew 플러그인
# 설치
$ kubectl krew install access-matrix rbac-tool rbac-view rolesum# Show an RBAC access matrix for server resources
$ kubectl access-matrix # Review access to cluster-scoped resources
$ kubectl access-matrix --namespace default # Review access to namespaced resources in 'default'
# RBAC Lookup by subject (user/group/serviceaccount) name
$ kubectl rbac-tool lookup
$ kubectl rbac-tool lookup system:masters
SUBJECT | SUBJECT TYPE | SCOPE | NAMESPACE | ROLE
+----------------+--------------+-------------+-----------+---------------+
system:masters | Group | ClusterRole | | cluster-admin
$ kubectl rbac-tool lookup system:nodes # eks:node-bootstrapper
W0529 17:21:21.365796 8020 warnings.go:67] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
SUBJECT | SUBJECT TYPE | SCOPE | NAMESPACE | ROLE
+--------------+--------------+-------------+-----------+-----------------------+
system:nodes | Group | ClusterRole | | eks:node-bootstrapper
$ kubectl rbac-tool lookup system:bootstrappers # eks:node-bootstrapper
W0529 17:21:48.620535 8073 warnings.go:67] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
SUBJECT | SUBJECT TYPE | SCOPE | NAMESPACE | ROLE
+----------------------+--------------+-------------+-----------+-----------------------+
system:bootstrappers | Group | ClusterRole | | eks:node-bootstrapper
$ kubectl describe ClusterRole eks:node-bootstrapper
# RBAC List Policy Rules For subject (user/group/serviceaccount) name
$ kubectl rbac-tool policy-rules
$ kubectl rbac-tool policy-rules -e '^system:.*'
# Generate ClusterRole with all available permissions from the target cluster
$ kubectl rbac-tool show
# Shows the subject for the current context with which one authenticates with the cluster
kubectl rbac-tool whoami
{Username: "kubernetes-admin",
UID: "aws-iam-authenticator:90XXXXXXXX:90XXXXXXXX",
Groups: ["system:masters",
"system:authenticated"],
Extra: {accessKeyId: ["AKIA...."],
arn: ["arn:aws:iam::90XXXXXXXX:root"],
canonicalArn: ["arn:aws:iam::90XXXXXXXX:root"],
principalId: ["90XXXXXXXX"],
sessionName: [""]}}
# Summarize RBAC roles for subjects : ServiceAccount(default), User, Group
$ kubectl rolesum -h
$ kubectl rolesum aws-node -n kube-system
ServiceAccount: kube-system/aws-node
Secrets:
Policies:
• [CRB] */aws-node ⟶ [CR] */aws-node
Resource Name Exclude Verbs G L W C U P D DC
*.extensions [*] [-] [-] ✖ ✔ ✔ ✖ ✖ ✖ ✖ ✖
eniconfigs.crd.k8s.amazonaws.com [*] [-] [-] ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
events.[,events.k8s.io] [*] [-] [-] ✖ ✔ ✖ ✔ ✖ ✔ ✖ ✖
namespaces [*] [-] [-] ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
nodes [*] [-] [-] ✔ ✔ ✔ ✖ ✔ ✖ ✖ ✖
pods [*] [-] [-] ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
$ kubectl rolesum -k User system:kube-proxy
User: system:kube-proxy
Policies:
• [CRB] */system:node-proxier ⟶ [CR] */system:node-proxier
Resource Name Exclude Verbs G L W C U P D DC
endpoints [*] [-] [-] ✖ ✔ ✔ ✖ ✖ ✖ ✖ ✖
endpointslices.discovery.k8s.io [*] [-] [-] ✖ ✔ ✔ ✖ ✖ ✖ ✖ ✖
events.[,events.k8s.io] [*] [-] [-] ✖ ✖ ✖ ✔ ✔ ✔ ✖ ✖
nodes [*] [-] [-] ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
services [*] [-] [-] ✖ ✔ ✔ ✖ ✖ ✖ ✖ ✖
$ kubectl rolesum -k Group system:masters
Group: system:masters
Policies:
• [CRB] */cluster-admin ⟶ [CR] */cluster-admin
Resource Name Exclude Verbs G L W C U P D DC
*.* [*] [-] [-] ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔
# [터미널1] A tool to visualize your RBAC permissions
$ kubectl rbac-view
INFO[0000] Getting K8s client
INFO[0000] serving RBAC View and http://localhost:8800
## 이후 해당 작업용PC 공인 IP:8800 웹 접속
$ echo -e "RBAC View Web http://$(curl -s ipinfo.io/ip):8800"
RBAC View Web http://43.201.19.50:8800
rbac-view
인증/인가 완벽 분석 해보기
인증은 AWS IAM, 인가는 K8S RBAC에서 처리합니다.
https://awskoreamarketingasset.s3.amazonaws.com/2022 Summit/pdf/T10S1_EKS 환경을 더 효율적으로 더 안전하게.pdf
https://docs.aws.amazon.com/eks/latest/userguide/cluster-auth.html
- 사용자가 kubectl 명령을 내리면
→ kube config 파일 내 aws eks get-token 명령을 통해
→ EKS Service endpoint(STS)로 해당 Request 전송하여 토큰 요청
→ Response 값으로 토큰이 전송됨. 해당 토큰은 Base64 인코딩된 값으로 이 값을 디코딩해보면 Secure Token Service GetCallerIdentity를 호출하는 Pre-Signed URL인 것을 확인
# sts caller id의 ARN 확인
$ aws sts get-caller-identity --query Arn
"arn:aws:iam::90XXXXXXXXXXq:root"
# kubeconfig 정보 확인
$ cat ~/.kube/config | yh
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1EVXlPVEExTWpFME5Wb1hEVE16TURVeU5qQTFNakUwTlZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTDBYCjIyazFxNCthSTFodkpWYWRjMlhpK0pudHVZSlZ5b1E4TlZuelZWbWd0MVNISXAyaUw4WVFVRVdEa3FCT09yaXkKL01UVmhTbG5Zd1JBbEpNU3BkMkNOT0ZBUWpLQ3RhWnNWeTF3YldSK3NyclJDNCs0Yjg0a3hrTzRXY29xZkNHZgphUlEwcjhhT0g4MHdJSHFXSC9Qd0ozb096U0prcXU2Qk5lUlpPMHNuQkxPWVpGc05xamhGNEJLTVpyZC9ZdjBxCjlYSXc1ZjZSSkxxWnR1YzlaNVUwYlkxOTNYWUcrQWEyNWFDSTRtWEtuREtyU01kbzZVYkh2ZHlQNXB5RVhxNm8KNkxqcVNLbEJoSW9uZm8xTEZkbng4SFR3QlVBMk52ckQ5aGZROENSYjVWNWxFdFlxVVE5Z3F1cmNUcHZSdmpPUwpCeVI2NGgvOFRla3ZNYythZk1VQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZKaUxVOVZibFRGZG1TWFpvbDQ0WWVUeG1UWmFNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBRVp1RmN4N2xuR0dyMEZQMUpTYQpYSlJUMnBiMTN0Ty9zMGRaSHdIL3VodTB1dUZCZUVuZ05zeDBlaU9jQy8vNHRLUCtnTFhEYm50NkZxK3kvM0tOCmtEcGplZ3UvcGdsaVFDUWtCVDlUNGZ3dFpGMGNOeERJeUZZcnNvaVBWODBwc2xuWG5UcmZubTJoc2hOZnNwNTUKTSs1LzJ2M3BtYlN5K3RXenJWN1RYb0xMS0NJVll3R3R3c0hwV0ZiQ0NTQ0VQOTZ6cStyOU5VM25FdVpmNDdOUwozRkVmNWE3eTdQa1kwdTBIck1GL3BXWFdwektwL1F4UDE1MU9zUE11aDV1ZjRzbzkyekJyVFN5b2xiZnFvWEZpCjdQZXg3N1pQNEVVOTE5MEVsa3NKalcrWG95Vk56a3MwRHU1TXRBRmM4TXdPM1ZTY0t3MkdVeHUvZ2UrdEpVTnUKMDYwPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
server: https://442308BCEF7A13DE72CFA354A0708CB7.yl4.ap-northeast-2.eks.amazonaws.com
name: eks-hayley.ap-northeast-2.eksctl.io
contexts:
- context:
cluster: eks-hayley.ap-northeast-2.eksctl.io
namespace: default
user: iam-root-account@eks-hayley.ap-northeast-2.eksctl.io
name: iam-root-account@eks-hayley.ap-northeast-2.eksctl.io
current-context: iam-root-account@eks-hayley.ap-northeast-2.eksctl.io
kind: Config
preferences: {}
users:
- name: iam-root-account@eks-hayley.ap-northeast-2.eksctl.io
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- eks
- get-token
- --output
- json
- --cluster-name
- eks-hayley
- --region
- ap-northeast-2
command: aws
env:
- name: AWS_STS_REGIONAL_ENDPOINTS
value: regional
interactiveMode: IfAvailable
provideClusterInfo: false
# Get a token for authentication with an Amazon EKS cluster.
# This can be used as an alternative to the aws-iam-authenticator.
$ aws eks get-token help
# 임시 보안 자격 증명(토큰)을 요청 : expirationTimestamp 시간경과 시 토큰 재발급됨
$ aws eks get-token --cluster-name $CLUSTER_NAME | jq
{
"kind": "ExecCredential",
"apiVersion": "client.authentication.k8s.io/v1beta1",
"spec": {},
"status": {
"expirationTimestamp": "2023-05-29T11:16:37Z",
"token": "k8s-aws-v1.aHR0cHM6Ly9zdHMuYXAtbm9ydGhlYXNXXXXXXX"
}
}
$ aws eks get-token --cluster-name $CLUSTER_NAME | jq -r '.status.token'
k8s-aws-v1.aHR0cHM6Ly9zdHMuYXAtbm9ydGhlYXN0LTIuYW1hem9uYXdzLmNvbS8_QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNSZYLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUE1RVlLSk1OVUY3WjYyNjdYJTJGMjAyMzA1MjklMkZhcC1ub3J0aGVhc3QtMiUyRnN0cyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjMwNTI5VDExMDMxN1omWC1BbXotRXhwaXJlcz02MCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QlM0J4LWs4cy1hd3MtaWQmWC1BbXotU2lnbmF0dXJlPWRkZWI5OTJjZDYzNDQ3MjFjM2U1MDVlYmQ0YWIzYTMwOTBkNTdiMjA1NGUzOGUwOWZmNmFiNzkyMDlmMWExMjM2. kubecl Client Go 라이브러리는 이 Pre-Signed URL을 Bearer Token으로 EKS API Cluster Endpoint로 요청을 보내면
3. IAM Authentication Server는 Token Authentication Webhook이 호출되고 STS GetCallerIdentity를 호출해서 해당 호출에 대한 인증을 거쳐 UserID와 User 나 Role에 대한 ARN을 반환하게 됨
→ 이 정보를 IAM의 User나 Role을 K8S의 User 나 Group으로 매핑하는 Config Map인 AWS Auth에서 K8S의 보안 주체(subjects)를 리턴하게 됨
→ Authentication에서는 Token Review라는 데이터 타입으로 username과 group을 반환함
# tokenreviews api 리소스 확인
$ kubectl api-resources | grep authentication
tokenreviews authentication.k8s.io/v1 false TokenReview
$ kubectl explain tokenreviews
...
DESCRIPTION:
TokenReview attempts to authenticate a token to a known user. Note:
TokenReview requests may be cached by the webhook token authenticator
plugin in the kube-apiserver.4. 이 값을 이용해 K8S의 RBAC 체계에서 요청에 대한 허용이나 거부를 하게 됨
# Webhook api 리소스 확인
$ kubectl api-resources | grep Webhook
mutatingwebhookconfigurations admissionregistration.k8s.io/v1 false MutatingWebhookConfiguration
validatingwebhookconfigurations admissionregistration.k8s.io/v1 false ValidatingWebhookConfiguration
# validatingwebhookconfigurations 리소스 확인
$ kubectl get validatingwebhookconfigurations
NAME WEBHOOKS AGE
aws-load-balancer-webhook 3 4h44m
eks-aws-auth-configmap-validation-webhook 1 5h44m
kube-prometheus-stack-admission 1 4h34m
vpc-resource-validating-webhook 2 5h44m
$ kubectl get validatingwebhookconfigurations eks-aws-auth-configmap-validation-webhook -o yaml | kubectl neat | yh
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: eks-aws-auth-configmap-validation-webhook
webhooks:
- admissionReviewVersions:
- v1
clientConfig:
caBundle: LS0tLS1CRU......
url: https://127.0.0.1:21375/validate
failurePolicy: Ignore
matchPolicy: Equivalent
name: eks-aws-auth-configmap-validation-webhook.amazonaws.com
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
rules:
- apiGroups:
- ""
apiVersions:
- v1
operations:
- UPDATE
resources:
- configmaps
scope: '*'
sideEffects: None
timeoutSeconds: 5
# aws-auth 컨피그맵 확인
$ kubectl get cm -n kube-system aws-auth -o yaml | kubectl neat | yh
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::90XXXXXXXX:role/eksctl-eks-hayley-nodegroup-ng1-NodeInstanceRole-U2A82OIJHE30
username: system:node:{{EC2PrivateDNSName}}
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
#---<아래 생략(추정), ARN은 EKS를 설치한 IAM User , 여기 있었을경우 만약 실수로 삭제 시 복구가 가능했을까?---
mapUsers: |
- groups:
- system:masters
userarn: arn:aws:iam::111122223333:user/admin
username: kubernetes-admin
# EKS 설치한 IAM User 정보 >> system:authenticated는 어떤 방식으로 추가가 되었을까
$ kubectl rbac-tool whoami
{Username: "kubernetes-admin",
UID: "aws-iam-authenticator:90XXXXXXXX:90XXXXXXXX",
Groups: ["system:masters",
"system:authenticated"],
Extra: {accessKeyId: ["AKIA5EYKJMNUF7Z6267X"],
arn: ["arn:aws:iam::90XXXXXXXX:root"],
canonicalArn: ["arn:aws:iam::90XXXXXXXX:root"],
principalId: ["90XXXXXXXX"],
sessionName: [""]}}
# system:masters , system:authenticated 그룹의 정보 확인
$ kubectl rbac-tool lookup system:masters
W0529 20:11:32.181082 10082 warnings.go:67] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
SUBJECT | SUBJECT TYPE | SCOPE | NAMESPACE | ROLE
+----------------+--------------+-------------+-----------+---------------+
system:masters | Group | ClusterRole | | cluster-admin
$ kubectl rbac-tool lookup system:authenticated
W0529 20:11:49.329790 10138 warnings.go:67] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
SUBJECT | SUBJECT TYPE | SCOPE | NAMESPACE | ROLE
+----------------------+--------------+-------------+-----------+----------------------------------+
system:authenticated | Group | ClusterRole | | system:discovery
system:authenticated | Group | ClusterRole | | system:basic-user
system:authenticated | Group | ClusterRole | | eks:podsecuritypolicy:privileged
system:authenticated | Group | ClusterRole | | system:public-info-viewer
$ kubectl rolesum -k Group system:masters
Group: system:masters
Policies:
• [CRB] */cluster-admin ⟶ [CR] */cluster-admin
Resource Name Exclude Verbs G L W C U P D DC
*.* [*] [-] [-] ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔
$ kubectl rolesum -k Group system:authenticated
W0529 20:12:21.244018 10245 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
Group: system:authenticated
Policies:
• [CRB] */eks:podsecuritypolicy:authenticated ⟶ [CR] */eks:podsecuritypolicy:privileged
Name PRIV RO-RootFS Volumes Caps SELinux RunAsUser FSgroup SUPgroup
eks.privileged True False [*] [*] RunAsAny RunAsAny RunAsAny RunAsAny
• [CRB] */system:basic-user ⟶ [CR] */system:basic-user
Resource Name Exclude Verbs G L W C U P D DC
selfsubjectaccessreviews.authorization.k8s.io [*] [-] [-] ✖ ✖ ✖ ✔ ✖ ✖ ✖ ✖
selfsubjectrulesreviews.authorization.k8s.io [*] [-] [-] ✖ ✖ ✖ ✔ ✖ ✖ ✖ ✖
• [CRB] */system:discovery ⟶ [CR] */system:discovery
• [CRB] */system:public-info-viewer ⟶ [CR] */system:public-info-viewer
# system:masters 그룹이 사용 가능한 클러스터 롤 확인 : cluster-admin
kubectl describe clusterrolebindings.rbac.authorization.k8s.io cluster-admin
Name: cluster-admin
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
Role:
Kind: ClusterRole
Name: cluster-admin
Subjects:
Kind Name Namespace
---- ---- ---------
Group system:masters
# cluster-admin 의 PolicyRule 확인 : 모든 리소스 사용 가능!
$ kubectl describe clusterrole cluster-admin
Name: cluster-admin
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
*.* [] [] [*]
[*] [] [*]
# system:authenticated 그룹이 사용 가능한 클러스터 롤 확인
$ kubectl describe ClusterRole system:discovery
Name: system:discovery
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
[/api/*] [] [get]
[/api] [] [get]
[/apis/*] [] [get]
[/apis] [] [get]
[/healthz] [] [get]
[/livez] [] [get]
[/openapi/*] [] [get]
[/openapi] [] [get]
[/readyz] [] [get]
[/version/] [] [get]
[/version] [] [get]
$ kubectl describe ClusterRole system:public-info-viewer
Name: system:public-info-viewer
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
[/healthz] [] [get]
[/livez] [] [get]
[/readyz] [] [get]
[/version/] [] [get]
[/version] [] [get]
$ kubectl describe ClusterRole system:basic-user
Name: system:basic-user
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
selfsubjectaccessreviews.authorization.k8s.io [] [] [create]
selfsubjectrulesreviews.authorization.k8s.io [] [] [create]
$ kubectl describe ClusterRole eks:podsecuritypolicy:privileged
Name: eks:podsecuritypolicy:privileged
Labels: eks.amazonaws.com/component=pod-security-policy
kubernetes.io/cluster-service=true
Annotations: <none>
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
podsecuritypolicies.policy [] [eks.privileged] [use]이와 같이 kubectl 명령을 내리면 IAM을 활용해 사용자를 인증하고 K8S에 정의된 권한에 따라 해당 명령을 인가하게 됨 — 링크
실습 — 데브옵스 신입 사원을 위한 myeks-bastion-2에 설정
- [myeks-bastion] testuser 사용자 생성
# testuser 사용자 생성
$ aws iam create-user --user-name testuser
{
"User": {
"Path": "/",
"UserName": "testuser",
"UserId": "AIDA5EYKJMNUNCSMOFLM2",
"Arn": "arn:aws:iam::90XXXXXXXXXX:user/testuser",
"CreateDate": "2023-05-29T11:33:46+00:00"
}
}
# 사용자에게 프로그래밍 방식 액세스 권한 부여
$ aws iam create-access-key --user-name testuser
{
"AccessKey": {
"UserName": "testuser",
"AccessKeyId": "AKIAXXXXXX",
"Status": "Active",
"SecretAccessKey": "Wo8XXXXXXXX",
"CreateDate": "2023-05-29T11:34:15+00:00"
}
}
# testuser 사용자에 정책을 추가
$ aws iam attach-user-policy --policy-arn arn:aws:iam::aws:policy/AdministratorAccess --user-name testuser
# get-caller-identity 확인
$ aws sts get-caller-identity --query Arn
"arn:aws:iam::90XXXXXXXXXX:root"
# EC2 IP 확인 : myeks-bastion-EC2-2 PublicIPAdd 확인
$ aws ec2 describe-instances --query "Reservations[*].Instances[*].{PublicIPAdd:PublicIpAddress,PrivateIPAdd:PrivateIpAddress,InstanceName:Tags[?Key=='Name']|[0].Value,Status:State.Name}" --filters Name=instance-state-name,Values=running --output table
----------------------------------------------------------------------------
| DescribeInstances |
+---------------------------+----------------+-----------------+-----------+
| InstanceName | PrivateIPAdd | PublicIPAdd | Status |
+---------------------------+----------------+-----------------+-----------+
| eks-hayley-ng1-Node | 192.168.3.57 | 13.125.186.152 | running |
| eks-hayley-ng1-Node | 192.168.2.145 | 15.165.209.247 | running |
| eks-hayley-bastion-EC2-2 | 192.168.1.200 | 13.124.177.23 | running |
| eks-hayley-bastion-EC2 | 192.168.1.100 | 43.201.19.50 | running |
| eks-hayley-ng1-Node | 192.168.1.83 | 13.125.104.220 | running |
+---------------------------+----------------+-----------------+-----------+
2. [myeks-bastion-2] testuser 자격증명 설정 및 확인
# get-caller-identity 확인 >> 왜 안될까요?
$ aws sts get-caller-identity --query Arn
Unable to locate credentials. You can configure credentials by running "aws configure".
# testuser 자격증명 설정
$ aws configure
AWS Access Key ID [None]: AKIA5ILF2F...
AWS Secret Access Key [None]: ePpXdhA3cP....
Default region name [None]: ap-northeast-2
# get-caller-identity 확인
$ aws sts get-caller-identity --query Arn
"arn:aws:iam::90XXXXXXXXX:user/testuser"
# kubectl 시도 >> testuser도 AdministratorAccess 권한을 가지고 있는데, 실패 이유는?
$ kubectl get node -v6
I0529 20:36:54.382755 2931 round_trippers.go:553] GET http://localhost:8080/api?timeout=32s in 1 milliseconds
E0529 20:36:54.383205 2931 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
I0529 20:36:54.383345 2931 cached_discovery.go:120] skipped caching discovery info due to Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
$ ls ~/.kube
ls: cannot access /root/.kube: No such file or directory
3. [myeks-bastion] testuser에 system:masters 그룹 부여로 EKS 관리자 수준 권한 설정
# 방안1 : eksctl 사용 >> iamidentitymapping 실행 시 aws-auth 컨피그맵 작성해줌
# Creates a mapping from IAM role or user to Kubernetes user and groups
$ eksctl create iamidentitymapping --cluster $CLUSTER_NAME --username testuser --group system:masters --arn arn:aws:iam::$ACCOUNT_ID:user/testuser
2023-05-29 20:38:14 [ℹ] checking arn arn:aws:iam::90XXXXX:user/testuser against entries in the auth ConfigMap
2023-05-29 20:38:14 [ℹ] adding identity "arn:aws:iam::90XXXX:user/testuser" to auth ConfigMap
# 확인
$ kubectl get cm -n kube-system aws-auth -o yaml | kubectl neat | yh
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::90XXXXXXXX:role/eksctl-eks-hayley-nodegroup-ng1-NodeInstanceRole-U2A82OIJHE30
username: system:node:{{EC2PrivateDNSName}}
mapUsers: |
- groups:
- system:masters
userarn: arn:aws:iam::90XXXXXXXX:user/testuser
username: testuser
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
# 확인 : 기존에 있는 role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-YYYYY 는 어떤 역할/동작을 하는 걸까요?
# role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-YYYYY 역할은
# : EKS 클러스터 및 상호 작용하는 AWS 리소스 내에서 갖는 권한 및 액세스 권한을 정의
$ eksctl get iamidentitymapping --cluster $CLUSTER_NAME
ARN USERNAME GROUPS ACCOUNT
arn:aws:iam::90XXXXXXXX:role/eksctl-eks-hayley-nodegroup-ng1-NodeInstanceRole-U2A82OIJHE30 system:node:{{EC2PrivateDNSName}} system:bootstrappers,system:nodes
arn:aws:iam::90XXXXXXXX:user/testuser testuser system:masters 4. [myeks-bastion-2] testuser kubeconfig 생성 및 kubectl 사용 확인
# testuser kubeconfig 생성 >> aws eks update-kubeconfig 실행이 가능한 이유는?, 3번 설정 후 약간의 적용 시간 필요
$ aws eks update-kubeconfig --name $CLUSTER_NAME --user-alias testuser
Added new context testuser to /root/.kube/config
# 첫번째 bastic ec2의 config와 비교해보자
$ cat ~/.kube/config | yh
[myeks-bastion]
$ cat ~/.kube/config | yh
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0....
server: https://442308BCEF7A13DE72CFA354A0708CB7.yl4.ap-northeast-2.eks.amazonaws.com
name: eks-hayley.ap-northeast-2.eksctl.io
contexts:
- context:
cluster: eks-hayley.ap-northeast-2.eksctl.io
namespace: default
user: iam-root-account@eks-hayley.ap-northeast-2.eksctl.io
name: iam-root-account@eks-hayley.ap-northeast-2.eksctl.io
current-context: iam-root-account@eks-hayley.ap-northeast-2.eksctl.io
kind: Config
preferences: {}
users:
- name: iam-root-account@eks-hayley.ap-northeast-2.eksctl.io
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- eks
- get-token
- --output
- json
- --cluster-name
- eks-hayley
- --region
- ap-northeast-2
command: aws
env:
- name: AWS_STS_REGIONAL_ENDPOINTS
value: regional
interactiveMode: IfAvailable
provideClusterInfo: false
[myeks-bastion-2]
$ cat ~/.kube/config | yh
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0...
server: https://442308BCEF7A13DE72CFA354A0708CB7.yl4.ap-northeast-2.eks.amazonaws.com
name: arn:aws:eks:ap-northeast-2:90XXXXXX:cluster/eks-hayley
contexts:
- context:
cluster: arn:aws:eks:ap-northeast-2:90XXXXXX:cluster/eks-hayley
user: testuser
name: testuser
current-context: testuser
kind: Config
preferences: {}
users:
- name: testuser
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- --region
- ap-northeast-2
- eks
- get-token
- --cluster-name
- eks-hayley
- --output
- json
command: aws
# kubectl 사용 확인
$ kubectl ns default
Context "testuser" modified.
Active namespace is "default".
$ kubectl get node -v6
NAME STATUS ROLES AGE VERSION
ip-192-168-1-83.ap-northeast-2.compute.internal Ready <none> 6h15m v1.24.13-eks-0a21954
ip-192-168-2-145.ap-northeast-2.compute.internal Ready <none> 6h16m v1.24.13-eks-0a21954
ip-192-168-3-57.ap-northeast-2.compute.internal Ready <none> 6h15m v1.24.13-eks-0a21954
# rbac-tool 후 확인 >> 기존 계정과 비교해보자 >> system:authenticated 는 system:masters 설정 시 따라오는 것 같은데, 추가 동작 원리는 모르겠네요???
$ kubectl krew install rbac-tool && kubectl rbac-tool whoami
{Username: "testuser",
UID: "aws-iam-authenticator:90XXXXXX:AIDA5EYKJMNUNCSMOFLM2",
Groups: ["system:masters",
"system:authenticated"],
Extra: {accessKeyId: ["AKIA5EYKJMNUJAY5P4VQ"],
arn: ["arn:aws:iam::90XXXXXX:user/testuser"],
canonicalArn: ["arn:aws:iam::90XXXXXX:user/testuser"],
principalId: ["AIDA5EYKJMNUNCSMOFLM2"],
sessionName: [""]}}5. [myeks-bastion] testuser 의 Group 변경(system:masters → system:authenticated)으로 RBAC 동작 확인
# 방안2 : 아래 edit로 mapUsers 내용 직접 수정 system:authenticated
$ kubectl edit cm -n kube-system aws-auth
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::90XXXXXXXX:role/eksctl-eks-hayley-nodegroup-ng1-NodeInstanceRole-U2A82OIJHE30
username: system:node:{{EC2PrivateDNSName}}
mapUsers: |
- groups:
- system:authenticated
userarn: arn:aws:iam::90XXXXXXXX:user/testuser
username: testuser
kind: ConfigMap
metadata:
creationTimestamp: "2023-05-29T05:30:09Z"
name: aws-auth
namespace: kube-system
resourceVersion: "91372"
uid: 3523cf31-f7e6-49d7-8bba-2646c16f9fed
# 확인
$ eksctl get iamidentitymapping --cluster $CLUSTER_NAME
ARN USERNAME GROUPS ACCOUNT
arn:aws:iam::90XXXXXXXX:role/eksctl-eks-hayley-nodegroup-ng1-NodeInstanceRole-U2A82OIJHE30 system:node:{{EC2PrivateDNSName}} system:bootstrappers,system:nodes
arn:aws:iam::90XXXXXXXX:user/testuser testuser system:authenticated6. [myeks-bastion-2] testuser kubectl 사용 확인
# 시도
$ kubectl get node -v6
I0529 20:52:46.202811 3509 loader.go:373] Config loaded from file: /root/.kube/config
I0529 20:52:47.010377 3509 round_trippers.go:553] GET https://442308BCEF7A13DE72CFA354A0708CB7.yl4.ap-northeast-2.eks.amazonaws.com/api/v1/nodes?limit=500 403 Forbidden in 796 milliseconds
I0529 20:52:47.010647 3509 helpers.go:246] server response object: [{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "nodes is forbidden: User \"testuser\" cannot list resource \"nodes\" in API group \"\" at the cluster scope",
"reason": "Forbidden",
"details": {
"kind": "nodes"
},
"code": 403
}]
Error from server (Forbidden): nodes is forbidden: User "testuser" cannot list resource "nodes" in API group "" at the cluster scope
$ kubectl api-resources -v57. [myeks-bastion]에서 testuser IAM 맵핑 삭제
# testuser IAM 맵핑 삭제
$ eksctl delete iamidentitymapping --cluster $CLUSTER_NAME --arn arn:aws:iam::$ACCOUNT_ID:user/testuser
# Get IAM identity mapping(s)
$ eksctl get iamidentitymapping --cluster $CLUSTER_NAME
ARN USERNAME GROUPS ACCOUNT
arn:aws:iam::90XXXXXXXX:role/eksctl-eks-hayley-nodegroup-ng1-NodeInstanceRole-U2A82OIJHE30 system:node:{{EC2PrivateDNSName}} system:bootstrappers,system:nodes
$ kubectl get cm -n kube-system aws-auth -o yaml | yh
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::90XXXXXXXX:role/eksctl-eks-hayley-nodegroup-ng1-NodeInstanceRole-U2A82OIJHE30
username: system:node:{{EC2PrivateDNSName}}
mapUsers: |
[]
kind: ConfigMap
metadata:
creationTimestamp: "2023-05-29T05:30:09Z"
name: aws-auth
namespace: kube-system
resourceVersion: "95257"
uid: 3523cf31-f7e6-49d7-8bba-2646c16f9fed8. [myeks-bastion-2] testuser kubectl 사용 확인
# 시도
$ kubectl get node -v6
I0529 20:55:43.538833 3705 loader.go:373] Config loaded from file: /root/.kube/config
I0529 20:55:45.202367 3705 round_trippers.go:553] GET https://442308BCEF7A13DE72CFA354A0708CB7.yl4.ap-northeast-2.eks.amazonaws.com/api/v1/nodes?limit=500 401 Unauthorized in 1649 milliseconds
I0529 20:55:45.202633 3705 helpers.go:246] server response object: [{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "Unauthorized",
"reason": "Unauthorized",
"code": 401
}]
error: You must be logged in to the server (Unauthorized)
$ kubectl api-resources -v5
E0529 20:56:02.330254 3760 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
I0529 20:56:02.330300 3760 cached_discovery.go:120] skipped caching discovery info due to the server has asked for the client to provide credentials
NAME SHORTNAMES APIVERSION NAMESPACED KIND
I0529 20:56:02.330544 3760 helpers.go:246] server response object: [{
"metadata": {},
"status": "Failure",
"message": "the server has asked for the client to provide credentials",
"reason": "Unauthorized",
"details": {
"causes": [
{
"reason": "UnexpectedServerResponse",
"message": "unknown"
}
]
},
"code": 401
}]
error: You must be logged in to the server (the server has asked for the client to provide credentials)
GetCallerIdentity
위 설명과 같이 kube config 파일 내 aws eks get-token 명령을 통해 EKS Service endpoint(STS)로 해당 Request 전송하여 토큰 요청하고 Response 값으로 토큰이 전송됩니다. 해당 토큰은 Base64 인코딩된 값으로 이 값을 디코딩해보면 Secure Token Service GetCallerIdentity를 호출하는 Pre-Signed URL인 것을 확인할 수 있습니다. (콘솔 — CloudTrail)
IRSA
- 파드가 특정 IAM 역할로 Assume 할때 토큰을 AWS에 전송하고, AWS는 토큰과 EKS IdP를 통해 해당 IAM 역할을 사용할 수 있는지 검증합니다.
k8s파드 → AWS 서비스 사용 시 ⇒ AWS STS/IAM ↔ IAM OIDC Identity Provider(EKS IdP) 인증/인가
- Service Account Token Volume Projection : ‘서비스 계정 토큰’의 시크릿 기반 볼륨 대신 ‘projected volume’ 사용
- JWT(JSON Web Token) : X.509 Certificate의 lightweight JSON 버전
- OIDC : 사용자를 인증해 사용자에게 액세스 권한을 부여할 수 있게 해주는 프로토콜 ⇒ [커피고래]님 블로그 OpenID Connect — 링크
https://awskoreamarketingasset.s3.amazonaws.com/2022 Summit/pdf/T10S1_EKS 환경을 더 효율적으로 더 안전하게.pdf
IRSA 실습1
# 파드1 생성
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: eks-iam-test1
spec:
containers:
- name: my-aws-cli
image: amazon/aws-cli:latest
args: ['s3', 'ls']
restartPolicy: Never
automountServiceAccountToken: false
EOF
# 확인
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
eks-iam-test1 1/1 Running 0 13s
$ kubectl describe pod
# 로그 확인
$ kubectl logs eks-iam-test1
An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied
# 파드1 삭제
$ kubectl delete pod eks-iam-test1
###Cloud Trail
JSON view
{
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROAXXXXXX",
"arn": "arn:aws:sts::903XXXXXXXXX:assumed-role/eksctl-eks-hayley-nodegroup-ng1-NodeInstanceRole-U2A82OIJHE30/i-0ffdbe366e6ee8da9",
"accountId": "903XXXXXXXXX",
"accessKeyId": "ASI.....",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "AROAXXXXXX",
"arn": "arn:aws:iam::903XXXXXXXXX:role/eksctl-eks-hayley-nodegroup-ng1-NodeInstanceRole-U2AXXXXXXX",
"accountId": "903XXXXXXXXX",
"userName": "eksctl-eks-hayley-nodegroup-ng1-NodeInstanceRole-U2AXXXXXXX"
},
"webIdFederationData": {},
"attributes": {
"creationDate": "2023-05-29T12:34:57Z",
"mfaAuthenticated": "false"
},
"ec2RoleDelivery": "2.0"
}
},
"eventTime": "2023-05-29T12:48:57Z",
"eventSource": "s3.amazonaws.com",
"eventName": "ListBuckets",
"awsRegion": "ap-northeast-2",
"sourceIPAddress": "15.165.209.247",
"userAgent": "[aws-cli/2.11.23 Python/3.11.3 Linux/5.10.178-162.673.amzn2.x86_64 docker/x86_64.amzn.2 prompt/off command/s3.ls]",
"errorCode": "AccessDenied",
"errorMessage": "Access Denied",
"requestParameters": {
"Host": "s3.ap-northeast-2.amazonaws.com"
},
...IRSA 실습2 — Kubernetes Service Accounts
- Service Accounts이 생성되면 JWT 토큰이 자동으로 Kubernetes Secret로 생성됩니다.
# 파드2 생성
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: eks-iam-test2
spec:
containers:
- name: my-aws-cli
image: amazon/aws-cli:latest
command: ['sleep', '36000']
restartPolicy: Never
EOF
# 확인
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
eks-iam-test2 1/1 Running 0 8s
$ kubectl describe pod
# aws 서비스 사용 시도
$ kubectl exec -it eks-iam-test2 -- aws s3 ls
An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied
command terminated with exit code 254
# 서비스 어카운트 토큰 확인
$ SA_TOKEN=$(kubectl exec -it eks-iam-test2 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)
$ echo $SA_TOKEN
eyJhbGciOiJSUzI1NiIsImtpZCI6IjQ0YTg0NjczZjQwYWNmYmRjNzE2MzZjMzhkYjU2MTJhMjlmOTQ3MDIifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTcxNjkwMDg2NCwiaWF0IjoxNjg1MzY0ODY0LCJpc3MiOiJodHRwczovL29pZGMuZWtzLmFwLW5vcnRoZWFzdC0yLmFtYXpvbmF3cy5jb20vaWQvNDQyMzA4QkNFRjdBMTNERTcyQ0ZBMzU0QTA3MDhDQjciLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6ImRlZmF1bHQiLCJwb2QiOnsibmFtZSI6ImVrcy1pYW0tdGVzdDIiLCJ1aWQiOiJjOTQzODkxNi0zOTU2LTQ5YjQtYjVlMi0yYmQ3MmZkY2M0YTkifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImRlZmF1bHQiLCJ1aWQiOiI2NTY4Yjc5Ni04NGUwLTQwNTMtODEyYy02MWU3OTViMWY5NmQifSwid2FybmFmdGVyIjoxNjg1MzY4NDcxfSwibmJmIjoxNjg1MzY0ODY0LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpkZWZhdWx0In0.pytH1Xku2N9rx7CNGfkqbh3Ic8Loi3xi6S9sL65-y2Q-WE3p16-sDBbI_iw5dN5nWlnxP0S08svwGhAXiyfqjUVSEY4b1WEMmnMSnzQTJvJtBuiJthsNtJpT3l4KBj9xbmRcuIP6gNL8ZYfMIiq3rFC3CsudGqwaPd3oIHT02KjHRbIukOnxDJ-NdwiJcf9JEONgFWzNN9yM94soJeVrXUCs1ZGsxoVv-rhU6jlSWF1Xo64NLt6N9JH2XP0EhYAFZqMu1gWNGlRB20xZFtsupJJ9fpiWkKeW3V29T9LvWmhG5ti7lIDaui9ppp1e_P6CC0XpfBEyC8GZXWECepc_Hg
# jwt 혹은 아래 JWT 웹 사이트 이용 (https://jwt.io/)
jwt decode $SA_TOKEN --json --iso8601
...
#헤더
{
"alg": "RS256",
"kid": "44a84673f40acfbdc71636c38db5612a29f94702"
}
# 페이로드 : OAuth2에서 쓰이는 aud, exp 속성 확인! > projectedServiceAccountToken 기능으로 토큰에 audience,exp 항목을 덧붙힘
## iss 속성 : EKS OpenID Connect Provider(EKS IdP) 주소 > 이 EKS IdP를 통해 쿠버네티스가 발급한 토큰이 유요한지 검증
{
"aud": [
"https://kubernetes.default.svc"
],
"exp": 1716900864,
"iat": 1685364864,
"iss": "https://oidc.eks.ap-northeast-2.amazonaws.com/id/442308BCEF7A13DE72CFA354A0708CB7",
"kubernetes.io": {
"namespace": "default",
"pod": {
"name": "eks-iam-test2",
"uid": "c9438916-3956-49b4-b5e2-2bd72fdcc4a9"
},
"serviceaccount": {
"name": "default",
"uid": "6568b796-84e0-4053-812c-61e795b1f96d"
},
"warnafter": 1685368471
},
"nbf": 1685364864,
"sub": "system:serviceaccount:default:default"
}
# 파드2 삭제
$ kubectl delete pod eks-iam-test2IRSA 실습3 — Amazon EKS Pod Identity Webhook
The eksctl create iamserviceaccount command creates:
- A Kubernetes Service Account
- An IAM role with the specified IAM policy
- A trust policy on that IAM role
생성된 IAM Role Arn으로 Kubernetes 서비스 계정에 주석을 추가합니다.
# Create an iamserviceaccount - AWS IAM role bound to a Kubernetes service account
$ eksctl create iamserviceaccount \
--name my-sa \
--namespace default \
--cluster $CLUSTER_NAME \
--approve \
--attach-policy-arn $(aws iam list-policies --query 'Policies[?PolicyName==`AmazonS3ReadOnlyAccess`].Arn' --output text)
# 확인 >> 웹 관리 콘솔에서 CloudFormation Stack >> IAM Role 확인
# aws-load-balancer-controller IRSA는 어떤 동작을 수행할 것 인지 생각해보자!
$ eksctl get iamserviceaccount --cluster $CLUSTER_NAME
NAMESPACE NAME ROLE ARN
default my-sa arn:aws:iam::90XXXXXXXX:role/eksctl-eks-hayley-addon-iamserviceaccount-de-Role1-1HXXXXXX
kube-system aws-load-balancer-controller arn:aws:iam::90XXXXXXXX:role/eksctl-eks-hayley-addon-iamserviceaccount-ku-Role1-1HXXXXXX
# Inspecting the newly created Kubernetes Service Account, we can see the role we want it to assume in our pod.
$ kubectl get sa
NAME SECRETS AGE
default 0 7h40m
my-sa 0 57s
$ kubectl describe sa my-sa
Name: my-sa
Namespace: default
Labels: app.kubernetes.io/managed-by=eksctl
Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::90XXXXXXXX:role/eksctl-eks-hayley-addon-iamserviceaccount-de-Role1-1HXXXXXX
Image pull secrets: <none>
Mountable secrets: <none>
Tokens: <none>
Events: <none>-> IAM Role 확인(eksctl-eks-hayley-addon-iamserviceaccount-de-Role1–1HXXXXXX) -> Select the Trust relationships -> Edit trust relationship to view the policy document.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::90XXXXXXXX:oidc-provider/oidc.eks.ap-northeast-2.amazonaws.com/id/442308BCEF7A13DE72CFA354A0708CB7"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.ap-northeast-2.amazonaws.com/id/442308BCEF7A13DE72CFA354A0708CB7:aud": "sts.amazonaws.com",
"oidc.eks.ap-northeast-2.amazonaws.com/id/442308BCEF7A13DE72CFA354A0708CB7:sub": "system:serviceaccount:default:my-sa"
}
}
}
]
}- 이 정책은 identity system:serviceaccount:default:my-sa가 sts:AssumeRoleWithWebIdentity 작업을 사용하여 role을 맡도록 허용하고 있음을 알 수 있습니다. 이 정책의 주체는 OIDC 공급자입니다.
- 이제 Kubernetes 포드 내에서 이 새로운 서비스 계정을 사용할 때 어떤 일이 발생하는지 살펴보겠습니다.
# 파드3번 생성
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: eks-iam-test3
spec:
serviceAccountName: my-sa
containers:
- name: my-aws-cli
image: amazon/aws-cli:latest
command: ['sleep', '36000']
restartPolicy: Never
EOF
# 해당 SA를 파드가 사용 시 mutatingwebhook으로 Env,Volume 추가함
$ kubectl get mutatingwebhookconfigurations pod-identity-webhook -o yaml | kubectl neat | yh
# 파드 생성 yaml에 없던 내용이 추가됨!!!!!
# Pod Identity Webhook은 mutating webhook을 통해 아래 Env 내용과 1개의 볼륨을 추가함
$ kubectl get pod eks-iam-test3
NAME READY STATUS RESTARTS AGE
eks-iam-test3 1/1 Running 0 105s
$ kubectl describe pod eks-iam-test3
...
Environment:
AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_DEFAULT_REGION: ap-northeast-2
AWS_REGION: ap-northeast-2
AWS_ROLE_ARN: arn:aws:iam::90XXXXX:role/eksctl-eks-hayley-addon-iamserviceaccount-de-Role1-1HXXXXX
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-r76z8 (ro)
...
Volumes:
aws-iam-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 86400
kube-api-access-r76z8:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
...
# 파드에서 aws cli 사용 확인
$ eksctl get iamserviceaccount --cluster $CLUSTER_NAME
NAMESPACE NAME ROLE ARN
default my-sa arn:aws:iam::90XXXXX:role/eksctl-eks-hayley-addon-iamserviceaccount-de-Role1-1HXXXXX
kube-system aws-load-balancer-controller arn:aws:iam::90XXXXX:role/eksctl-eks-hayley-addon-iamserviceaccount-ku-Role1-1HXXXXX
$ kubectl exec -it eks-iam-test3 -- aws sts get-caller-identity --query Arn
"arn:aws:sts::90XXXXX:assumed-role/eksctl-eks-hayley-addon-iamserviceaccount-de-Role1-1HXXXXX/botocore-session-1685365912"
# 되는 것고 안되는 것은 왜그런가?
$ kubectl exec -it eks-iam-test3 -- aws s3 ls
2023-03-05 10:05:52 hayley-bucket-test
$ kubectl exec -it eks-iam-test3 -- aws ec2 describe-instances --region ap-northeast-2
An error occurred (UnauthorizedOperation) when calling the DescribeInstances operation: You are not authorized to perform this operation.
command terminated with exit code 254
$ kubectl exec -it eks-iam-test3 -- aws ec2 describe-vpcs --region ap-northeast-2
An error occurred (UnauthorizedOperation) when calling the DescribeVpcs operation: You are not authorized to perform this operation.
command terminated with exit code 254
AssumeRoleWithWebIdentity
- Kubectl 및 jq를 사용하여 Pod를 검사하면 이제 Pod에 두 개의 볼륨이 마운트된 것을 볼 수 있습니다.
- 두 번째는 mutating webhook을 통해 마운트되었습니다.
- aws-iam-token은 여전히 Kubernetes API 서버에서 생성되지만 새로운 OIDC JWT 대상이 있습니다.
# 파드에 볼륨 마운트 2개 확인
$ kubectl get pod eks-iam-test3 -o json | jq -r '.spec.containers | .[].volumeMounts'
[
{
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
"name": "kube-api-access-r76z8",
"readOnly": true
},
{
"mountPath": "/var/run/secrets/eks.amazonaws.com/serviceaccount",
"name": "aws-iam-token",
"readOnly": true
}
]
# aws-iam-token 볼륨 정보 확인 : JWT 토큰이 담겨져있고, exp, aud 속성이 추가되어 있음
kubectl get pod eks-iam-test3 -o json | jq -r '.spec.volumes[] | select(.name=="aws-iam-token")'
{
"name": "aws-iam-token",
"projected": {
"defaultMode": 420,
"sources": [
{
"serviceAccountToken": {
"audience": "sts.amazonaws.com",
"expirationSeconds": 86400,
"path": "token"
}
}
]
}
}
# api 리소스 확인
$ kubectl api-resources |grep hook
mutatingwebhookconfigurations admissionregistration.k8s.io/v1 false MutatingWebhookConfiguration
validatingwebhookconfigurations admissionregistration.k8s.io/v1 false ValidatingWebhookConfiguration
#
$ kubectl explain mutatingwebhookconfigurations
#
$ kubectl get MutatingWebhookConfiguration
NAME WEBHOOKS AGE
aws-load-balancer-webhook 3 6h54m
kube-prometheus-stack-admission 1 6h44m
pod-identity-webhook 1 7h54m
vpc-resource-mutating-webhook 1 7h54m
# pod-identity-webhook 확인
$ kubectl describe MutatingWebhookConfiguration pod-identity-webhook
...
Webhooks:
Admission Review Versions:
v1beta1
Client Config:
Ca Bundle: LS0tLS1CRUdJTiB....
URL: https://127.0.0.1:23443/mutate
Failure Policy: Ignore
Match Policy: Equivalent
Name: iam-for-pods.amazonaws.com
Namespace Selector:
Object Selector:
Reinvocation Policy: IfNeeded
Rules:
API Groups:
API Versions:
v1
Operations:
CREATE
Resources:
pods
Scope: *
Side Effects: None
Timeout Seconds: 10
Events: <none>
$ kubectl get MutatingWebhookConfiguration pod-identity-webhook -o yaml | yh
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"admissionregistration.k8s.io/v1","kind":"MutatingWebhookConfiguration","metadata":{"annotations":{},"name":"pod-identity-webhook"},"webhooks":[{"admissionReviewVersions":["v1beta1"],"clientConfig":{"caBundle":"LS0tLS1CR...","url":"https://127.0.0.1:23443/mutate"},"failurePolicy":"Ignore","name":"iam-for-pods.amazonaws.com","reinvocationPolicy":"IfNeeded","rules":[{"apiGroups":[""],"apiVersions":["v1"],"operations":["CREATE"],"resources":["pods"]}],"sideEffects":"None"}]}
creationTimestamp: "2023-05-29T05:22:30Z"
generation: 1
name: pod-identity-webhook
resourceVersion: "265"
uid: 278172af-cbc0-4275-ac74-750b950c6ebb
webhooks:
- admissionReviewVersions:
- v1beta1
clientConfig:
caBundle: LS0t...
url: https://127.0.0.1:23443/mutate
failurePolicy: Ignore
matchPolicy: Equivalent
name: iam-for-pods.amazonaws.com
namespaceSelector: {}
objectSelector: {}
reinvocationPolicy: IfNeeded
rules:
- apiGroups:
- ""
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
scope: '*'
sideEffects: None
timeoutSeconds: 10- 실행 중인Pod를 실행하고 이 token을 검사하면 이전 SA Token과 약간 다르게 보이는 것을 확인할 수 있습니다.
- 이 토큰의 대상이 이제 `sts.amazonaws.com`이고, 이 토큰을 생성하고 서명한 issuer가 여전히 OIDC 공급자이며, 마지막으로 토큰의 만료가 훨씬 더 오래되었음을 알 수 있습니다. 포드 정의 또는 서비스 계정 정의에서 ‘eks.amazonaws.com/token-expiration’ 주석을 사용하여 서비스 계정의 만료 기간을 수정할 수 있습니다.
- mutating webhook는 포드에 추가 토큰을 마운트하는 것 이상의 작업을 수행합니다. mutating webhook는 환경 변수도 주입합니다.
# AWS_WEB_IDENTITY_TOKEN_FILE 확인
$ IAM_TOKEN=$(kubectl exec -it eks-iam-test3 -- cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token)
$ echo $IAM_TOKEN
eyJhbGciOiJSUzI1NiIs....
# JWT 웹 확인
{
"aud": [
"sts.amazonaws.com"
],
"exp": 1685452118,
"iat": 1685365718,
"iss": "https://oidc.eks.ap-northeast-2.amazonaws.com/id/442308BCEF7A13DE72CFA354A0708CB7",
"kubernetes.io": {
"namespace": "default",
"pod": {
"name": "eks-iam-test3",
"uid": "71ed99e3-e602-4908-a71a-9d62ec3a1cbd"
},
"serviceaccount": {
"name": "my-sa",
"uid": "836979f7-c32a-4aa5-8dfc-e3a96b667cd1"
}
},
"nbf": 1685365718,
"sub": "system:serviceaccount:default:my-sa"
}
# env 변수 확인
$ kubectl get pod eks-iam-test3 -o json | jq -r '.spec.containers | .[].env'
[
{
"name": "AWS_STS_REGIONAL_ENDPOINTS",
"value": "regional"
},
{
"name": "AWS_DEFAULT_REGION",
"value": "ap-northeast-2"
},
{
"name": "AWS_REGION",
"value": "ap-northeast-2"
},
{
"name": "AWS_ROLE_ARN",
"value": "arn:aws:iam::90XXXXXX:role/eksctl-eks-hayley-addon-iamserviceaccount-de-Role1-1HXXXX"
},
{
"name": "AWS_WEB_IDENTITY_TOKEN_FILE",
"value": "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
}
]- 이제 워크로드에 IAM으로 authenticate(인증)을 시도하는 데 사용할 수 있는 token이 있으므로 다음 부분은 AWS IAM이 이러한 토큰을 신뢰하도록 하는 것입니다. AWS IAM은 [OIDC 자격 증명 공급자를 사용하는 연합 자격 증명](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html)을 지원합니다. 이 기능을 사용하면 IAM이 유효한 OIDC JWT를 수신한 후 지원되는 자격 증명 공급자로 AWS API 호출을 인증할 수 있습니다. 그런 다음 이 토큰을 AWS STS `AssumeRoleWithWebIdentity` API 작업에 전달하여 임시 IAM 자격 증명을 얻을 수 있습니다.
- Kubernetes 워크로드에 있는 OIDC JWT token은 암호화 서명되어 있으며 IAM은 AWS STS `AssumeRoleWithWebIdentity` API 작업이 임시 자격 증명을 보낼 수 있기 전에 이러한 토큰을 신뢰하고 검증해야 합니다. Kubernetes의 [서비스 계정 발급자 검색](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-issuer-discovery) 기능의 일부로 EKS는 public OpenID provider 구성 문서(Discovery 엔드포인트) 및 ‘https://OIDC_PROVIDER_URL/.well-known/openid-configuration'에서 토큰 서명(JSON 웹 키 세트 — JWKS)의 유효성을 검사하기 위한 공개 키를 호스팅합니다.
# Let’s take a look at this endpoint. We can use the aws eks describe-cluster command to get the OIDC Provider URL.
$ IDP=$(aws eks describe-cluster --name myeks --query cluster.identity.oidc.issuer --output text)
# Reach the Discovery Endpoint
$ curl -s $IDP/.well-known/openid-configuration | jq -r '.'
# In the above output, you can see the jwks (JSON Web Key set) field, which contains the set of keys containing the public keys used to verify JWT (JSON Web Token).
# Refer to the documentation to get details about the JWKS properties.
$ curl -s $IDP/keys | jq -r '.'# 실습 확인 후 파드 삭제
$ kubectl delete pod eks-iam-test3Amazon EKS Best Practices Guide for Security
참고 : https://aws.github.io/aws-eks-best-practices/security/docs/
공동 책임 모델 이해
- EKS에서 AWS는 EKS 관리형 Kubernetes 컨트롤 플레인 관리를 담당합니다. 여기에는 Kubernetes 컨트롤 플레인 노드, ETCD 데이터베이스 및 AWS가 안전하고 안정적인 서비스를 제공하는 데 필요한 기타 인프라가 포함됩니다. EKS의 소비자는 IAM, 포드 보안, 런타임 보안, 네트워크 보안 등과 같은 이 가이드의 주제에 대해 주로 책임이 있습니다
파드/컨테이너 보안 컨텍스트
- 지난 블로그 참고 : [k8s] Security — 파드/컨테이너 보안 컨텍스트
blog migration project
written in 2023.5.29
https://medium.com/techblog-hayleyshim/aws-eks-security-97a604d465fb
'IT > Infra&Cloud' 카테고리의 다른 글
| [aws] Control Tower (0) | 2023.10.30 |
|---|---|
| [aws] EKS Automation (0) | 2023.10.30 |
| [aws] EKS Autoscaling (0) | 2023.10.30 |
| [aws] EKS Observability (0) | 2023.10.29 |
| [aws] EKS Storage & Node (0) | 2023.10.29 |