brunch

You can make anything
by writing

C.S.Lewis

by Master Seo Oct 19. 2023

38탄-1. EKS DB - EKS 설치 1/3

 EKS 설치하고 EKS에서 DB를 운영해보자~



<1> EKS 생성 준비

<2> EKS 생성

<3> 워커노드에 접속 확인 하기

<4> EKS addon확인

<5> 프로그램들 설치

<6> 프로메테우스-스택 설치 = 모니터링



<1> EKS 설치 준비


1

도메인 1개를  route53에 설정

masterseo0.link



2

AWS console 로그인 > 서울 리전  

https://console.aws.amazon.com/console/home



EC2 > 키페어 생성  > 11-14-ec2

IAM >   11-05-access  > Access-key , Secret-key 생성  


Cloud Formation 으로 EKS  생성


ec2에서 키페어 생성

EC2 > 키페어 > 키페어 생성 > 10-19-ec2.ppk  (내 pc가 windows pc 경우 ppk,  mac / linux경우 pem)


iam에서 access-key /secret-key  만들기

= IAM >  eks-10-19  > admin 권한 > 보안 자격증명 > 액세스키 만들기 > CLI

> CSV 파일 다운로드 클릭 >  access-key /secret-key를 복사해 둔다.



3

AWS 콘솔 로그인.

새탭에서 보기~   





<2> EKS 생성


1

Cloudformation  링크 클릭해 설치- 1.27

cloud formation  파일



ec2-key 입력

access-key , secret-key 입력

여기서 worknode를 t3 미듐으로 변경 (이번장에서는 고사양일 필요 없음)

t3.medium 

배포


or

1.28 

myeks




2

20분 걸림

베스천과 노드 3대가 만들어짐.



타이머

https://vclock.kr/timer/#countdown=00:10:00&enabled=0&seconds=0&sound=xylophone&loop=1


혹, eks 생성 안되면?

해당 리전에 EC2 최대 갯수 제한 일 경우, Service Quotas (EC2) 증설 요청으로 해결 가능

https://somaz.tistory.com/171

https://ap-northeast-2.console.aws.amazon.com/servicequotas/home/services/ec2/quotas

Limit Type(EC2 Instances) ⇒ 서울 리전, All Standard (A, C, D, H, I, M, R, T, Z) Instances, New limit value(40 정도)



4

설치 결과

20분 후 콘솔에서 확인


public subnet에 EKS 생성

public subnet에 베스천 1대 생성



5

접속과 기본 확인 ?


브라우저~~~

새탭에서 보기



베스천 서버 로그인


k get nodes





다운로드

https://s3.ap-northeast-2.amazonaws.com/cloudformation.cloudneta.net/EKS/eks-oneclick.yaml



AWSTemplateFormatVersion: '2010-09-09'


Metadata:

  AWS::CloudFormation::Interface:

    ParameterGroups:

      - Label:

          default: "<<<<< Deploy EC2 >>>>>"

        Parameters:

          - KeyName

          - MyIamUserAccessKeyID

          - MyIamUserSecretAccessKey

          - SgIngressSshCidr

          - MyInstanceType

          - LatestAmiId

:

:

:

    Type: String

    Default: t3.medium

    AllowedValues: 

      - t2.micro

      - t2.small

      - t2.medium

      - t3.micro

      - t3.small

      - t3.medium

  LatestAmiId:

    Description: (DO NOT CHANGE)

    Type: 'AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>'

    Default: '/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2'

    AllowedValues:

      - /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2


  ClusterBaseName:

    Type: String

    Default: myeks

    AllowedPattern: "[a-zA-Z][-a-zA-Z0-9]*"

    Description: must be a valid Allowed Pattern '[a-zA-Z][-a-zA-Z0-9]*'

    ConstraintDescription: ClusterBaseName - must be a valid Allowed Pattern

  KubernetesVersion:

    Description: Enter Kubernetes Version, 1.24 ~ 1.28

    Type: String

    Default: 1.27

  WorkerNodeInstanceType:

    Description: Enter EC2 Instance Type. Default is t3.large.

    Type: String

    Default: t3.large

  WorkerNodeCount:

    Description: Worker Node Counts

    Type: String

    Default: 3

  WorkerNodeVolumesize:

    Description: Worker Node Volumes size

    Type: String

    Default: 30


:

:

:

:


      UserData:

        Fn::Base64:

          !Sub |

            #!/bin/bash

            hostnamectl --static set-hostname "${ClusterBaseName}-bastion-EC2"


            # Config convenience

            echo 'alias vi=vim' >> /etc/profile

            echo "sudo su -" >> /home/ec2-user/.bashrc


            # Change Timezone

            sed -i "s/UTC/Asia\/Seoul/g" /etc/sysconfig/clock

            ln -sf /usr/share/zoneinfo/Asia/Seoul /etc/localtime


            # Install Packages

            cd /root

            yum -y install tree jq git htop lynx amazon-efs-utils


            # Install kubectl & helm

            curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/1.27.5/2023-09-14/bin/linux/amd64/kubectl

            install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

            curl -s https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash


            # Install eksctl

            curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp

            mv /tmp/eksctl /usr/local/bin


            # Install aws cli v2

            curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

            unzip awscliv2.zip >/dev/null 2>&1

            sudo ./aws/install

            complete -C '/usr/local/bin/aws_completer' aws

            echo 'export AWS_PAGER=""' >>/etc/profile

            export AWS_DEFAULT_REGION=${AWS::Region}

            echo "export AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION" >> /etc/profile


            # Install YAML Highlighter

            wget https://github.com/andreazorzetto/yh/releases/download/v0.4.0/yh-linux-amd64.zip

            unzip yh-linux-amd64.zip

            mv yh /usr/local/bin/


            # Install krew

            curl -LO https://github.com/kubernetes-sigs/krew/releases/download/v0.4.4/krew-linux_amd64.tar.gz

            tar zxvf krew-linux_amd64.tar.gz

            ./krew-linux_amd64 install krew

            export PATH="$PATH:/root/.krew/bin"

            echo 'export PATH="$PATH:/root/.krew/bin"' >> /etc/profile


            # Install krew plugin

            kubectl krew install ctx ns get-all df-pv neat resource-capacity # ktop mtail tree

:

:

:

:


            # Create EKS Cluster & Nodegroup 

            eksctl create cluster --name $CLUSTER_NAME --region=$AWS_DEFAULT_REGION --nodegroup-name=ng1 --node-type=${WorkerNodeInstanceType} --nodes ${WorkerNodeCount} --node-volume-size=${WorkerNodeVolumesize} --vpc-public-subnets "$PubSubnet1","$PubSubnet2","$PubSubnet3" --version ${KubernetesVersion} --max-pods-per-node 50 --ssh-access --ssh-public-key /root/.ssh/id_rsa.pub --with-oidc --external-dns-access --full-ecr-access --dry-run > myeks.yaml

            sed -i 's/certManager: false/certManager: true/g' myeks.yaml

            sed -i 's/ebs: false/ebs: true/g' myeks.yaml

            sed -i 's/cloudWatch: false/cloudWatch: true/g' myeks.yaml

            sed -i 's/xRay: false/xRay: true/g' myeks.yaml

            cat <<EOT >> myeks.yaml

            addons:

            - name: vpc-cni

              version: latest

              attachPolicyARNs:

                - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy

            - name: kube-proxy

              version: latest

            - name: coredns

              version: latest

            - name: aws-ebs-csi-driver

              wellKnownPolicies:

                ebsCSIController: true

            - name: aws-efs-csi-driver

              wellKnownPolicies:

                efsCSIController: true

            EOT

            cat <<EOT > irsa.yaml

              serviceAccounts:

              - metadata:

                  name: aws-load-balancer-controller

                  namespace: kube-system

                wellKnownPolicies:

                  awsLoadBalancerController: true

              - metadata:

                  name: ebs-csi-controller-sa

                  namespace: kube-system

                wellKnownPolicies:

                  efsCSIController: true

              - metadata:

                  name: efs-csi-controller-sa

                  namespace: kube-system

                wellKnownPolicies:

                  efsCSIController: true

            EOT

            sed -i -n -e '/withOIDC/r irsa.yaml' -e '1,$p' myeks.yaml

            cat <<EOT > precmd.yaml

              preBootstrapCommands:

                - "yum install htop links tree jq tcpdump sysstat -y"

            EOT

            sed -i -n -e '/instanceType/r precmd.yaml' -e '1,$p' myeks.yaml

            nohup eksctl create cluster -f myeks.yaml --verbose 4 --kubeconfig "/root/.kube/config" 1> /root/create-eks.log 2>&1 &


            echo 'cloudinit End!'


Outputs:

  eksctlhost:

    Value: !GetAtt EKSEC2.PublicIp



---------------------------------------------------------------



# SSH 접속

ssh -i ~/.ssh/xxxxxxxxx.pem ec2-user@$(aws cloudformation describe-stacks --stack-name myeks --query 'Stacks[*].Outputs[0].OutputValue' --output text)


# cloud-init 실행 과정 로그 확인

tail -f /var/log/cloud-init-output.log


# cloud-init 정상 완료 후 eksctl 실행 과정 로그 확인

tail -f /root/create-eks.log


-----------------------------------------------------------------


# 설치 확인

kubectl cluster-info

eksctl get cluster

eksctl get nodegroup --cluster $CLUSTER_NAME



# 환경변수 정보 확인

export | egrep 'ACCOUNT|AWS_|CLUSTER|KUBERNETES|VPC|Subnet'

export | egrep 'ACCOUNT|AWS_|CLUSTER|KUBERNETES|VPC|Subnet' | egrep -v 'SECRET|KEY'



# 인증 정보 확인

cat /root/.kube/config | yh

kubectl config view | yh

kubectl ctx


# 노드 정보 확인

kubectl get node --label-columns=node.kubernetes.io/instance-type,eks.amazonaws.com/capacityType,topology.kubernetes.io/zone






eksctl get iamidentitymapping --cluster myeks


# krew 플러그인 확인

kubectl krew list


PLUGIN             VERSION

ctx                v0.9.5

df-pv              v0.3.0

get-all            v1.3.8

krew               v0.4.4

neat               v2.0.3

ns                 v0.9.5

resource-capacity  v0.7.4



kubectl resource-capacity




# default 네임스페이스 적용

kubectl ns default



# 모든 네임스페이스에서 모든 리소스 확인

kubectl get-all






<3> 워커노드에 접속 확인 하기


필요에 따라 워커노드에 들어갈 때 사용한다.



# 노드 IP 확인 및 PrivateIP 변수 지정


aws ec2 describe-instances --query "Reservations[*].Instances[*].{PublicIPAdd:PublicIpAddress,PrivateIPAdd:PrivateIpAddress,InstanceName:Tags[?Key=='Name']|[0].Value,Status:State.Name}" --filters Name=instance-state-name,Values=running --output table





N1=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2a -o jsonpath={.items[0].status.addresses[0].address})


N2=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2b -o jsonpath={.items[0].status.addresses[0].address})


N3=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2c -o jsonpath={.items[0].status.addresses[0].address})


echo "export N1=$N1" >> /etc/profile

echo "export N2=$N2" >> /etc/profile

echo "export N3=$N3" >> /etc/profile

echo $N1, $N2, $N3



# 보안그룹 ID와 보안그룹 이름(Name아님을 주의!) 확인

aws ec2 describe-security-groups --query 'SecurityGroups[*].[GroupId, GroupName]' --output text



# 노드 보안그룹 ID 확인

aws ec2 describe-security-groups --filters Name=group-name,Values=*ng1* --query "SecurityGroups[*].[GroupId]" --output text


NGSGID=$(aws ec2 describe-security-groups --filters Name=group-name,Values=*ng1* --query "SecurityGroups[*].[GroupId]" --output text)

echo $NGSGID



# 노드 보안그룹에 eksctl-host 에서 노드(파드)에 접속 가능하게 룰(Rule) 추가 설정

aws ec2 authorize-security-group-ingress --group-id $NGSGID --protocol '-1' --cidr 192.168.1.100/32



# eksctl-host 에서 노드의IP나 coredns 파드IP로 ping 테스트

ping -c 2 $N1

ping -c 2 $N2

ping -c 2 $N3



# 워커 노드 SSH 접속 : '-i ~/.ssh/id_rsa' 생략 가능

for node in $N1 $N2 $N3; do ssh ec2-user@$node hostname; done

ssh ec2-user@$N1

exit

ssh ec2-user@$N2

exit

ssh ec2-user@$N3

exit





<4> EKS addon확인


1

최신 버전 - kube-proxy, coredns, aws vpc cni aws ebs csi drive  확인

https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html

https://docs.aws.amazon.com/eks/latest/userguide/managing-add-ons.html



2

코솔 

EKS가서 추가기능 (addon)  확인하자.


3

# eksctl 설치/업데이트 추가기능 addon 확인


eksctl get addon --cluster $CLUSTER_NAME


ebs csi 드라이버

efs csi 드라이브

coredns

kube-proxy

vpc-cni








# (참고) eks 설치 yaml 중 addon 내용

tail -n 12 myeks.yaml

addons:

- name: vpc-cni

  version: latest

  attachPolicyARNs:

    - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy

- name: kube-proxy

  version: latest

- name: coredns

  version: latest

- name: aws-ebs-csi-driver

  wellKnownPolicies:

    ebsCSIController: true





<5> 프로그램들 설치 하자


1

AWS Load Balancer Controller 설치,

ExternalDNS 설치,

kube-ops-view 설치, 

스토리지 클래스 생성, 

AWS Cert Manager 인증서 확인/



2

AWS Load Balancer Controller 설치


https://docs.aws.amazon.com/ko_kr/eks/latest/userguide/aws-load-balancer-controller.html



kubectl get crd




# 설치

helm repo add eks https://aws.github.io/eks-charts


helm repo update


helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=$CLUSTER_NAME  --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller




# 설치 확인

kubectl get crd


NAME                                         CREATED AT

cninodes.vpcresources.k8s.aws                2023-11-05T07:27:30Z

eniconfigs.crd.k8s.amazonaws.com             2023-11-05T07:27:26Z

ingressclassparams.elbv2.k8s.aws             2023-11-05T08:30:52Z

policyendpoints.networking.k8s.aws           2023-11-05T07:27:31Z

securitygrouppolicies.vpcresources.k8s.aws   2023-11-05T07:27:30Z

targetgroupbindings.elbv2.k8s.aws            2023-11-05T08:30:52Z


2개가 설치됨.



kubectl get deployment -n kube-system aws-load-balancer-controller

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE

aws-load-balancer-controller   2/2     2            2           61s



kubectl describe deploy -n kube-system aws-load-balancer-controller




3

ExternalDNS 설치


https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/aws.md


MyDomain=<자신의 도메인>

echo "export MyDomain=<자신의 도메인>" >> /etc/profile


MyDomain=masterseo0.link

echo "export MyDomain=masterseo0.link" >> /etc/profile


MyDnsHostedZoneId=$(aws route53 list-hosted-zones-by-name --dns-name "${MyDomain}." --query "HostedZones[0].Id" --output text)


echo $MyDomain, $MyDnsHostedZoneId

masterseo0.link, /hostedzone/Z00791123ED8WVU46SO8F




# ExternalDNS 컨트롤러 설치


curl -s -O https://raw.githubusercontent.com/cloudneta/cnaeblab/master/_data/externaldns.yaml


MyDomain=$MyDomain MyDnsHostedZoneId=$MyDnsHostedZoneId envsubst < externaldns.yaml | kubectl apply -f -


serviceaccount/external-dns created

clusterrole.rbac.authorization.k8s.io/external-dns created

clusterrolebinding.rbac.authorization.k8s.io/external-dns-viewer created

deployment.apps/external-dns created



(eks-10-19@myeks:default) [root@myeks-bastion-EC2 ~]# k get pod -A

NAMESPACE     NAME                                            READY   STATUS    RESTARTS   AGE

kube-system   aws-load-balancer-controller-7b657486db-964fn   1/1     Running   0          10m

kube-system   aws-load-balancer-controller-7b657486db-b8wmk   1/1     Running   0          10m

kube-system   aws-node-72xkz                                  2/2     Running   0          83m

kube-system   aws-node-rlvj8                                  2/2     Running   0          83m

kube-system   aws-node-xm5ph                                  2/2     Running   0          83m

kube-system   coredns-754fbc56c6-9sk27                        1/1     Running   0          80m

kube-system   coredns-754fbc56c6-m8ppb                        1/1     Running   0          80m

kube-system   ebs-csi-controller-cc4576b65-6pl24              6/6     Running   0          78m

kube-system   ebs-csi-controller-cc4576b65-cpqn5              6/6     Running   0          78m

kube-system   ebs-csi-node-4f276                              3/3     Running   0          78m

kube-system   ebs-csi-node-88wb8                              3/3     Running   0          78m

kube-system   ebs-csi-node-zmcdz                              3/3     Running   0          78m

kube-system   external-dns-5df5bb58b5-lj6ns                   1/1     Running   0          117s



4

kube-ops-view 설치


helm repo add geek-cookbook https://geek-cookbook.github.io/charts/


helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 --set env.TZ="Asia/Seoul" --namespace kube-system


kubectl patch svc -n kube-system kube-ops-view -p '{"spec":{"type":"LoadBalancer"}}'


kubectl annotate service kube-ops-view -n kube-system "external-dns.alpha.kubernetes.io/hostname=kubeopsview.$MyDomain"


echo -e "Kube Ops View URL = http://kubeopsview.$MyDomain:8080/#scale=1.5"


( 5분 걸림)



http://kubeopsview.masterseo0.link:8080



 k get svc -A



web브라우저로 접속 확인




# 확인 및 로그 모니터링

kubectl get pod -l app.kubernetes.io/name=external-dns -n kube-system

NAME                            READY   STATUS    RESTARTS   AGE

external-dns-5df5bb58b5-8n5d7   1/1     Running   0          2m16s


// external dns pod가 있다.



kubectl logs deploy/external-dns -n kube-system -f


time="2023-11-05T08:50:34Z" level=info msg="Applying provider record filter for domains: [masterseo0.link. .masterseo0.link.]"

time="2023-11-05T08:50:34Z" level=info msg="All records are already up to date"

time="2023-11-05T08:51:35Z" level=info msg="Applying provider record filter for domains: [masterseo0.link. .masterseo0.link.]"

time="2023-11-05T08:51:35Z" level=info msg="All records are already up to date"

time="2023-11-05T08:52:35Z" level=info msg="Applying provider record filter for domains: [masterseo0.link. .masterseo0.link.]"

time="2023-11-05T08:52:35Z" level=info msg="All records are already up to date"

time="2023-11-05T08:53:36Z" level=info msg="Applying provider record filter for domains: [masterseo0.link. .masterseo0.link.]"

time="2023-11-05T08:53:36Z" level=info msg="All records are already up to date"




5

스토리지 클래스 생성


kubectl get sc


# gp2 기본 스토리지 클래스 default 설정 제거

kubectl patch sc gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'


# gp3 스토리지 클래스 생성

kubectl apply -f https://raw.githubusercontent.com/gasida/DOIK/main/1/gp3-sc.yaml


kind: StorageClass

apiVersion: storage.k8s.io/v1

metadata:

  annotations:

    storageclass.kubernetes.io/is-default-class: "true"

  name: gp3

allowVolumeExpansion: true

provisioner: ebs.csi.aws.com

volumeBindingMode: WaitForFirstConsumer

parameters:

  type: gp3

  allowAutoIOPSPerGBIncrease: 'true'

  encrypted: 'true'



# 스토리지 클래스 확인

kubectl get sc


gp3가 디폴트로 되었다.




6

AWS Cert Manager에서 도메인으로  인증 발급 


https://docs.aws.amazon.com/acm/latest/userguide/dns-validation.html



 route53 에서 *.도메인을 cname으로  설정해줘야한다.


# ACM 인증서 확인

aws acm list-certificates

           "Status": "ISSUED",


# ACM 인증서 변수 선언

CERT_ARN=`aws acm list-certificates --query 'CertificateSummaryList[].CertificateArn[]' --output text`; echo $CERT_ARN





<6> 프로메테우스-스택 설치 = 모니터링


https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack


# 네임스페이스 생성

kubectl create ns monitoring


# 사용 리전의 인증서 ARN 확인

CERT_ARN=`aws acm list-certificates --query 'CertificateSummaryList[].CertificateArn[]' --output text`


echo $CERT_ARN



# repo 추가

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts



# 파라미터 파일 생성

cat <<EOT > monitor-values.yaml

prometheus:

  prometheusSpec:

    podMonitorSelectorNilUsesHelmValues: false

    serviceMonitorSelectorNilUsesHelmValues: false

    retention: 5d

    retentionSize: "10GiB"

    scrapeInterval: '15s'

    evaluationInterval: '15s'

  ingress:

    enabled: true

    ingressClassName: alb

    hosts:

      - prometheus.$MyDomain

    paths:

      - /*

    annotations:

      alb.ingress.kubernetes.io/scheme: internet-facing

      alb.ingress.kubernetes.io/target-type: ip

      alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'

      alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN

      alb.ingress.kubernetes.io/success-codes: 200-399

      alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb

      alb.ingress.kubernetes.io/group.name: study

      alb.ingress.kubernetes.io/ssl-redirect: '443'

grafana:

  defaultDashboardsTimezone: Asia/Seoul

  adminPassword: prom-operator

  ingress:

    enabled: true

    ingressClassName: alb

    hosts:

      - grafana.$MyDomain

    paths:

      - /*

    annotations:

      alb.ingress.kubernetes.io/scheme: internet-facing

      alb.ingress.kubernetes.io/target-type: ip

      alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'

      alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN

      alb.ingress.kubernetes.io/success-codes: 200-399

      alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb

      alb.ingress.kubernetes.io/group.name: study

      alb.ingress.kubernetes.io/ssl-redirect: '443'

defaultRules:

  create: false

kubeEtcd:

  enabled: false

alertmanager:

  enabled: false

EOT



# 설명

exernal dns를 통해 도메인 들어감

https 로 접근 가능

80은 443으로 리다이렉션


그라파나

admin / pass



# 배포

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 51.7.0 -f monitor-values.yaml --namespace monitoring



Pod 7개 추가




# 확인

helm list -n monitoring

kubectl get pod,svc,ingress -n monitoring

kubectl get-all -n monitoring

kubectl get prometheus,servicemonitors -n monitoring

kubectl get crd | grep monitoring


# operator : 시스템 경고 메시지 정책(prometheus rule), 애플리케이션 모니터링 대상 추가 등의 작업을 편리하게 할수 있게 CRD 지원


# prometheus-0 : 모니터링 대상이 되는 파드는 ‘exporter’라는 별도의 사이드카 형식의 파드에서 모니터링 메트릭을 노출, pull 방식으로 가져와 내부의 시계열 데이터베이스에 저장


# node-exporter : 노드익스포터는 물리 노드에 대한 자원 사용량(네트워크, 스토리지 등 전체) 정보를 메트릭 형태로 변경하여 노출


# kube-state-metrics : 쿠버네티스의 클러스터의 상태(kube-state)를 메트릭으로 변환하는 파드


# grafana : 프로메테우스는 메트릭 정보를 저장하는 용도로 사용하며, 그라파나로 시각화 처리





3

확인?


# 프로메테우스 ingress 확인

kubectl get ingress -n monitoring kube-prometheus-stack-prometheus

kubectl describe ingress -n monitoring kube-prometheus-stack-prometheus


# 프로메테우스 ingress 도메인으로 웹 접속

echo -e "Prometheus Web URL = https://prometheus.$MyDomain"


# 그라파나 ingress 확인

kubectl get ingress -n monitoring kube-prometheus-stack-grafana

kubectl describe ingress -n monitoring kube-prometheus-stack-grafana


# 그라파나 ingress 도메인으로 웹 접속 : 기본 계정 - admin / prom-operator

echo -e "Grafana Web URL = https://grafana.$MyDomain"



(10분 걸린다. 기다려라~)




4

그라파나 대시보드 사용 = 기본 대시보드와 공식 대시보드 가져오기


스택을 통해서 설치된 기본 대시보드 확인 : Dashboards → Browse


공식 대시보드

https://grafana.com/grafana/dashboards/?pg=docs-grafana-latest-dashboards


추천 대시보드

https://grafana.com/orgs/imrtfm/dashboards



Dashboard → New → Import → **15757** 입력 후 Load ⇒ 

데이터소스(Prometheus 선택) 후 **Import** 클릭





Kubernetes All-in-one Cluster Monitoring KR****] 


Dashboard → New → Import → **13770** or **17900** 입력 후 Load ⇒ 데이터소스(Prometheus 선택) 후 **Import** 클릭




Node Exporter Full****] 


Dashboard → New → Import → **1860** 입력 후 Load ⇒ 데이터소스(Prometheus 선택) 후 **Import** 클릭


Node Exporter for Prometheus Dashboard based on 11074] 15172**



- kube-state-metrics-v2 가져와보자 : **Dashboard ID copied!** (13332) 클릭 - [링크](https://grafana.com/grafana/dashboards/13332-kube-state-metrics-v2/)

https://grafana.com/grafana/dashboards/13332-kube-state-metrics-v2/



[**kube-state-metrics-v2**] Dashboard → New → Import → **13332** 입력 후 Load ⇒ 데이터소스(Prometheus 선택) 후 **Import** 클릭

https://grafana.com/grafana/dashboards/16032-aws-cni-metrics/




# PodMonitor 배포

cat <<EOF | kubectl create -f -

apiVersion: monitoring.coreos.com/v1

kind: PodMonitor

metadata:

  name: aws-cni-metrics

  namespace: kube-system

spec:

  jobLabel: k8s-app

  namespaceSelector:

    matchNames:

    - kube-system

  podMetricsEndpoints:

  - interval: 30s

    path: /metrics

    port: metrics

  selector:

    matchLabels:

      k8s-app: aws-node

EOF



# PodMonitor 확인

kubectl get podmonitor -n kube-system

NAME              AGE

aws-cni-metrics   40s




삭제

eksctl delete cluster --name $CLUSTER_NAME && aws cloudformation delete-stack --stack-name $CLUSTER_NAME



다음은

https://brunch.co.kr/@topasvga/3486



주말 CloudNet  스터디 내용 참고하여  정리한 부분입니다.

https://gasidaseo.notion.site/gasidaseo/CloudNet-Blog-c9dfa44a27ff431dafdd2edacc8a1863  




감사합니다.

                    

매거진의 이전글 모두의 연구소.무료. 30명 공간은 로비임,EKS스터디
브런치는 최신 브라우저에 최적화 되어있습니다. IE chrome safari