brunch

You can make anything
by writing

C.S.Lewis

by Master Seo Nov 21. 2023

38탄-18. EKS DB -Getting Start

Stackable Documentation

Getting Started Stackable Documentation

https://docs.stackable.tech/home/stable/getting_started


Overview : Stackable is based on Kubernetes and uses this as the control plane to manage clusters

In this guide we will build a simple cluster with 3 services; Apache ZooKeeper, Apache Kafka and Apache NiFi.



<1> Installing StackableOperators - Link

<2> 클러스터 삭제




<1> Installing StackableOperators - Link



# [터미널1] 모니터링

watch -d "kubectl get pod -n stackable-operators"



# [터미널2] 설치

stackablectl release list



#  -i 옵션으로 원하는 오퍼레이터를 설치할수 있다.

# 이번은 데모로 설치하지 않고 오퍼레이터를 하나씩 설치하는 법이다.


stackablectl release install -i commons -i secret -i zookeeper -i kafka -i nifi 23.7

[INFO ] Installing release 23.7

[INFO ] Installing commons operator in version 23.7.0

[INFO ] Installing kafka operator in version 23.7.0

[INFO ] Installing nifi operator in version 23.7.0

[INFO ] Installing secret operator in version 23.7.0

[INFO ] Installing zookeeper operator in version 23.7.0



# 설치 확인

helm list -n stackable-operators

stackablectl operator installed

kubectl get crd | grep stackable.tech

kubectl get pod




오퍼레이터 설치가 되었다.   주키퍼 오퍼레이터도 설치되었다!!!


이제 주키퍼를 사용하자.





<2> Deploying Stackable Services - Link


1

watch -d kubectl get pod,job,svc,pvc




2

Apache ZooKeeper


kubectl apply -f - <<EOF

---

apiVersion: zookeeper.stackable.tech/v1alpha1

kind: ZookeeperCluster

metadata:

  name: simple-zk

spec:

  image:

    productVersion: "3.8.1"

    stackableVersion: "23.7"

  clusterConfig:

    tls:

      serverSecretClass: null

  servers:

    roleGroups:

      primary:

        replicas: 1

        config:

          myidOffset: 10

---

apiVersion: zookeeper.stackable.tech/v1alpha1

kind: ZookeeperZnode

metadata:

  name: simple-zk-znode

spec:

  clusterRef:

    name: simple-zk

EOF



3

# 설치 확인

kubectl get zookeepercluster,zookeeperznode

kubectl get pod,svc,ep,pvc -l app.kubernetes.io/instance=simple-zk

kubectl describe pod -l app.kubernetes.io/instance=simple-zk



# 실시간 로그 확인

kubectl logs -l app.kubernetes.io/instance=simple-zk -c zookeeper -f




4

Apache Kafka : We will deploy an Apache Kafka broker that depends on the ZooKeeper service we just deployed.


The zookeeperReference property below points to the namespace and name we gave to the ZooKeeper service deployed previously.



kubectl apply -f - <<EOF

---

apiVersion: kafka.stackable.tech/v1alpha1

kind: KafkaCluster

metadata:

  name: simple-kafka

spec:

  image:

    productVersion: "3.4.0"

    stackableVersion: "23.7"

  clusterConfig:

    zookeeperConfigMapName: simple-kafka-znode

    tls:

      serverSecretClass: null

  brokers:

    roleGroups:

      brokers:

        replicas: 3

---

apiVersion: zookeeper.stackable.tech/v1alpha1

kind: ZookeeperZnode

metadata:

  name: simple-kafka-znode

spec:

  clusterRef:

    name: simple-zk

    namespace: default

EOF



5

# 설치 확인

kubectl get kafkacluster,zookeeperznode

kubectl get pod,svc,ep,pvc -l app.kubernetes.io/instance=simple-kafka

kubectl describe pod -l app.kubernetes.io/instance=simple-kafka



# 실시간 로그 확인

kubectl logs -l app.kubernetes.io/instance=simple-kafka -c kafka -f




6

Kafka UI - 링크


#

helm repo add kafka-ui https://provectus.github.io/kafka-ui-charts

cat <<EOF > kafkaui-values.yml

yamlApplicationConfig:

  kafka:

    clusters:

      - name: yaml

        bootstrapServers: simple-kafka-broker-brokers:9092

  auth:

    type: disabled

  management:

    health:

      ldap:

        enabled: false

EOF



# 설치

helm install kafka-ui kafka-ui/kafka-ui -f kafkaui-values.yml



# 접속 확인

kubectl patch svc kafka-ui -p '{"spec":{"type":"LoadBalancer"}}'

kubectl annotate service kafka-ui "external-dns.alpha.kubernetes.io/hostname=kafka-ui.$MyDomain"

echo -e "kafka-ui Web URL = http://kafka-ui.$MyDomain"



7

Apache NiFi - 링크소개


https://nifi.apache.org/docs.html




kubectl apply -f - <<EOF

---

apiVersion: zookeeper.stackable.tech/v1alpha1

kind: ZookeeperZnode

metadata:

  name: simple-nifi-znode

spec:

  clusterRef:

    name: simple-zk

---

apiVersion: v1

kind: Secret

metadata:

  name: nifi-admin-credentials-simple

stringData:

  username: admin

  password: AdminPassword

---

apiVersion: nifi.stackable.tech/v1alpha1

kind: NifiCluster

metadata:

  name: simple-nifi

spec:

  image:

    productVersion: "1.21.0"

    stackableVersion: "23.7"

  clusterConfig:

    listenerClass: external-unstable

    zookeeperConfigMapName: simple-nifi-znode

    authentication:

      method:

        singleUser:

          adminCredentialsSecret: nifi-admin-credentials-simple

    sensitiveProperties:

      keySecret: nifi-sensitive-property-key

      autoGenerate: true

  nodes:

    roleGroups:

      default:

        replicas: 1

EOF




8

# 설치 확인 : job 완료까지 다소 시간 소요됨

kubectl get nificluster,zookeeperznode

kubectl get pod,svc,ep,pvc,job -l app.kubernetes.io/instance=simple-nifi

kubectl describe job.batch/simple-nifi-create-reporting-task-1-21-0

kubectl describe pod -l app.kubernetes.io/instance=simple-nifi

...

Args:

      /stackable/python/create_nifi_reporting_task.py -n https://simple-nifi.default.svc.cluster.local:8443/nifi-api -u "$(cat /stackable/adminuser/username | grep -oP '((cn|dn|uid)=\K[^,]+|.*)' | head -n 1)" -p "$(cat /stackable/adminuser/password)" -v 1.21.0 -m 8081 -c /stackable/cert/ca.crt

...



# 실시간 로그 확인

kubectl logs -l app.kubernetes.io/instance=simple-nifi -c nifi -f





9

Testing your cluster - Link


설치 확인

stackablectl stacklet list





10

Apache ZooKeeper


# ZooKeeper CLI shell

kubectl exec -i -t simple-zk-server-primary-0 -c zookeeper -- bin/zkCli.sh

------------------

# znodes 확인

# You can run the ls / command to see the list of znodes in the root path, 

# which should include those created by Apache Kafka and Apache NiFi.

ls /

[znode-5fef78a9-71e1-4250-bc35-ced60243d60f, znode-b0bf14f8-a1f6-4b31-aaba-4f4bbc68767d, znode-c45d9efd-a071-4723-943a-79d5fe49a162, zookeeper]

quit

------------------




11

Apache Kafka


# 토픽 생성

kubectl exec -it simple-kafka-broker-brokers-0 -c kafka -- bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic demo

...

Created topic demo.

...

# 토픽 확인

kubectl exec -it simple-kafka-broker-brokers-0 -c kafka -- bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

...

demo

...



12


Apache NiFi : PRODUCT nifi ENDPOINTS https 접속 : admin / AdminPassword


# NiFi admin 계정의 암호 확인

kubectl get secrets nifi-admin-credentials-simple -o jsonpath="{.data.password}" | base64 -d && echo

AdminPassword




13


# Apache NiFi 삭제

kubectl delete nificluster simple-nifi && kubectl delete zookeeperznode simple-nifi-znode



# kafka-ui 삭제

helm uninstall kafka-ui



# Apache kafka 삭제

kubectl delete kafkacluster simple-kafka && kubectl delete zookeeperznode simple-kafka-znode



# Apache ZooKeeper 삭제

kubectl delete zookeepercluster simple-zk && kubectl delete zookeeperznode simple-zk-znode



# secret, pvc 삭제

kubectl delete secret nifi-admin-credentials-simple nifi-sensitive-property-key secret-provisioner-tls-ca

kubectl delete pvc --all



# operator 삭제

stackablectl operator uninstall nifi kafka zookeeper secret commons



# 남은 리소스 확인

kubectl get-all -n stackable-operators




<2> 클러스터 삭제


eksctl delete cluster --name $CLUSTER_NAME && aws cloudformation delete-stack --stack-name $CLUSTER_NAME




EKS DB  몰아 보기

https://brunch.co.kr/@topasvga/3511



EKS 전체 보기

https://brunch.co.kr/@topasvga/1679



keyword
매거진의 이전글 38탄-17. EKS DB -Demo:TrinoTaxi
브런치는 최신 브라우저에 최적화 되어있습니다. IE chrome safari