출처1 : https://incredible.ai/kubernetes/2020/09/08/Prometheus_And_Grafana/
출처2 : https://yongho1037.tistory.com/812
출처3 : https://docs.aws.amazon.com/ko_kr/eks/latest/userguide/prometheus.html
출처4 : https://medium.com/@shikharsrivastava_14544/amazon-elastic-kubernetes-service-eks-setup-66a4ab49e3a2
출처5 : https://linux.systemv.pe.kr/kubernetes-monitoring-prometheus-%EC%84%A4%EC%B9%98/
출처6 : https://blog.naver.com/PostView.naver?blogId=isc0304&logNo=222515904650&redirect=Dlog&widgetTypeCall=true&directAccess=false
그라파나 대시보드는 11074 번이 괜찮은 것 같다.

https://grafana.com/grafana/dashboards/11074
EKS v1.21 에서 적용

#### Prometheus & Grafana 설치
# https://incredible.ai/kubernetes/2020/09/08/Prometheus_And_Grafana/

## metrics-server 설치
# helm 설치
# k8s에서 metrics를 수집 및 전달 담당
# 자세한 설치방법은 아래 링크에서
# https://github.com/kubernetes-sigs/metrics-server
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

## storage 설정
# https://www.jeffgeerling.com/blog/2019/expanding-k8s-pvs-eks-on-aws

cat <<EOF > prometheus-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: prometheus
  namespace: prometheus
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  fsType: ext4
reclaimPolicy: Retain
mountOptions:
  - debug
EOF

# storage 적용
kubectl apply -f prometheus-storageclass.yaml

## Prometheus 배포
# helm 저장소 등록
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# 설치 리스트 확인
helm search repo prometheus-community

# 배포
kubectl create namespace prometheus
helm install prometheus prometheus-community/prometheus \
  --namespace prometheus \
  --set alertmanager.persistentVolume.storageClass="gp2" \
  --set alertmanager.persistentVolume.size="128Gi" \
  --set server.persistentVolume.storageClass="gp2" \
  --set server.persistentVolume.size="128Gi"

#############################################################################################################################################
# output start
#############################################################################################################################################
NAME: prometheus
LAST DEPLOYED: Tue Oct 19 01:14:15 2021
NAMESPACE: prometheus
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.prometheus.svc.cluster.local


Get the Prometheus server URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace prometheus port-forward $POD_NAME 9090


The Prometheus alertmanager can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-alertmanager.prometheus.svc.cluster.local


Get the Alertmanager URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace prometheus port-forward $POD_NAME 9093
#################################################################################
######   WARNING: Pod Security Policy has been moved to a global property.  #####
######            use .Values.podSecurityPolicy.enabled with pod-based      #####
######            annotations                                               #####
######            (e.g. .Values.nodeExporter.podSecurityPolicy.annotations) #####
#################################################################################


The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
prometheus-pushgateway.prometheus.svc.cluster.local


Get the PushGateway URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace prometheus port-forward $POD_NAME 9091

For more information on running Prometheus, visit:
https://prometheus.io/
#############################################################################################################################################
# output end
#############################################################################################################################################


# 배포 확인
kubectl get pods -n prometheus

# PersistentVolume 확인
kubectl get pv

# 혹시나 삭제하고 싶을 땐
helm uninstall prometheus -n prometheus


## Connect Prometheus & AlertManager Server
# Prometheus 접속
POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace prometheus port-forward $POD_NAME 9090

# Prometheus 접속하는 다른 방법
kubectl port-forward deploy/prometheus-server 9090 -n prometheus

# AlertManager 접속
export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace prometheus port-forward $POD_NAME 9093



## Grafana 배포
# Grafana 설정파일 작성
cat <<EOF > grafana.yaml
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.prometheus.svc.cluster.local
      access: proxy
      isDefault: true
EOF

# Grafana 배포
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
kubectl create namespace grafana
helm install grafana grafana/grafana \
  --namespace grafana \
  --set persistence.storageClassName="gp2" \
  --set persistence.size="128Gi" \
  --set persistence.enabled=true \
  --values grafana.yaml \
  --set service.type=LoadBalancer

#############################################################################################################################################
# output start
#############################################################################################################################################
W1019 06:26:15.143150   11240 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1019 06:26:15.159925   11240 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1019 06:26:15.340594   11240 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1019 06:26:15.341456   11240 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME: grafana
LAST DEPLOYED: Tue Oct 19 06:26:14 2021
NAMESPACE: grafana
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:

   kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

   grafana.grafana.svc.cluster.local

   Get the Grafana URL to visit by running these commands in the same shell:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
        You can watch the status of by running 'kubectl get svc --namespace grafana -w grafana'
     export SERVICE_IP=$(kubectl get svc --namespace grafana grafana -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
     http://$SERVICE_IP:80

3. Login with the password from step 1 and the username: admin
#############################################################################################################################################
# output end
#############################################################################################################################################


# 암호얻기
kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo


# 접속 주소
kubectl get svc --namespace grafana grafana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' ; echo
[AWS] Prometheus & Grafana on EKS
