<a href="https://github.com/wazuh/wazuh-puppet/issues/222">https://github.com/wazuh/wazuh-puppet/issues/222</a><a href="https://discuss.elastic.co/t/how-to-change-number-of-shards-and-number-of-replica-in-elasticseatch-template/167855/4">https://discuss.elastic.co/t/how-to-change-number-of-shards-and-number-of-replica-in-elasticseatch-template/167855/4</a>&nbsp;&nbsp;<h2>오류</h2>최대치의 shards가 열려있어서 추가로 열 수 없다는 오류가 나왔다.<pre style="background-color:#2b2b2b;color:#a9b7c6;font-family:'D2Coding',monospace;font-size:11.3pt;">org.springframework.data.elasticsearch.UncategorizedElasticsearchException: Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [995]/[1000] maximum shards open;]; nested exception is ElasticsearchStatusException[Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [995]/[1000] maximum shards open;]] at org.springframework.data.elasticsearch.core.ElasticsearchExceptionTranslator.translateExceptionIfPossible(ElasticsearchExceptionTranslator.java:66) ~[spring-data-elasticsearch-4.1.3.jar:4.1.3] at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.translateException(ElasticsearchRestTemplate.java:362) ~[spring-data-elasticsearch-4.1.3.jar:4.1.3] at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.execute(ElasticsearchRestTemplate.java:345) ~[spring-data-elasticsearch-4.1.3.jar:4.1.3] at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.doIndex(ElasticsearchRestTemplate.java:143) ~[spring-data-elasticsearch-4.1.3.jar:4.1.3] at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.index(AbstractElasticsearchTemplate.java:201) ~[spring-data-elasticsearch-4.1.3.jar:4.1.3] at com.fm.webinar.service.QuizService.saveTemplate(QuizService.java:132) ~[classes/:na] at com.fm.webinar.service.QuizService.sendTemplate(QuizService.java:117) ~[classes/:na] at com.fm.webinar.controller.message.QuizController.chatMessage(QuizController.java:26) ~[classes/:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na] at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na] at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:169) ~[spring-messaging-5.3.15.jar:5.3.15] at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:119) ~[spring-messaging-5.3.15.jar:5.3.15] at org.springframework.messaging.handler.invocation.AbstractMethodMessageHandler.handleMatch(AbstractMethodMessageHandler.java:569) ~[spring-messaging-5.3.15.jar:5.3.15] at org.springframework.messaging.simp.annotation.support.SimpAnnotationMethodMessageHandler.handleMatch(SimpAnnotationMethodMessageHandler.java:511) ~[spring-messaging-5.3.15.jar:5.3.15] at org.springframework.messaging.simp.annotation.support.SimpAnnotationMethodMessageHandler.handleMatch(SimpAnnotationMethodMessageHandler.java:94) ~[spring-messaging-5.3.15.jar:5.3.15] at org.springframework.messaging.handler.invocation.AbstractMethodMessageHandler.handleMessageInternal(AbstractMethodMessageHandler.java:524) ~[spring-messaging-5.3.15.jar:5.3.15] at org.springframework.messaging.handler.invocation.AbstractMethodMessageHandler.handleMessage(AbstractMethodMessageHandler.java:458) ~[spring-messaging-5.3.15.jar:5.3.15] at org.springframework.messaging.support.ExecutorSubscribableChannel$SendTask.run(ExecutorSubscribableChannel.java:144) ~[spring-messaging-5.3.15.jar:5.3.15] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na] Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [995]/[1000] maximum shards open;] at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187) ~[elasticsearch-7.10.2.jar:7.10.2] at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1911) ~[elasticsearch-rest-high-level-client-7.10.2.jar:7.10.2] at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1888) ~[elasticsearch-rest-high-level-client-7.10.2.jar:7.10.2] at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1645) ~[elasticsearch-rest-high-level-client-7.10.2.jar:7.10.2] at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1602) ~[elasticsearch-rest-high-level-client-7.10.2.jar:7.10.2] at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1572) ~[elasticsearch-rest-high-level-client-7.10.2.jar:7.10.2] at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:989) ~[elasticsearch-rest-high-level-client-7.10.2.jar:7.10.2] at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.lambda$doIndex$0(ElasticsearchRestTemplate.java:143) ~[spring-data-elasticsearch-4.1.3.jar:4.1.3] at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.execute(ElasticsearchRestTemplate.java:343) ~[spring-data-elasticsearch-4.1.3.jar:4.1.3] ... 20 common frames omitted Suppressed: org.elasticsearch.client.ResponseException: method [PUT], host [https://vpc-elasticsearch-dev.ap-northeast-2.es.amazonaws.com:443], URI [/quiz_index/_doc/6D016EAB27984F4E82457271230B9538?timeout=1m], status line [HTTP/1.1 400 Bad Request] {"error":{"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [995]/[1000] maximum shards open;"}],"type":"validation_exception","reason":"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [995]/[1000] maximum shards open;"},"status":400} at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326) ~[elasticsearch-rest-client-7.10.2.jar:7.10.2] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296) ~[elasticsearch-rest-client-7.10.2.jar:7.10.2] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270) ~[elasticsearch-rest-client-7.10.2.jar:7.10.2] at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1632) ~[elasticsearch-rest-high-level-client-7.10.2.jar:7.10.2] ... 25 common frames omitted</pre>&nbsp;<hr>&nbsp;<h2>클러스터 상태 확인</h2>뭐 때문인지 클러스터 상태를 확인해 본다.&nbsp;<pre style="background-color:#2b2b2b;color:#a9b7c6;font-family:'D2Coding',monospace;font-size:11.3pt;">GET /_cluster/health?pretty</pre>&nbsp;&nbsp;<h3>결과</h3>문제시 되는 부분은 unassigned_shards 의 수가 500개 정도가 있어서 추가로 shard를 만들지 못해서 생긴 오류인 것 같다.<pre style="background-color:#2b2b2b;color:#a9b7c6;font-family:'D2Coding',monospace;font-size:11.3pt;">{ "cluster_name" : "747206413945:fxtl-elasticsearch-dev", "status" : "yellow", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "discovered_master" : true, "active_primary_shards" : 499, "active_shards" : 499, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 496, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 50.150753768844226 }</pre>&nbsp;<h3>목표 결과(추후 정상일 대 확인)</h3><pre style="background-color:#2b2b2b;color:#a9b7c6;font-family:'D2Coding',monospace;font-size:11.3pt;">{ "cluster_name" : "747206413945:fxtl-elasticsearch-dev", "status" : "green", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "discovered_master" : true, "active_primary_shards" : 560, "active_shards" : 560, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }</pre>&nbsp;<hr><h2>&nbsp;</h2><h2>문제 찾기</h2><pre style="background-color:#2b2b2b;color:#a9b7c6;font-family:'D2Coding',monospace;font-size:11.3pt;"># 문제가 있는 인덱스를 확인할 수 있다. GET /_cluster/allocation/explain?pretty # 인덱스를 검색했을 때 status가 yellow인 인덱스로도 알 수 있다. GET /_cat/indices?v</pre>&nbsp;<h3>클러스터 할당 설명 결과(/_cluster/allocation/explain?pretty)</h3>첫 줄에 문제되는 인덱스 명이 찍힌다.<pre style="background-color:#2b2b2b;color:#a9b7c6;font-family:'D2Coding',monospace;font-size:11.3pt;">{ "index" : "chat_11ed121f622ba1b7a4740247a2f173de", "shard" : 3, "primary" : false, "current_state" : "unassigned", "unassigned_info" : { "reason" : "CLUSTER_RECOVERED", "at" : "2022-08-29T01:23:24.339Z", "last_allocation_status" : "no_attempt" }, "can_allocate" : "no", "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes", "node_allocation_decisions" : [ { "node_id" : "MQo8VhNaTL-DS-uH9GiN1w", "node_name" : "46320cbe2666e8f6782572421cfde8ae", "node_decision" : "no", "deciders" : [ { "decider" : "same_shard", "decision" : "NO", "explanation" : "a copy of this shard is already allocated to this node [[chat_11ed121f622ba1b7a4740247a2f173de][3], node[MQo8VhNaTL-DS-uH9GiN1w], [P], s[STARTED], a[id=yeVG3-9dTKmIwLyWzoZ8Gw]]" } ] } ] }</pre>&nbsp;<hr>&nbsp;<h2>문제 해결</h2><h3>&nbsp;</h3><h3>1. 임시 해결(기존 인덱스에만 적용)</h3>노드 수는 1개인데 복제본 설정이 1로 되어있어서 경고(yellow)가 계속 떴었던 것 같다. 기존 인덱스에 대해서는 다음과 같이 복제본 수를 줄일 수 있었다.replicas는 현재 AWS에 1개의 노드만 운영중이므로 0으로 설정했다.<pre style="background-color:#2b2b2b;color:#a9b7c6;font-family:'D2Coding',monospace;font-size:11.3pt;">PUT /chat_*/_settings { "index": { "number_of_replicas": "0" } }</pre>&nbsp;<h3>2. template를 이용한 해결</h3><pre style="background-color:#2b2b2b;color:#a9b7c6;font-family:'D2Coding',monospace;font-size:11.3pt;"># 기본 템플릿 설정 확인 GET _template</pre>&nbsp;위의 문제 수정은 추가로 생성되는 인덱스에는 적용되지 않으므로 템플릿 설정으로 조정할 필요가 있었다. 모든 인덱스에 대해서 shards와 replicas에 대해서 설정한다 노드당 가질 수 있는 샤드 수가 정해진 것 같아서 우선 2로 할당하여 수를 줄였다.replicas는 현재 AWS에 1개의 노드만 운영중이므로 0으로 설정했다.<pre style="background-color:#2b2b2b;color:#a9b7c6;font-family:'D2Coding',monospace;font-size:11.3pt;">PUT _template/general_template { "index_patterns" : ["*"], "order" : 0, "settings" : { "number_of_shards" : "2", "number_of_replicas" : "0" } }</pre>&nbsp;&nbsp;<h3>2-1. 기존 template 적용(_template vs _index_template 차이 알아놔야함?)</h3><pre style="background-color:#2b2b2b;color:#a9b7c6;font-family:'D2Coding',monospace;font-size:11.3pt;"># 기존에 가지고 있던 인덱스 템플릿 조회 GET _index_template</pre>&nbsp;위에 기본 템플릿을 설정한다고 해도 별도로 생성한 인덱스 템플릿이 있다면 적용되지 않았다. 다음과 같이 settings를 추가하여 수정해 준다. 중간에 수정이 되지 않으므로 미리 템플릿 만들 때 넣어주는 게 좋을 것 같다.<pre style="background-color:#2b2b2b;color:#a9b7c6;font-family:'D2Coding',monospace;font-size:11.3pt;">PUT _index_template/cumulative_access_time { "index_patterns": ["user_cumulative_access_time_*"], "priority": 1000, "template": { "settings": { "number_of_shards": "2", "number_of_replicas": "0" }, "mappings": { "properties": { "dateRange": { "type": "date_range" } } } } }</pre>&nbsp;<h3>3. 노드 당 최대 샤드 수 조정</h3><pre style="background-color:#2b2b2b;color:#a9b7c6;font-family:'D2Coding',monospace;font-size:11.3pt;"># 클러스터 설정 확인 GET /_cluster/settings</pre>&nbsp;노드 당 최대 샤드 수를 3000으로 늘림 인덱스가 계속 늘다보면 active_primary_shards의 숫자도 계속 줄어들 것으로 판단하여 숫자 자체도 좀 늘려주었다.<pre style="background-color:#2b2b2b;color:#a9b7c6;font-family:'D2Coding',monospace;font-size:11.3pt;">PUT /_cluster/settings { "persistent": { "cluster.max_shards_per_node": "3000" } }</pre>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

ElasticSearch 오류 잡기 maximum shards open