Skip to content

Elasticsearch

Elasticsearch는 점점 많은 문제를 해결하는 분산형 RESTful 검색 및 분석 엔진입니다. Elastic Stack의 핵심으로, 데이터를 중심부에 저장하여 예상되는 항목을 검색하고 예상치 못한 항목을 밝혀낼 수 있습니다.

Categories

Shards and Replicas 

DELETE /your_index/your_type/_query
{
  "query": {
    "range": {
      "timestamp": {
        "lte": "now-10y"
      }
    }
  }
}

Install Elasticsearch with Docker

cluster.name=docker-cluster
ES Cluster명입니다. ES 서비스마다 동일한 명칭을 사용해야합니다.
node.name=master-node1
ES Node명을 설정합니다.
bootstrap.memory_lock=true
ES운영중 메모리 스왑을 막기 위한 설정을 추가합니다.
"ES_JAVA_OPTS=-Xms512m -Xmx512m"
JVM Heap메모리 설정입니다. Xms/Xmx 옵션은 항상 같게 설정합니다.
ulimits
리눅스 시스템 자원제한 관련 옵션입니다. ES는 많은 파일디스크립터와 핸들러를 사용하기 때문에 제한 해제가 필요합니다.

Starting a single node cluster with Docker 

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.10.1

Starting a multi-node cluster with Docker Compose 

version: '2.2'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.1
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic
  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.1
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
    networks:
      - elastic
  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.1
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data03:/usr/share/elasticsearch/data
    networks:
      - elastic

volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local

networks:
  elastic:
    driver: bridge

use elasticsearch in gitlab-ci

gitlab-ci에서 테스트용으로 사용하고 싶을 경우 다음과 같이 적용하면 된다. 

test-elasticsearch:
  stage: test
  services:
    - name: "docker.elastic.co/elasticsearch/elasticsearch:7.10.1"
      alias: "elasticsearch"
      command: [ "bin/elasticsearch", "-Expack.security.enabled=false", "-Ediscovery.type=single-node" ]
  script:
    - curl "http://elasticsearch:9200/_cat/health"

Docker GELF logging

Logstash#Docker GELF logging 항목 참조.

ELK Stack

Elasticsearch vs Solr

Troubleshooting

error 137 in docker container

간단히, 메모리 크기가 부족할 수 있다. Docker#error code 137 항목 참조.

max virtual memory areas vm.max_map_count is too low

Elasticsearch 가 종료될 때 로그를 보면: 

~/elasticsearch-5.6.1/bin$ [2017-10-11T06:39:53,758][DEBUG][o.e.a.ActionModule       ] Using REST 
wrapper from plugin org.elasticsearch.xpack.XPackPlugin
[2017-10-11T06:39:55,926][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/16678] [Main.cc@128] controller (6
4 bit): Version 5.6.1 (Build e81fa9e3f3e0c1) Copyright (c) 2017 Elasticsearch BV
[2017-10-11T06:39:55,972][INFO ][o.e.d.DiscoveryModule    ] [node1] using discovery type [zen]
[2017-10-11T06:39:57,291][INFO ][o.e.n.Node               ] [node1] initialized
[2017-10-11T06:39:57,292][INFO ][o.e.n.Node               ] [node1] starting ...
[2017-10-11T06:39:57,666][INFO ][o.e.t.TransportService   ] [node1] publish_address {10.146.0.5:9300}, bound_addres
ses {[::]:9300}
[2017-10-11T06:39:57,679][INFO ][o.e.b.BootstrapChecks    ] [node1] bound or publishing to a non-loopback or non-li
nk-local address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2017-10-11T06:39:57,692][INFO ][o.e.n.Node               ] [node1] stopping ...
[2017-10-11T06:39:57,731][INFO ][o.e.n.Node               ] [node1] stopped
[2017-10-11T06:39:57,732][INFO ][o.e.n.Node               ] [node1] closing ...
[2017-10-11T06:39:57,750][INFO ][o.e.n.Node               ] [node1] closed
[2017-10-11T06:39:57,753][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new nativ
e processes can be started
[1]+  Exit 78                 ./elasticsearch

vm.max_map_count를 제시된 값과 같이 수정하면 된다: 

sudo sysctl -w vm.max_map_count=262144

See also

Favorite site

Documentation

Tutorials

Docker