Elasticsearch workspace: dockerfile and readme

workspaces/elasticsearch-workspace/Dockerfile

@@ -0,0 +1,42 @@
+
+ARG docker_registry=docker.io/alnoda
+ARG image_tag=18.04-0.11
+
+FROM ${docker_registry}/workspace-in-docker:${image_tag}
+USER root
+
+COPY settings.json /home/abc/.theia/settings.json
+COPY README.md /home/docs/docs/docs.md
+
+RUN apt-get -y update \
+    # && echo "------------------------------------------------------ utils" \
+    # && rm -rf /home/abc/utils || true \
+    # && git clone https://github.com/bluxmit/alnoda-workspaces /tmp/alnoda-workspaces \
+    # && mv /tmp/alnoda-workspaces/utils /home/abc/ \
+    # && rm -rf /tmp/alnoda-workspaces \
+    && echo "------------------------------------------------------ nodejs-17" \
+    && curl -fsSL https://deb.nodesource.com/setup_17.x | bash - \
+    && apt-get install -y nodejs \
+    && echo "------------------------------------------------------ esbulk" \
+    && cd /tmp && wget https://github.com/miku/esbulk/releases/download/v0.7.5/esbulk_0.7.5_amd64.deb \
+    && apt install /tmp/esbulk_0.7.5_amd64.deb \
+    && rm /tmp/esbulk_0.7.5_amd64.deb \
+    && echo "------------------------------------------------------ elasticsearch-dump" \
+    && npm install -g elasticdump@6.79.3 \
+    && echo "------------------------------------------------------ vulkanizer" \
+    && cd /tmp && wget https://github.com/github/vulcanizer/releases/download/v0.8.0/vulcanizer_0.8.0_Linux_i386.tar.gz \
+    && tar -xzf /tmp/vulcanizer_0.8.0_Linux_i386.tar.gz \
+    && chmod +x /tmp/vulcanizer \
+    && mv /tmp/vulcanizer /usr/bin/vulcanizer \
+    && rm /tmp/vulcanizer_0.8.0_Linux_i386.tar.gz \
+    && echo "------------------------------------------------------ user" \
+    && chown -R abc /home/docs \
+    && chown -R abc /home/abc/utils \
+    && chown -R abc /home/abc/installed-python-packages \
+    && find /home -type d | xargs -I{} chown -R abc {} \
+    && find /home -type f | xargs -I{} chown abc {} 
+    
+USER abc
+
+
+

workspaces/elasticsearch-workspace/README.md

@@ -0,0 +1,177 @@
+# Elasticsearch workspace
+
+A tool to quickly develop and schedule Elasticsearch/Opensearch maintenance scripts. 
+Such as taking snapshot backups of indexes, migrating to another Elasticsearch cluster, transferring Elasticsearch data to S3, importing data into Elasticsearch. 
+As well as deleting/moving/archeiving old indexes, reindexing, rebalancing uneven shard distribution, export report queries to S3, etc.
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/workspace-in-docker/img/wid-collage.png" alt="Collage">
+</p>
+
+Workspace has IDE to quickly develop scripts, visual scheduler to run those scripts periodically and monitor succesful/failed jobs, 
+Filebrowser to upload/download data files directly inside the workspace, browser-based terminal. All the workspace tools are 
+browser-based, which makes it suitable for running workspace in kubernetes.
+
+```
+docker run --name rwid-1 -d -p 8020-8035:8020-8035 alnoda/elasticsearch-workspace
+```
+
+open your browser on [http://localhost:8020](http://localhost:8020)
+
+## What's includes
+- [elasticdump](https://github.com/elasticsearch-dump/elasticsearch-dump) - sawesome tool for moving and saving indices.
+- [esbulk](https://github.com/miku/esbulk) - fast parallel command line bulk loading utility for Elasticsearch. 
+- [vulcanizer](https://github.com/github/vulcanizer) - cli for interacting with an Elasticsearch cluster.
+
+in addition - [the common workspace-in-docker toolset](https://github.com/bluxmit/alnoda-workspaces/tree/main/workspaces/workspace-in-docker). 
+
+## Get started
+
+Create `docker-compose.yaml` file to launch locally 3-node Elasticsearch cluster with Kibana 
+```yaml
+version: '2.2'
+services:
+  es01:
+    image: docker.elastic.co/elasticsearch/elasticsearch:7.16.3
+    container_name: es01
+    environment:
+      - node.name=es01
+      - cluster.name=es-docker-cluster
+      - discovery.seed_hosts=es02,es03
+      - cluster.initial_master_nodes=es01,es02,es03
+      - bootstrap.memory_lock=true
+      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+    volumes:
+      - data01:/usr/share/elasticsearch/data
+    ports:
+      - 9200:9200
+    networks:
+      - elastic
+
+  es02:
+    image: docker.elastic.co/elasticsearch/elasticsearch:7.16.3
+    container_name: es02
+    environment:
+      - node.name=es02
+      - cluster.name=es-docker-cluster
+      - discovery.seed_hosts=es01,es03
+      - cluster.initial_master_nodes=es01,es02,es03
+      - bootstrap.memory_lock=true
+      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+    volumes:
+      - data02:/usr/share/elasticsearch/data
+    networks:
+      - elastic
+
+  es03:
+    image: docker.elastic.co/elasticsearch/elasticsearch:7.16.3
+    container_name: es03
+    environment:
+      - node.name=es03
+      - cluster.name=es-docker-cluster
+      - discovery.seed_hosts=es01,es02
+      - cluster.initial_master_nodes=es01,es02,es03
+      - bootstrap.memory_lock=true
+      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+    volumes:
+      - data03:/usr/share/elasticsearch/data
+    networks:
+      - elastic
+
+  kib01:
+    image: docker.elastic.co/kibana/kibana:7.16.3
+    container_name: kib01
+    ports:
+      - 5601:5601
+    environment:
+      ELASTICSEARCH_URL: http://es01:9200
+      ELASTICSEARCH_HOSTS: '["http://es01:9200","http://es02:9200","http://es03:9200"]'
+    networks:
+      - elastic
+  
+  workspace:
+    image: alnoda/elasticsearch-workspace
+    container_name: workspace
+    ports:
+      - 8020-8030:8020-8030
+    networks:
+      - elastic
+
+volumes:
+  data01:
+    driver: local
+  data02:
+    driver: local
+  data03:
+    driver: local
+
+networks:
+  elastic:
+    driver: bridge
+```
+and start it with `docker-compose up`   
+
+Wait untill the cluster is fully ready, open Kibana on [http://localhost:5601](/http://localhost:5601/) and import 
+all sample datasets.  
+
+Open workspace UI [http://localhost:8020/](http://localhost:8020/) for quick access to all the workspace tools 
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/workspace-in-docker/img/wid-demo.gif" alt="WID demo" width="900">
+</p>
+
+Open browser-based terminal [http://localhost:8026/](http://localhost:8026/), check cluster nodes and shards
+
+```
+vulcanizer --host es01 nodes
+vulcanizer --host es01 shards
+```
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/elasticsearch-workspace/img/vulcanizer-demo.png" alt="vulcanizer" width="450">
+</p>
+
+Use elasticdump to export index `kibana_sample_data_ecommerce` (from eCommerce sample dataset) to S3 
+```
+elasticdump \
+  --s3AccessKeyId "${access_key_id}" \
+  --s3SecretAccessKey "${access_key_secret}" \
+  --input=http://es01:9200/kibana_sample_data_ecommerce \
+  --output "s3://${bucket_name}/kibana_sample_data_ecommerce.json"
+```
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/elasticsearch-workspace/img/elasticdump-demo.png" alt="elasticdump" width="450">
+</p>
+
+Open browser-based IDE [http://localhost:8026/](http://localhost:8026/) and create file `/home/project/export.sh` file with the script to export data to S3. 
+Make it executable with `chmod +x /home/project/export.sh`. 
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/elasticsearch-workspace/img/IDE-demo.png" alt="IDE" width="450">
+</p>
+
+Open browser-based Scheduler [http://localhost:8026/](http://localhost:8026/) (user/pass: admin/admin), 
+and schedule script, for example weekly. Select category - "general", plugin - "Shell Script"
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/bluxmit/alnoda-workspaces/main/workspaces/elasticsearch-workspace/img/Cronicle-demo.png" alt="Cronicle" width="450">
+</p>
+
+
+
+
+
+

workspaces/elasticsearch-workspace/settings.json

@@ -0,0 +1,5 @@
+{
+    "terminal.integrated.shell.linux": "/bin/zsh",
+    "workbench.colorTheme": "Search Lights Italic",
+    "git-graph.maxDepthOfRepoSearch": 3
+}