I have a backend production with microservice architecture that I wrote with nestjs. All services here are running as separate docker containers. In addition to these services, in my docker-compose.yml file:
elasticsearch:
image: elasticsearch:7.6.2
container_name: elasticsearch
ports:
- '9200:9200'
- '9300:9300'
environment:
- cluster.name=docker-cluster
- node.name=node-1
- discovery.type=single-node
logstash:
labels:
com.example.service: 'logstash'
com.example.description: 'For logging data'
image: logstash:7.6.2
container_name: logstash
ports:
- '9600:9600'
depends_on:
- elasticsearch
volumes:
- '/var/lib/docker/containers:/var/lib/docker/containers:ro'
I added the parts and when I boot the project both elastichsearch and logstash start properly. then inside the log stash container docker exec -it logstash bash I entered with the command and configured the logstash.conf file in the pipline folder as follows.
input {
file {
path => "/var/lib/docker/containers/*/*-json.log"
type => "docker"
}
}
filter {
if [type] == "docker" {
grok {
match => { "source" => "/var/lib/docker/containers/%{DATA:container_id}/*-json.log" }
}
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "container-logs-%{container_id}"
}
}
then by logging out of the container, I ran the container again with the following command
docker restart logstash
The container is started properly, but when I run the command below, I observe that no logs are indexed in elastic search. Where could this be the reason? Where do you think I am doing wrong?
curl -XGET "http://localhost:9200/_search"
I get the following result as a result of the above command.
{"took":0,"timed_out":false,"_shards":{"total":0,"successful":0,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":0.0,"hits":[]}}
my logstash container logs
[2022-12-15T11:21:14,554][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2022-12-15T11:21:14,561][WARN ][logstash.outputs.elasticsearch][.monitoring-logstash] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2022-12-15T11:21:14,572][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] ES Output version determined {:es_version=>7}
[2022-12-15T11:21:14,573][WARN ][logstash.outputs.elasticsearch][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2022-12-15T11:21:14,615][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://elasticsearch:9200"]}
[2022-12-15T11:21:14,626][INFO ][logstash.javapipeline ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x1e9a9f6f run>"}
[2022-12-15T11:21:14,696][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2022-12-15T11:21:14,705][INFO ][logstash.agent ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}
[2022-12-15T11:21:14,934][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
my elasticsearch container log
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2022-12-14T14:04:56,680Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "using [1] data paths, mounts [[/ (overlay)]], net usable_space [5.8gb], net total_space [19.2gb], types [overlay]" }
{"type": "server", "timestamp": "2022-12-14T14:04:56,684Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "heap size [1007.3mb], compressed ordinary object pointers [true]" }
{"type": "server", "timestamp": "2022-12-14T14:04:56,759Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "node name [node-1], node ID [KSEf_YAqQwuCymU27MN1cA], cluster name [docker-cluster]" }
{"type": "server", "timestamp": "2022-12-14T14:04:56,760Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "version[7.6.2], pid[1], build[default/docker/ef48eb35cf30adf4db14086e8aabd07ef6fb113f/2020-03-26T06:34:37.794943Z], OS[Linux/5.15.0-1011-aws/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.2/13.0.2 8]" }
{"type": "server", "timestamp": "2022-12-14T14:04:56,761Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "JVM home [/usr/share/elasticsearch/jdk]" }
{"type": "server", "timestamp": "2022-12-14T14:04:56,761Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX: AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX: UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX: UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-10762875107622396116, -XX: HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,909Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [aggs-matrix-stats]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,910Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [analysis-common]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,910Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [flattened]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,911Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [frozen-indices]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,911Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [ingest-common]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,912Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [ingest-geoip]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,918Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [ingest-user-agent]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,919Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [lang-expression]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,919Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [lang-mustache]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,919Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [lang-painless]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,919Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [mapper-extras]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,920Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [parent-join]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,920Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [percolator]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,921Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [rank-eval]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,921Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [reindex]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,922Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [repository-url]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,922Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [search-business-rules]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,922Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [spatial]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,923Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [transform]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,924Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [transport-netty4]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,925Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [vectors]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,925Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-analytics]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,925Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-ccr]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,926Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-core]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,926Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-deprecation]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,926Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-enrich]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,926Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-graph]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,927Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-ilm]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,927Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-logstash]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,927Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-ml]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,927Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-monitoring]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,928Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-rollup]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,929Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-security]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,929Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-sql]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,929Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-voting-only-node]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,930Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "loaded module [x-pack-watcher]" }
{"type": "server", "timestamp": "2022-12-14T14:04:59,930Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "no plugins loaded" }
{"type": "server", "timestamp": "2022-12-14T14:05:06,042Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
{"type": "server", "timestamp": "2022-12-14T14:05:07,153Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "[controller/103] [Main.cc@110] controller (64 bit): Version 7.6.2 (Build e06ef9d86d5332) Copyright (c) 2020 Elasticsearch BV" }
{"type": "server", "timestamp": "2022-12-14T14:05:08,342Z", "level": "DEBUG", "component": "o.e.a.ActionModule", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "Using REST wrapper from plugin org.elasticsearch.xpack.security.Security" }
{"type": "server", "timestamp": "2022-12-14T14:05:08,546Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
{"type": "server", "timestamp": "2022-12-14T14:05:10,003Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "initialized" }
{"type": "server", "timestamp": "2022-12-14T14:05:10,004Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "starting ..." }
{"type": "server", "timestamp": "2022-12-14T14:05:10,184Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "publish_address {192.168.0.16:9300}, bound_addresses {0.0.0.0:9300}" }
{"type": "server", "timestamp": "2022-12-14T14:05:10,570Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "setting initial configuration to VotingConfiguration{KSEf_YAqQwuCymU27MN1cA}" }
{"type": "server", "timestamp": "2022-12-14T14:05:10,823Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "elected-as-master ([1] nodes joined)[{node-1}{KSEf_YAqQwuCymU27MN1cA}{7Qd6fK-7TuKA0JyF45eWHA}{192.168.0.16}{192.168.0.16:9300}{dilm}{ml.machine_memory=8230952960, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{node-1}{KSEf_YAqQwuCymU27MN1cA}{7Qd6fK-7TuKA0JyF45eWHA}{192.168.0.16}{192.168.0.16:9300}{dilm}{ml.machine_memory=8230952960, xpack.installed=true, ml.max_open_jobs=20}]}" }
{"type": "server", "timestamp": "2022-12-14T14:05:10,892Z", "level": "INFO", "component": "o.e.c.c.CoordinationState", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "cluster UUID set to [8_YjSvrMR_Gt53WnrnviLA]" }
{"type": "server", "timestamp": "2022-12-14T14:05:10,988Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "master node changed {previous [], current [{node-1}{KSEf_YAqQwuCymU27MN1cA}{7Qd6fK-7TuKA0JyF45eWHA}{192.168.0.16}{192.168.0.16:9300}{dilm}{ml.machine_memory=8230952960, xpack.installed=true, ml.max_open_jobs=20}]}, term: 1, version: 1, reason: Publication{term=1, version=1}" }
{"type": "server", "timestamp": "2022-12-14T14:05:11,104Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "publish_address {192.168.0.16:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T14:05:11,105Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "started", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T14:05:11,279Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "recovered [0] indices into cluster_state", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T14:05:11,671Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.triggered_watches] for index patterns [.triggered_watches*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T14:05:11,753Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.watches] for index patterns [.watches*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T14:05:11,879Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.watch-history-10] for index patterns [.watcher-history-10*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T14:05:11,965Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [ilm-history] for index patterns [ilm-history-1*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T14:05:12,033Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.slm-history] for index patterns [.slm-history-1*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T14:05:12,134Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-7-*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T14:05:12,245Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.monitoring-es] for index patterns [.monitoring-es-7-*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T14:05:12,317Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.monitoring-beats] for index patterns [.monitoring-beats-7-*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T14:05:12,377Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.monitoring-alerts-7] for index patterns [.monitoring-alerts-7]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T14:05:12,441Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-7-*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T14:05:12,523Z", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding index lifecycle policy [watch-history-ilm-policy]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T14:05:12,580Z", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding index lifecycle policy [ilm-history-ilm-policy]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T14:05:12,636Z", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding index lifecycle policy [slm-history-ilm-policy]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T14:05:12,794Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "license [3693ddfc-37ba-4874-bb54-0651c5976db2] mode [basic] - valid", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T14:05:12,796Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "Active license is now [BASIC]; Security is disabled", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-14T15:28:59,574Z", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "adding template [logstash] for index patterns [logstash-*]", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-15T01:30:00,014Z", "level": "INFO", "component": "o.e.x.s.SnapshotRetentionTask", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "starting SLM retention snapshot cleanup task", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-15T01:38:00,002Z", "level": "INFO", "component": "o.e.x.m.MlDailyMaintenanceService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "triggering scheduled [ML] maintenance tasks", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-15T01:38:00,004Z", "level": "INFO", "component": "o.e.x.m.a.TransportDeleteExpiredDataAction", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "Deleting expired data", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-15T01:38:00,058Z", "level": "INFO", "component": "o.e.x.m.a.TransportDeleteExpiredDataAction", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "Completed deletion of expired ML data", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
{"type": "server", "timestamp": "2022-12-15T01:38:00,060Z", "level": "INFO", "component": "o.e.x.m.MlDailyMaintenanceService", "cluster.name": "docker-cluster", "node.name": "node-1", "message": "Successfully completed [ML] maintenance tasks", "cluster.uuid": "8_YjSvrMR_Gt53WnrnviLA", "node.id": "KSEf_YAqQwuCymU27MN1cA" }
CodePudding user response：
You can run the following command and check the status of Logstash:
curl -XGET 'localhost:9600/_node/stats/events?pretty'
The output will be like this:
In the screenshot above my elasticsearch is not reachable so all events get stuck.
The error I get from logstash:
[2022-12-15T18:28:28,541][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused"}
CodePudding user response：
After trying to replicate your setup, it seems to me that it is a permission problem. Even though you are bind mounting with :r, host permissions persist into the container fs. Logstash runs with user 1000 and container logs require root permissions.
FROM logstash:7.6.4
USER root
docker build -t mylogstash .
Now get in and try your configs:
logstash:
labels:
com.example.service: 'logstash'
com.example.description: 'For logging data'
image: mylogstash
container_name: logstash
ports:
- '9600:9600'
depends_on:
- elasticsearch
volumes:
- '/var/lib/docker/containers:/var/lib/docker/containers:ro'
I personally would not use logstash this way, instead setup Filebeat to monitor the logs with docker input or docker autodiscover. I use logstash then to parse the log message into json fields, remove unnecessary meta etc.