Update existing document of Elasticsearch and insert current record through logstash-CodePudding

I am trying to insert a record into elasticsearch and also update a field of an existing document whose _id I'll be getting from the current record. After searching online, I found that we can use the _update_by_query api with the http plugin in logstash. This is the below configuration.

output {

    elasticsearch {
            hosts => ["localhost:9200"]
            index => "my_index_*"
            document_id => "%{id_field}"
       }

    http {
           url => "http://localhost:9200/my_index_*/_update_by_query"
           http_method => "post"
           content_type => "application/json"
           format => "message"
           message => '{"query":{"match":{"_id":"%{previous_record_id}"}},"script":{"source":"ctx._source.field_to_be_updated=xyz","lang":"painless"}}'

       }
}

The Elasticsearch has no password protection and so I haven't added an authorization header. But when I start logstash, the current record gets inserted but I always the below error for the http plugin.

2022-05-05T11:31:51,916][ERROR][logstash.outputs.http    ][logstash_txe] [HTTP Output Failure] Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"http://localhost:9200/my_index_*/_update_by_query", :event=>#<LogStash::Event:0x192606f8>}

CodePudding user response：

It's not how you're supposed to do it, you can simply use the elasticsearch output for both use cases.

The first one for indexing a new record and the following one for partial updating another record whose id is previous_record_id. The event data can be accessed in params.event within the script:

elasticsearch {
   hosts => ["localhost:9200"]
   index => "my_index_xyz"
   document_id => "%{previous_record_id}"
   action => "update"

   script => "ctx._source.field_to_be_updated = params.event.xyz"
   script_lang => "painless"
   script_type => "inline"
}