Getting error while trying to Import data from MongoDB to ElasticSearch using Nodejs-CodePudding

I am trying to import small documents from MongoDB to ElasticSearch but getting an error

      {
    "index": {
      "_index": "impact-fulltext",
      "_id": "t2oLkoUBwNXsTufYzszL",
      "status": 400,
      "error": {
        "type": "mapper_parsing_exception",
        "reason": "failed to parse field [_id] of type [_id] in document with id \u0027t2oLkoUBwNXsTufYzszL\u0027. Preview of field\u0027s value: \u0027605315a3b4f719d00f69f2d3\u0027",
        "caused_by": {
          "type": "mapper_parsing_exception",
          "reason": "Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters."
        }
      }
    }
  }

I am clueless as I also defined the _id but still getting the error.

    db.collection("article_beta")
  .find()
  .limit(100)
  .toArray((err, docs) => {
    if (err) throw err;

    esClient.bulk(
      {
        body: docs.flatMap((doc) => [
          {
            index: {
              _index: "impact-fulltext",
              _id: doc._id.$oid,
            },
          },
          doc,
        ]),
      },
      (err, resp) => {
        if (err) throw err;
        console.log(resp);
        client.close();
      }
    );
  });

CodePudding user response：

In elasticsearch, the documents have some meta information and also the source of the documents. So, the document structure is following :

{
   "_index": "index-name",
   "_id": "document-id-stored-by-elasticsearch",
   "_source": {
      // documents itself
   }
}

So, inside the document, you can not use _id field as follow because _id field is reserved field :

POST myindex/_doc
{
  "_id": "123123",
  "field1": "value1"
}

So this request will give an error as follow :

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters."
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "failed to parse field [_id] of type [_id] in document with id '16Y6koUBnxYxC21BP-tS'. Preview of field's value: '123123'",
    "caused_by": {
      "type": "mapper_parsing_exception",
      "reason": "Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters."
    }
  },
  "status": 400
}

In your example, doc variable which the value coming from Mongodb have a field as _id. You need to change the name of that field as id. You can put following lines inside of your anonymous function inside the flatMap function.

db.collection("article_beta")
  .find()
  .limit(100)
  .toArray((err, docs) => {
    if (err) throw err;

    esClient.bulk(
      {
        body: docs.flatMap((doc) => {
          doc.id = doc._id  // <---- added
          delete doc._id    // <---- added
          return  [
            {
              index: {
                _index: "impact-fulltext",
                _id: doc.id.$oid,
              },
            },
            doc,
          ]
        }),
      },
      (err, resp) => {
        if (err) throw err;
        console.log(resp);
        client.close();
      }
    );
  });