Export data from OpenSearch in private VPC and import it to local running container

I'm using aws OpenSearch in a private vpc. I've about 10000 entries under some index.
For local development i'm running an local OpeanSearch container and i'd like to export all the entries from the OpenSearch service into my local container.

I can get all the entries from the OpeanSerch API but the format of the response is different then the format that should be when doing _bulk operation.

Can someone please tell me how should i do it?

CodePudding user response：

Anna,

There are different strategies you can take to accomplish this, considering the fact that your domain is running in a private VPC.

Option 1: Exporting and Importing Snapshots

From the security standpoint, this is the recommended option, as you are moving entire indices out of the service without exposing the data. Please follow the AWS official documentation about how to create custom index snapshots. Once you complete the steps, you will have an index snapshot stored on an Amazon S3 bucket. After this, you can securely download the index snapshot to your local machine, then follow the instructions on the official OpenSearch documentation about how to restore the index snapshots.

Option 2: Using VPC Endpoints

Another way for you to export the data from your OpenSearch domain is accessing the data via a alternate endpoint using the VPC Endpoints feature from AWS OpenSearch. It allows you to to expose additional endpoints running on public or private subnets within the same VPC, different VPC, or different AWS accounts. In this case, you are essentially create a venue to access the OpenSearch REST APIs outside of the private VPC, to which you need to take care of who other than you will be able to do so as well. Please follow the best practices related to secure endpoints if you follow this option.

Option 3: Using the ElasticDump Open Source Utility

The ElasticDump utility allows you to retrieve data from Elasticsearch/OpenSearch clusters in a format of your preference, and then import that data back to another cluster. It is a very flexible way for you to move data around—but it requires the utility to access the REST API endpoints from the cluster. Run this utility in a bastion server that has ingress access to your OpenSearch domain in the private VPC. Keep in mind, though, that AWS doesn't provide any support to this utility, and you must use it at your own risk.

I hope that helps with your question. Let us know if you need any more help on this.