If you need to migrate Elasticsearch on another server with data, this may be a challenge for you if you have not done this already.
Before reaching here, you might have done various things including installation of Elasticsearch on new server. Now you want to import all existing data from old server to new server. Check your Elasticsearch version on old server, have you installed near same version on new server?
Suppose you had 7.x version on your old server, any version of 7 will work on new server too. Elasticsearch tutorial says, you can migrate 6.x version data on 7.x, however I have not tried it before.
I had 7.6 version on old server and I add 7.x in apt/sources.list of new server, I got 7.7 on new server and migration worked pretty well.
There is a method which is approved by elasticsearch itself to migrate data and it is based on creating snapshot and restoring them. I have not used it in the tutorial as there are several tutorial available for it. I have used different method to achieve same task.
Steps :
1) First list all indices on old server
Before reaching here, you might have done various things including installation of Elasticsearch on new server. Now you want to import all existing data from old server to new server. Check your Elasticsearch version on old server, have you installed near same version on new server?
Suppose you had 7.x version on your old server, any version of 7 will work on new server too. Elasticsearch tutorial says, you can migrate 6.x version data on 7.x, however I have not tried it before.
I had 7.6 version on old server and I add 7.x in apt/sources.list of new server, I got 7.7 on new server and migration worked pretty well.
There is a method which is approved by elasticsearch itself to migrate data and it is based on creating snapshot and restoring them. I have not used it in the tutorial as there are several tutorial available for it. I have used different method to achieve same task.
Steps :
1) First list all indices on old server
curl -L localhost:9200/_cat/indices
If your elasticsearch server has http authentication password, you need to pass it with curl.This is my output
green open .apm-custom-link NQlqIQaCwO_S8jWYSZA 1 0 0 0 208b 208b
yellow open kibana_sample_data_ecommerce eMXnWTJylltrthTGqgw 1 1 4675 0 4.3mb 4.3mb
green open .kibana_task_manager_1 g7hUKicRTHufkJ65Hlg 1 0 5 3 33.6kb 33.6kb
yellow open conzexant_index5 HXQXmMBaBgcUxzrWORw 1 1 1 0 3.5kb 3.5kb
yellow open conzexant_other_entities ZseVzSv6RaCxAMwn48w 1 1 1 0 99.1kb 99.1kb
green open .apm-agent-configuration rcZAEX0NSDVbYJtvi8g 1 0 0 0 208b 208b
yellow open conzexant_index1 Ne_S7UQSQuyyXwFgb_Q 1 1 13916 0 28.2mb 28.2mb
yellow open conzexant_index2 leoZmPUqQoekP9DhIYQ 1 1 14355 0 68.3mb 68.3mb
green open .kibana_1 _U1SYeCKT5aGXgTldvA 1 0 123 4 1005.7kb 1005.7kb
yellow open kibana_sample_data_ecommerce eMXnWTJylltrthTGqgw 1 1 4675 0 4.3mb 4.3mb
green open .kibana_task_manager_1 g7hUKicRTHufkJ65Hlg 1 0 5 3 33.6kb 33.6kb
yellow open conzexant_index5 HXQXmMBaBgcUxzrWORw 1 1 1 0 3.5kb 3.5kb
yellow open conzexant_other_entities ZseVzSv6RaCxAMwn48w 1 1 1 0 99.1kb 99.1kb
green open .apm-agent-configuration rcZAEX0NSDVbYJtvi8g 1 0 0 0 208b 208b
yellow open conzexant_index1 Ne_S7UQSQuyyXwFgb_Q 1 1 13916 0 28.2mb 28.2mb
yellow open conzexant_index2 leoZmPUqQoekP9DhIYQ 1 1 14355 0 68.3mb 68.3mb
green open .kibana_1 _U1SYeCKT5aGXgTldvA 1 0 123 4 1005.7kb 1005.7kb
The indices which I need to copy are kibana_sample_data_ecommerce, conzexant_index5, conzexant_index1, conzexant_index2 and conzexant_other_entities
In the output of curl, third column is indices name, if you run same command on new server, you will get default list of indices. While comparing indices name with old server you can easily find the list of indices what you need on new server.
2) Install elasticdump. Now here you need to verify the access, are you able to access elasticsearch from old server to new server or new server to old server using curl command. If you can do both then you can install elasticdump anywhere, if you access in one way, you need to install it there only.
If you can not access from anywhere then there is method B, export in json and import, we will talk about it shortly.
Suppose you can access elasticsearch of new server from old server, you need to log into the old server and install elasticdump.
Install node version greater than 8. npm will be installed with it.
Now install elasticdump.
3) Now export and import all indices on new server using following command.
a) First Dump Analyzer on new server
2) Install elasticdump. Now here you need to verify the access, are you able to access elasticsearch from old server to new server or new server to old server using curl command. If you can do both then you can install elasticdump anywhere, if you access in one way, you need to install it there only.
If you can not access from anywhere then there is method B, export in json and import, we will talk about it shortly.
Suppose you can access elasticsearch of new server from old server, you need to log into the old server and install elasticdump.
Install node version greater than 8. npm will be installed with it.
Now install elasticdump.
npm install elasticdump
If you have run this command in your home directory, it is installed inside node_modules folder.3) Now export and import all indices on new server using following command.
a) First Dump Analyzer on new server
~/node_modules/elasticdump/bin/elasticdump --input=http://localhost:9200/index_name --output=http://newserverIP:9200/index_name --type=analyzer
Here index_name is your index name which you want to migrate on new server.In my case the command was
~/node_modules/elasticdump/bin/elasticdump --input=http://localhost:9200/conzexant_index5 --output=http://newserverIP:9200/conzexant_index5 --type=analyzer
New server IP was 35.34.xxx.xxx
b) then Dump Mapping on new server
~/node_modules/elasticdump/bin/elasticdump --input=http://localhost:9200/index_name --output=http://newserverIP:9200/index_name --type=mapping
In my case the command was~/node_modules/elasticdump/bin/elasticdump --input=http://localhost:9200/conzexant_index5 --output=http://newserverIP:9200/conzexant_index5 --type=mapping
c) and finally Dump Data
~/node_modules/elasticdump/bin/elasticdump --input=http://localhost:9200/index_name --output=http://newserverIP:9200/index_name --type=data
Similarly I dumped data with index name.I have considered here, there is no authentication on elastic search (which is very bad) but if there is authentication, you need to pass the credentials with the command.
4) Once it is done you can verify data on new server. Log into the new server. Run.
4) Once it is done you can verify data on new server. Log into the new server. Run.
curl -L localhost:9200/_cat/indices
Imported index will be listed on new server with all data. Similarly you need to import rest of the indices.
Method : B
This is the process to import and export data of one index, similarly you can export and import data of all indexes.
If you can not access new server from old server or vice-versa, you need to export data in json on old server and then you need to import it in new server.
In this method, you need to install elasticdump on both servers.
Log into the old server
Export :
Export Analyzer :
~/node_modules/elasticdump/bin/elasticdump --input=http://localhost:9200/index_name --output=index_name-analyzer.json --type=analyzer
Export Mapping :
~/node_modules/elasticdump/bin/elasticdump --input=http://localhost:9200/index_name --output=index_name-mapping.json --type=mapping
Export Data :
~/node_modules/elasticdump/bin/elasticdump --input=http://localhost:9200/index_name --output=index_name-data.json --type=data
Copy all three json files on new server and log into the new server using ssh.
cd into the directory where json files are stored, now import all one by one.
Import :
Import Analyzer :
~/node_modules/elasticdump/bin/elasticdump --input=index_name-analyzer.json --output=http://localhost:9200/index_name --type=analyzer
Import Mapping :
~/node_modules/elasticdump/bin/elasticdump --input=index_name-mapping.json --output=http://localhost:9200/index_name --type=mapping
Import Data :
~/node_modules/elasticdump/bin/elasticdump --input=index_name-data.json --output=http://localhost:9200/index_name --type=data
You can write a shell script to export all indeces from old server and import all into new server.
Note : Elasticsearch also offers snapshot creation and Restore method to migrate data. This is just another method to give you same result.
Note : Elasticsearch also offers snapshot creation and Restore method to migrate data. This is just another method to give you same result.
No comments:
Post a Comment