Hevo can replicate indexes from your Elasticsearch cluster.
Please follow these steps to create a Pipeline with Elasticsearch as a Source.
1. Create a new Pipeline
Click on PIPELINES option in the left navigation bar and click on Create New Pipeline.
2. Select Source Type
Select Elasticsearch from the list on Select Source Type Screen.
3. Provide Connection Settings
Provide your Elasticsearch cluster connection details on Elasticsearch Connection Settings page. You will have following options in connection details block:
- Source Name - A unique name for this source
- Database Host - Elasticsearch's IP address or DNS
- Database Port - The port on which your Elasticsearch is listening for connections (default is 9200)
- Database User - The read-only user that can read the indexes.
- Database Password - Password for the read-only user.
- If you want to connect to Hevo using an SSH server, check How to Connect through SSH. Else, you will have to whitelist Hevo's IP addresses which will be highlighted on the screen, For eg. in this case you will have to whitelist following IP addresses:
Alternatively, you copy details from an existing source of Elasticsearch type. Please note that it will create an independent copy of the selected source.
Click TEST CONNECTION to test the credentials and click CONTINUE once test succeeds.
4. Provide Job Settings
Here, you will be presented with the list of indexes in your Elasticsearch cluster. You can deselect the indexes that you don't want to replicate. For every index, you can specify an incrementing field and a date field (which keeps the last modified timestamp for a document). If you specify the date field Hevo will be able to replicate updates happening to the documents in the index, otherwise, Hevo will replicate only new documents being inserted through the incrementing field. If both fields are not specified, Hevo will attempt to replicate the complete index at each run.
After entering the details, click CONTINUE.
5. Select the Destination
Select the Destination where you want to replicate Elasticsearch Data or Click on NEW DESTINATION to create a new Destination. Check out How to add Destination tutorial for the detailed walkthrough on steps needed for adding new Destination.
6. Pipeline Created
Your Pipeline will be created when you enter this page and you will have an option to see Sample Data and Map Schema.
While Hevo tries to load your schemas, you can select CONTINUE IN BACKGROUNDif it is taking too much time. Click on CREATE SCHEMA MAPPING to map Source and Destination Schemas, check out Introduction to Schema Mapper to learn about Schema Mapper or you can select DO IT LATER to directly head to Pipeline page. You can map schemas later on Schema Mapper page in your pipeline.
Please note that your data will not start replicating in Destination tables until you map source and Destination schemas.