Skip to content

Elasticsearch Index Mappings and Templates⚓︎

This video runs you through setting up a Elasticsearch index mapping and index template, using the dev tools console in Kibana. Using the Dev tools console is a simple and quick way to submit http requests to elasticsearch, as you will see in the video it provides auto complete on the URL endpoints and the json payload which is really handy. There is a far bit of typing in the video (and some spelling mistakes) so probably watch it at double speed.

The flow of the video and the commands executed are provided below, which might make them easier to be copied and pasted.

Background⚓︎

If you don't provide a index mapping elasticsearch will guess one for you. Generally the default mapping will meet most of your requirements. However there are a few good reasons to setup an index mapping, firstly its likely you can save a lot of storage space and it ensures that elasticsearch does not guess the wrong type (which could mean most of your data gets rejects) - this can happen when you have a text field that sometimes contains dates.

Default Mapping⚓︎

Lets see the default mapping applied by elasticsearch in action - to test this we just post a new document into a new index - like so; 

POST /transactions/_doc/1
{
    "amount": "6836.40",
    "direction": "in",
    "reference": "1031754877274",    
    "submittedDatetime": "26/01/2020",
    "processedDatetime" : "2020-01-26T22:02:35+10:00",
    "description": "Payment for invoice #123222"
}

To view the mapping generated we hit the index endpoint; 

GET transactions/

Reviewing the mapping we can see that it mostly created all the fields as multi type text and keyword. However what we want is; - Amount as a double. - Direction as a keyword as its a categorial value in our system. - Reference as a long. - submittedDatetime as a date. - Description as text as we want to support free text search but we don't need keyword for aggregrations.

Custom Mapping⚓︎

Note before we can create a custom mapping for an index we need to delete the existing index with the following command; 

DELETE transactions
The key lesson to remember is that mappings can not be changed once the index is created. You can add additional fields. However you can NOT change the type of an existing field. Now we know the types we want we can setup the index mapping using the below command;

PUT transactions/
{
  "mappings": {
    "properties": {
      "amount": {
        "type": "double"
      },
      "description": {
        "type": "text"
      },
      "direction": {
        "type": "keyword"
      },
      "processedDatetime": {
        "type": "date"
      },
      "reference": {
        "type": "long"
      },
      "submittedDatetime": {
        "type": "date",
        "format": "dd/MM/yyyy"
      }
    }
  }
}

Now we post in the original document again; 

POST /transactions/_doc/1
{
    "amount": "6836.40",
    "direction": "in",
    "reference": "1031754877274",    
    "submittedDatetime": "26/01/2020",
    "processedDatetime" : "2020-01-26T22:02:35+10:00​",
    "description": "Payment for invoice #123222​"
}

We then review the mapping to ensures our custom settings have been applied with the below command; 

GET transactions/
The response should confirm that the mappings have been applied successfully. It should look very similar to the below; 

{
  "transactions" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "amount" : {
          "type" : "double"
        },
        "description" : {
          "type" : "text"
        },
        "direction" : {
          "type" : "keyword"
        },
        "processedDatetime" : {
          "type" : "date"
        },
        "reference" : {
          "type" : "long"
        },
        "submittedDatetime" : {
          "type" : "date",
          "format" : "dd/MM/yyyy"
        }
      }
    },
    "settings" : {
      "index" : {
        "creation_date" : "1616210815184",
        "number_of_shards" : "1",
        "number_of_replicas" : "1",
        "uuid" : "lD8AhtBsRiOiAJ2cxN2dzw",
        "version" : {
          "created" : "7100299"
        },
        "provided_name" : "transactions"
      }
    }
  }
}

The application of the correct mappings can be further verified by running different query types to confirm the data types are being treated correctly. For example running a date range query on the summitted datetime field; 

GET /transactions/_search?q=submittedDatetime:[25/01/2020 TO 26/01/2020]

Index Templates⚓︎

Ok so our mapping is working - great. Best practice is to create an index template that sits in the background and waits for an index to be created with a specifc index pattern and applies the required template. Here is the command to create the index template for our index;

PUT _index_template/transaction_template
{
  "index_patterns": [
    "transaction*"
  ],
  "template": {
    "settings": {
      "number_of_shards": 2
    },
    "mappings": {
      "properties": {
        "amount": {
          "type": "double"
        },
        "description": {
          "type": "text"
        },
        "direction": {
          "type": "keyword"
        },
        "processedDatetime": {
          "type": "date"
        },
        "reference": {
          "type": "long"
        },
        "submittedDatetime": {
          "type": "date",
          "format": "dd/MM/yyyy"
        }
      }
    }
  }
}
To verify we delete the index again. 
DELETE transactions
Then post in our original document - elasticsearch will pickup the template based on the naming convension of the index. 
POST /transactions/_doc/1
{
    "amount": "6836.40",
    "direction": "in",
    "reference": "1031754877274",    
    "submittedDatetime": "26/01/2020",
    "processedDatetime" : "2020-01-26T22:02:35+10:00​",
    "description": "Payment for invoice #123222​"
}
We can then check the index template again with the below command. 
GET transactions

Happy days!