Specify ElasticSearch index template

[chschs]

I'm using fluentd with the in_syslog plugin and elasticsearch plugin to get syslog into elasticsearch, with a kibana frontend.

One of the problems I'm having though, is that the fields are indexed in elasticsearch so when I add a terms dashboard in kibana to give me, say, the top-10 hostnames, hostnames with dashes in them are broken up. so mysql-test-01 would come across as three hostnames: mysql, test, and 01.

Logstash got around this issue by making a "raw" version of several fields that is set to not-analyzed upon creation, so that you can run your dashboards against that instead.

More information here: http://www.elasticsearch.org/blog/logstash-1-3-1-released/

With syslog messages going into ES with this plugin, I'm finding that I'd like to have a "raw" or non-analyzed host (hostname) field and ident field (gives me the application). Unfortunately right now both of those fields are analyzed and it's messing with our dashboards.

[lxfontes]

Hey @chschs, have you tried adding a mapping template to change the index settings ?

example

{
  "mappings": {
    "_default_": {
      "_all": { "enabled": false }, 
      "_source": { "compress": true },
      "properties" : {
        "event_data": { "type": "object", "store": "no" },
        "@fields": { "type": "object", "dynamic": true, "path": "full" }, 
        "@message": { "type": "string", "index": "analyzed" },
        "@source": { "type": "string", "index": "not_analyzed" },
        "@source_host": { "type": "string", "index": "not_analyzed" },
        "@source_path": { "type": "string", "index": "not_analyzed" },
        "@tags": { "type": "string", "index": "not_analyzed" },
        "@timestamp": { "type": "date", "index": "not_analyzed" },
        "@type": { "type": "string", "index": "not_analyzed" }    
      }   
    }
  },
  "settings": {
    "index.cache.field.type" : "soft",
    "index.refresh_interval": "5s",
    "index.store.compress.stored": true,
    "index.number_of_shards": "3", 
    "index.query.default_field": "querystring", 
    "index.routing.allocation.total_shards_per_node": "2"
  }, 
  "template": "logstash-*"
}

This will be used every time a new index with the 'logstash-*' pattern is created

[ajturner]

+1 to make this part of this plugin. While we can manually modify the mapping, why require that overhead in an application to update this when Fluent-logstash is already creating the original mapping.

[ajturner]

Looking at the code - Fluent is not actually creating or modifying the index and merely writing to the current index. This would have to detect that a new index is being created and subsequently call to update the mapping.

Now I understand what you mean by indices templates. Perhaps worth adding this to the Readme

[pitr]

@ajturner good point, added 139184e

[openfirmware]

I was able to use a custom index based off the one logstash uses to auto-generate .raw versions of fields.

I started by deleting the current day's index ($ curl -XDELETE localhost:9200/logstash-2014.09.02) then using that curl PUT command to set the defaults for the index. I then restarted fluentd and the raw fields were available. You can check if the settings are sticking in ElasticSearch:

$ curl localhost:9200/logstash-2014.09.02/_mapping?pretty

[stanhu]

+1 for making this built into the plugin.

This is the template used by logstash:

https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template.json

[aleks-v-k]

There is a valuable reason to make index template support built into the plugin: in a containerized environment we do not know when elasticsearch container will start and we have to PUT index template to it before fluentd will send data. There are some possible workarounds but every of them looks really ugly. So +1 for making built in support of index templates. @pitr
Since version 2.x elasticsearch supports index templates creation only via API

[ssergiienko]

+1 for making built in support of index templates.

[pitr]

To implement this behaviour, this gem would need to do the following additional work before writing a record to elasticsearch:

1. Check if the index exists
2. If not, create the index and the mapping

Seems like that might impact the performance. What do you think @aleks-v-k @ssergiienko? What about deleting old indices?

Can you provide more information about how you run ElasticSearch in "containerized environment"? Is this something Cloudlinux is working on (I'm not familiar with their offerings)?

[stanhu]

The gem would only need to write the index templates once at startup since a wildcard match could be used. As seen in the Logstash template, just adding something like logstash-* should take care of things.

[pitr]

Hmm, sounds like a good idea then