What are index patterns and how can I use them to query Elasticsearch with Kibana on the Logit platform?
All logs and metrics that you send to Logit belong to an index pattern. To search your data with Kibana you have to select which Elasticsearch index or indices that you want to explore. You do this in Kibana by configuring index patterns.
An index pattern is a string with optional wildcards. It can therefore match the name of a single index or include wildcards (*) to match multiple indices.
All new Logit stacks come pre-configured with a number of useful index patterns to help you get started. To view your index patterns:
- Select "Management" from the left navigation
- Choose "Index Patterns"
All new Logit ELK stacks provide you with default indexes, including:
- *-* Default
The "filebeat-*" index pattern enables you to search all fields for any logs sent to Logit using the Filebeat shipper, this is an example of an index pattern matching on a single index.
Tip: When you access Kibana for the very first time the default index pattern is set to search log data from all indices being sent to Elasticsearch (a multiple indices match), the pattern is *-*.
Setting up a new index pattern
The "Create Index Pattern" button is found above the list of existing index patterns as shown below:
Select this and Kibana will display the list of indices for which logs are available. You will see that the "Next Step" button is disabled and will only become available when the specified index name that you have entered matches any indices.
When you are setting up a new index pattern, if your index contains one or more timestamp fields you will be asked to select one. This is the field that will be used to filter your data by time. If you do not wish to filter your search by timestamp you can select the "I don't want to use the Time Filter" option.
Tip: You can give fields a timestamp during mapping by using Index Templates, read more about Index Templates and Mappings.
Setting a default Index Pattern
If you want the new index pattern to be designated as your default pattern to load whenever you select the "Discover" tab then click the favourite star button after the index has finished being created. This is located in the top right-hand side of the screen.
Refreshing index patterns
If you add any further index mapping, Kibana automatically scans the indices that match each pattern to display a list of the new fields. It does not however, automatically pick them up. You can refresh the index pattern to pick up any newly-added fields by selecting the index pattern and then clicking the "refresh" icon in the top right-hand side of the screen.
When refreshing you will be prompted to reset the popularity counters for each field. Kibana keeps track of the fields that you've used the most often and the place where this data is stored is called a popularity counter. The data is used to sort fields within lists, by refreshing the index pattern this data and ability to sort will be reset.
Deleting index patterns
To delete an index pattern, select the index from the Index Patterns page and then click the "delete" icon in the top right-hand side of the screen (next to the default and refresh icons previously mentioned). You can recreate an index pattern again at any time in the future but you will also lose all visualisations, saved searches, and other saved objects that reference the pattern as well as all data in any popularity counters, so be careful!