How Do I Remove Fields Using Logstash Filters?

When transporting data from a source to your Logit stacks using Logstash, there may be fields you do not wish to retain or see in Kibana. You can remove these using the mutate filter plugin. There are several different ways of using this plugin to cover a wide range of use-cases, and so it is important to choose the right strategy depending on your situation.

If you always want to remove a field, or fields, from your data regardless of the situation, you can include the remove_field setting. This setting should point to an array of field names (or a single field name):

filter {    
       mutate {            
               remove_field => ["field_name"]    
       }
 }

If the array only contains one field name, you can omit the square brackets:

remove_field => "field_name"

If you want to remove more than one field you can supply additional field names to the array:

filter {    
    mutate {  
        remove_field => [ "%{@index}","%{@version}","%{@type}" ]   
    }
}

Conditional Removals

Often you only want to remove a field if a given condition is true. One of the most common use-cases for a conditional remove is to only remove a field if the field's value matches another value:

if [action] == "login" { 
   mutate {         
           remove_field => ["last_login"]     
           }
}

You can invert these conditions to remove a field if the condition is not true by using the not equals comparison operator (!=):

if [action] != "login"

To remove a field if the field's value makes up part of a larger string (i.e. a sub-string) you can use the special in keyword:

if [action] in "login" { 
   mutate {         
            remove_field => ["last_login"]     
     }
 }

You can also use the in keyword to check if a field's value matches at least one of many strings in an array:

if [action] in ["hello", "world", "foo"] { 
    mutate {         
        remove_field => ["field_name"]      
        }
 }

These types of conditions can be inverted using the not operator:

if [action] not in ["hello", "world", "foo"]

Removing by range

If your field's value is an integer, you can check if the value is inside a range of values. Before doing this, you must ensure that the value is interpreted as an integer by Logstash and not a number inside a string. To do this, use the convert plugin:

mutate {    convert => [ "field_name", "integer" ]} 

Then you will be able to use a compound statement to compare the value against a minimum and maximum value to check if the value is in range:

if [field_name] > 50 and [field_name] < 100 {
    mutate {        
         remove_field => [ "field_name" ]    
        }
 }

The above snippet will remove the field_name field if it's value is between 50 and 100.

Hosted ELK from Logit.io

While ELK (Elasticsearch, Logstash, and Kibana) offers powerful capabilities, many businesses discover that managing their Elastic Stack setup becomes increasingly complex, costly, and time-consuming as their data output scales exponentially. With our fully managed ELK platform, you can eliminate the need for training, managing, and maintaining your ELK Stack, streamlining your deployment and ensuring seamless scalability as your data grows.

Logit.io offers hundreds of regularly updated data source integrations for the essential software you rely on daily. Easily send log data from a wide range of tools and programming languages, including Microsoft Azure Event Hub, Ruby, Apache, .NET Core, Fluentd, Amazon CloudTrail, and Kubernetes, with some of these listed below. Also, our extensive documentation makes onboarding and navigating the platform painless, offering you a seamless transition for your organization.

If you’re interested in finding out more about Logit.io’s fully managed ELK platform then don’t hesitate to contact us or begin exploring the platform for yourself with a 14-day free trial.