How do I use Filebeat and Logstash on Amazon Linux to connect to OpenSearch Service?

7 minute read
3

I'm using Filebeat and Logstash on Amazon Linux to connect to an Amazon OpenSearch Service cluster, but I receive an error.

Short description

To use Logstash to connect to Amazon OpenSearch Service, complete the following steps:

1.    Set up your security ports, such as port 443, to forward logs to OpenSearch Service.

2.    Update your Filebeat, Logstash, and OpenSearch Service configurations.

3.    Install Filebeat on your source Amazon Elastic Compute Cloud (Amazon EC2) instance. Make sure that you correctly install and configure your YAML config file.

4.    Install Logstash on a separate Amazon EC2 instance where the logs are sent from.
Note: If you didn't correctly set up or configure Logstash, then you receive a 401 Authorization error, 403 Forbidden error, or x-pack installation error.

Resolution

Set up your security ports

Set up your security ports to forward logs from Logstash:

1.    Create an EC2 instance where you installed Apache and Filebeat so that the instance can forward logs from Logstash to OpenSearch Service.

2.    Make sure that your EC2 instances are in the same security group as your virtual private cloud (VPC) for OpenSearch Service.

3.    Make sure that ports 80, 443, and 5044 are open in your security group so that you can send data between Logstash and OpenSearch Service.

Update your Filebeat, Logstash, and OpenSearch Service configurations

OpenSearch Service with legacy a Elasticsearch version runs best when you use the same Filebeat and Logstash versions. Use compatible versions for the following service versions:

  • Filebeat version x.x
  • Logstash version x.x (v7.16.2 and v6.8.22 for Log4j security patch)
  • OpenSearch Service with a legacy Elasticsearch version x.x

In the preceding example, the "x.x" of the version numbers must match your configurations.

Note:

  • If you're using OpenSearch Service 1.x with compatibility mode, then OpenSearch Service reports its version as 7.10. Check the compatibility matrix to select the correct version of Filebeat and Logstash. For more information, see Support Matrix on the Elasticsearch website.
  • If you're using OpenSearch Service versions 1.0 or later, then make sure that compatibility mode is turned on when you first launch your domain.

To make sure that the downloaded software remains in sync, download RPMs to each EC2 instance. To prevent a single point of failure in your pipeline, don't run Filebeat and Logstash on the same EC2 instance.

Install Filebeat on the source EC2 instance

1.    Download the RPM for the compatible Filebeat version:

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-oss-7.16.2-x86_64.rpm

2.    Install the Filebeat RPM file:

rpm -ivh filebeat-oss-7.16.2-x86_64.rpm

Install Logstash on a separate EC2 instance that the logs are sent from

1.    Download the RPM file for the compatible Logstash version:

wget https://artifacts.elastic.co/downloads/logstash/logstash-oss-7.16.2-x86_64.rpm

Note: This example uses version 7.16 of Filebeat and Logstash with OpenSearch Service version 1.1 with compatibility mode activated.

2.    Install Java or OpenJDK on your EC2 instance:

yum install java-1.8.0-*

3.    Run the rpm command to install the RPM file that you downloaded for Logstash:

rpm -ivh logstash-oss-7.16.2-x86_64.rpm

Note: Logstash requires Java to run. This example uses Java version 8 (Open JDK 1.8) that's supported by all versions of Logstash. For more information about the supported versions of Java and Logstash, see Support Matrix on the Elasticsearch website.

4.    Install the Logstash plugin for OpenSearch Service:

bin/logstash-plugin install logstash-output-opensearch

5.    Verify that the Logstash plugin for OpenSearch is installed:

bin/logstash-plugin list

Note: Logstash OSS version 7.12.1 supports both logstash-output-elasticsearch and logstash-output-opensearch plugins. Logstash OSS versions 7.13 and later support only the logstash-output-opensearch plugin.

6.    Check the /etc/filebeat and /etc/logstash directories to verify the configuration files.

7.    For Filebeat, update the output to either Logstash or OpenSearch Service (from the GitHub website). Specify to send logs, and then start your service.
Note: If you try to upload templates to OpenSearch Dashboards with Filebeat, then your upload fails. Filebeat assumes that your cluster has x-pack plugin support.

8.    Update your Filebeat YAML configuration file to send Apache access logs to Logstash.

Example YAML file:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
 
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
 
setup.template.settings:
 
index.number_of_shards: 1
index.codec: best_compression
 
#output.elasticsearch:
#hosts: ["vpc-examplestack-5crrfyysa2ratcl3ursmung33q.us-east-1.es.amazonaws.com:443"]
#protocol: "https"
 
output.logstash:
  # The Logstash hosts
  hosts: ["Logstash-EC2-InstanceIP:5044"]
 
setup.ilm.enabled: false 
ilm.enabled: false

9.    Make sure that your Logstash configuration file can access Filebeat on Port 5044. Port 5044 access allows Logstash to forward requests to your OpenSearch Service VPC endpoint.

Example Logstash file:

input {
  beats {
    port => 5044
  }
}

output {
  opensearch {
    hosts => ["https://domain-endpoint:443"]
    ssl => true
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    user => "my-username"
    password => "my-password"
  }
}

10.    Make sure that you can reach the OpenSearch Service endpoint from your Logstash instance:

curl -XGET https://vpc-examplestack-5crrfyysa2ratcl3ursmung33q.us-east-1.es.amazonaws.com

11.    (Optional) If you activated fine-grained access control (FGAC), then run the following command:

curl -u my-username:my-password -XGET https://vpc-examplestack-5crrfyysa2ratcl3ursmung33q.us-east-1.es.amazonaws.com

Note: If you have a timeout issue, then make sure that your Logstash EC2 instance is in the same security group as your VPC for OpenSearch Service. Also, make sure that ports 80, 443, and 5044 are open in your security group.

12.    Run the following commands to start the Filebeat and Logstash services on each instance:

Filebeat

systemctl start filebeat (service filebeat start)

Logstash

cp /etc/logstash/logstash.conf /etc/logstash/conf.d/
systemctl start logstash (service logstash start)

13.    Run a cat indices API call to your OpenSearch Service domain to confirm that the Filebeat logs are being sent. If your logs are successfully sent, then you receive the following response:

curl -XGET https://vpc-examplestack-5crrfyysa2ratcl3ursmung33q.us-east-1.es.amazonaws.com/_cat/indices
green open filebeat-7.16.2-2022.01.27 f97c4WnuQ-CtsAJJaJHUlg
1 1 1511515 0 249.7mb 124.7mb
green open .kibana_1                 Ioco6fUoSCGkaOvHNCL39g 1
1       1 0   7.4kb  
3.7kb

By default, the Filebeat indices rotate daily.

Example output of a Filebeat index:

curl -XGET https://vpc-examplestack-5crrfyysa2ratcl3ursmung33q.us-east-1.es.amazonaws.com/_cat/indices
green open filebeat-7.16.2-2022.01.27 f97c4WnuQ-CtsAJJaJHUlg
1 1 1511515 0 249.7mb 124.7mb
green open .kibana_1                 Ioco6fUoSCGkaOvHNCL39g 1
1       1 0   7.4kb  
3.7kb
green open filebeat-7.16.2-2022.01.28 4i8W0smlRGGFcQOaDMxonA
1 1      89 0 207.1kb 118.1kb

If you successfully configured Filebeat, Logstash, and OpenSearch Dashboards with Amazon EC2 Linux, then your pipeline looks like this:

Filebeat > Logstash > Amazon OpenSearch Service/Dashboards

401 Unauthorized error

A 401 Unauthorized error from Logstash indicates that your OpenSearch Service domain is protected by FGAC or Amazon Cognito. FGAC requires signed requests by a user or role that you must define in the domain's access policy. If you receive a 401 Unauthorized error, then turn on FGAC in your Logstash configuration file.

Example Logstash file:

output {
  opensearch {
    hosts => ["https://vpc-examplestack-5crrfyysa2ratcl3ursmung33q.us-east-1.es.amazonaws.com:443"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    ilm_enabled => false
    user => "my-username"
    password => "my-password"
  }
}

403 Forbidden error

When you use Logstash to send data to OpenSearch Service, you might receive a 403 Forbidden error. This error occurs when Logstash doesn't have the necessary permissions and fails to verify your AWS Identity Access Management (IAM) identity. To resolve this issue, use your IAM credentials to sign your requests to OpenSearch Service.

Use Logstash to sign OpenSearch Service requests:

1.    Install the Logstash plugin for OpenSearch Service:

bin/logstash-plugin install logstash-output-opensearch

2.    Attach an IAM role to the instance:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "es:ESHttp*"
      ],
      "Resource": "[Amazon-OpenSearch-Domain-ARN]"
    }
  ]
}

3.    Update your Logstash configuration settings to use the "opensearch" Logstash plugin as the output in your pipeline:

output {
  opensearch {
    hosts => ["domain-endpoint"]
    ssl => true
    region => "us-east-1"
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

Logstash x-pack installation error

If you get errors with x-pack when you start up Logstash, then manually turn off the x-pack plugin from your registry file.

Manually turn off the x-pack plugin:

1.    Open the following file:

/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb

2.    Find load_xpack and comment inline:

"load_xpack unless LogStash::OSS" >> "#load_xpack unless LogStash::OSS"

Note: You can check your configuration files to confirm that the Index Life Management (ILM) settings ilm.enabled and ilm_enabled are both set to false. Turning off these ILM settings in your configuration files eliminates startup errors for the x-pack plugin.

Related information

Loading data into Amazon OpenSearch Service with Logstash

AWS OFFICIAL
AWS OFFICIALUpdated 10 months ago