OpenTelemetry Collector

The OpenTelemetry Collector enables you to quickly and securely ship your data according to your specific configuration requirements and facilitates the forwarding of logs, metrics, and trace data directly to Logit.io's observability platform.

Setting up and operating the OpenTelemetry Collector offers multiple benefits, such as:

Simplified setup: With Logit.io’s pre-configured OTEL stack config, you can streamline your data transmission setup.

Comprehensive data collection: This tool gathers logs, metrics, and tracing information, offering an all-encompassing platform for data observability to enhance monitoring and optimization efforts.

Understand how to instrument your application with OpenTelemetry with this official guide Getting Started for Developers or continue and use the Demo App in the next step.

The Demo App generates logs, traces and metrics for a sample application in cases where you don't want to instrument your application at this stage.

Install Docker if you don't already have it.

Clone the Demo repository

git clone https://github.com/open-telemetry/opentelemetry-demo.git

Change to the Demo folder

cd opentelemetry-demo/

The config below will be pre-configured if you have launched this source integration from your stack dashboard.

If not, make sure to replace the following placeholders with the correct details: <your-opentelemetry-username>, <your-opentelemetry-password>,<your-elasticsearch-username>, <your-elasticsearch-password>, <your-log-stack-id>, <your-metrics-username>, <your-metrics-password>, <your-metrics-stack-id>, <your-opentelemetry-endpoint-address> and <your-opentelemetry-endpoint-port>.

Copy the configuration file below (making the above changes if necessary) and overwrite the contents of otelcol-config.yml (this file can be found in the otelcollector folder where you installed the OpenTelemetry Collector demo application e.g. opentelemetry-demo/src/otelcollector/otelcol-config.yml).

extensions:
  basicauth/apm:
    client_auth:
      username: <your-opentelemetry-username>
      password: <your-opentelemetry-password>
  basicauth/logs:
    client_auth:
      username: <your-elasticsearch-username>
      password: <your-elasticsearch-password>

receivers:
  otlp:
    protocols:
      http:
        cors:
          allowed_origins:
            - "http://*"
            - "https://*"
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  opensearch:
    logs_index: otel-logs
    http:
      endpoint: "https://<your-log-stack-id>-es.logit.io"
      auth:
        authenticator: basicauth/logs
  prometheusremotewrite:
    endpoint: "https://<your-metrics-username>:<your-metrics-password>@<your-metrics-stack-id>-vm.logit.io:0/api/v1/write"
    resource_to_telemetry_conversion:
      enabled: true # Convert resource attributes to metric labels
  otlp:
    auth:
      authenticator: basicauth/apm
      # use the grpc port
    endpoint: <your-opentelemetry-endpoint-address>:<your-opentelemetry-endpoint-port>
  logging:
    verbosity: detailed

service:
  extensions: [basicauth/apm, basicauth/logs]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      exporters: [opensearch]
    metrics:
      receivers: [otlp]
      exporters: [prometheusremotewrite]

If you are using the Demo app, you can start it by running the following command in a Terminal (make sure you run it in the opentelemetry-demo folder):

docker compose up --force-recreate --remove-orphans --detach

You can now view the Demo App: http://localhost:8080/ and Logs, Metrics and Traces are being sent to your stacks.

Data should now have been sent to your Stack.

View my data

If you don't see logs take a look at How to diagnose no data in Stack below for how to diagnose common issues.

If you don't see data appearing in your Stack after following the steps, visit the Help Centre guide for steps to diagnose no data appearing in your Stack or Chat to support now.

The OpenTelemetry Collector, renowned for its efficiency and streamlined memory footprint, is becoming the preferred choice for funneling logs, metrics, and tracing data into observability platforms. As a pivotal component of the comprehensive suite of open-source telemetry tools, it builds upon the foundational features of predecessors, crafted in the efficient Go programming language. The Collector excels in ingesting, processing, tailing, and exporting telemetry data, positioning itself as a versatile and essential tool in the telemetry data pipeline.

Originating from the integration of best practices and core functionalities from earlier telemetry tools, the OpenTelemetry Collector is engineered to offer flexibility and robustness in handling telemetry data. Unlike its early versions, which were limited in scope and destination options for telemetry data, recent iterations have expanded its capabilities, enabling seamless integration with a variety of data processing and storage systems, including but not limited to Redis and Kafka.

Despite the Collector's adaptability and comprehensive feature set, an improperly configured OpenTelemetry setup can lead to intricate issues in telemetry data management. These complications might manifest as unwieldy registry files or errors in file handling when logs are deleted or renamed, posing significant challenges in maintaining a clean and efficient logging environment. Additionally, the task of monitoring and managing multiple telemetry pipelines can be daunting for those operating their own telemetry infrastructure, prompting some to explore managed observability solutions as an effective way to alleviate these burdens.

If you need any further assistance with migrating your log, metrics and tracing data to Logit.io we're here to help you get started. Feel free to get in contact with our support team by sending us a message via live chat & we'll be happy to assist.