Deep-dive on the Next Gen Platform. Join the Webinar!

Skip Navigation
Show nav
Dev Center
  • Get Started
  • Documentation
  • Changelog
  • Search
  • Get Started
    • Node.js
    • Ruby on Rails
    • Ruby
    • Python
    • Java
    • PHP
    • Go
    • Scala
    • Clojure
    • .NET
  • Documentation
  • Changelog
  • More
    Additional Resources
    • Home
    • Elements
    • Products
    • Pricing
    • Careers
    • Help
    • Status
    • Events
    • Podcasts
    • Compliance Center
    Heroku Blog

    Heroku Blog

    Find out what's new with Heroku on our blog.

    Visit Blog
  • Log inorSign up
Hide categories

Categories

  • Heroku Architecture
    • Compute (Dynos)
      • Dyno Management
      • Dyno Concepts
      • Dyno Behavior
      • Dyno Reference
      • Dyno Troubleshooting
    • Stacks (operating system images)
    • Networking & DNS
    • Platform Policies
    • Platform Principles
  • Developer Tools
    • Command Line
    • Heroku VS Code Extension
  • Deployment
    • Deploying with Git
    • Deploying with Docker
    • Deployment Integrations
  • Continuous Delivery & Integration (Heroku Flow)
    • Continuous Integration
  • Language Support
    • Node.js
      • Working with Node.js
      • Troubleshooting Node.js Apps
      • Node.js Behavior in Heroku
    • Ruby
      • Rails Support
      • Working with Bundler
      • Working with Ruby
      • Ruby Behavior in Heroku
      • Troubleshooting Ruby Apps
    • Python
      • Working with Python
      • Background Jobs in Python
      • Python Behavior in Heroku
      • Working with Django
    • Java
      • Java Behavior in Heroku
      • Working with Java
      • Working with Maven
      • Working with Spring Boot
      • Troubleshooting Java Apps
    • PHP
      • PHP Behavior in Heroku
      • Working with PHP
    • Go
      • Go Dependency Management
    • Scala
    • Clojure
    • .NET
      • Working with .NET
  • Databases & Data Management
    • Heroku Postgres
      • Postgres Basics
      • Postgres Getting Started
      • Postgres Performance
      • Postgres Data Transfer & Preservation
      • Postgres Availability
      • Postgres Special Topics
      • Migrating to Heroku Postgres
    • Heroku Key-Value Store
    • Apache Kafka on Heroku
    • Other Data Stores
  • AI
    • Working with AI
  • Monitoring & Metrics
    • Logging
  • App Performance
  • Add-ons
    • All Add-ons
  • Collaboration
  • Security
    • App Security
    • Identities & Authentication
      • Single Sign-on (SSO)
    • Private Spaces
      • Infrastructure Networking
    • Compliance
  • Heroku Enterprise
    • Enterprise Accounts
    • Enterprise Teams
    • Heroku Connect (Salesforce sync)
      • Heroku Connect Administration
      • Heroku Connect Reference
      • Heroku Connect Troubleshooting
  • Patterns & Best Practices
  • Extending Heroku
    • Platform API
    • App Webhooks
    • Heroku Labs
    • Building Add-ons
      • Add-on Development Tasks
      • Add-on APIs
      • Add-on Guidelines & Requirements
    • Building CLI Plugins
    • Developing Buildpacks
    • Dev Center
  • Accounts & Billing
  • Troubleshooting & Support
  • Integrating with Salesforce
  • Monitoring & Metrics
  • OpenTelemetry Concepts and Heroku

OpenTelemetry Concepts and Heroku

Last updated December 04, 2024

Table of Contents

  • Understanding Telemetry and Observability
  • Understanding OpenTelemetry
  • Heroku Fir and OpenTelemetry
  • OpenTelemetry Protocol Specification
  • Additional Reading

Telemetry and observability are crucial for maintaining the health and performance of applications. OpenTelemetry (OTel) has emerged as a standard for collecting telemetry data.

Fir, Heroku’s next generation, leverages OpenTelemetry to provide robust observability features, supporting the OpenTelemetry Protocol (OTLP) for seamless exporting of trace, metric, and log signals.

This article explores the basics of OpenTelemetry concepts within the context of the Fir generation of our platform.

Understanding Telemetry and Observability

Telemetry refers to tools that generate and distribute logs, metrics, and traces across a system’s architecture in a standardized way. An example of such a tool is OpenTelemetry.

Observability refers to services to collect and present ad hoc telemetry data for deeper insights into application performance and debugging.

Understanding OpenTelemetry

OpenTelemetry is an open-standard framework that provides a standardized way to collect and export telemetry data from applications. It supports three primary signals:

  • Traces: Record the execution path of requests through a system. These help in understanding the flow of requests and diagnosing latency issues.
  • Metrics: Provide quantitative measurements of system behavior captured at runtime. Metrics offer insights into system performance and resource utilization.
  • Logs: Capture discrete events that happen over time. This signal type provides detailed context for events, aiding in debugging and auditing.

These signals are correlated via context propagation, which allows telemetry information to build over distributed systems.

Heroku Fir and OpenTelemetry

We designed the Fir generation of the Heroku platform to simplify application observability by integrating OpenTelemetry directly into its core. It offers developers a streamlined way to collect and export telemetry data without extensive configuration.

Fir fully supports OpenTelemetry’s trace, metric, and log signals over the OTLP protocol. By adopting OTLP, Heroku ensures compatibility and interoperability with various observability backends and tools. This deep integration allows for automatic instrumentation of applications without additional overhead. Furthermore, context propagation works out of the box.

Heroku Telemetry Drains

You can configure Heroku Telemetry Drains for an application or space to send telemetry to observability provider destinations.

Drains act as multiplexers that redirect telemetry signals to their appropriate observability platform. They enable flexible routing of data based on defined rules and conditions.

By configuring drains, it’s possible to send traces to one backend and metrics to another. You can configure drains at an application or space level.

Here’s a visual showing a drain configured for an app or a space that automatically captures traces, metrics and logs for the entire Heroku ecosystem. These include router logs, runtime events, release events and dyno metrics.

Heroku Telemetry Drains Architecture

Traditional Heroku Application Log Collection

Heroku pioneered the idea that an application should only be concerned with streaming its logs to stdout. One of the key features of our integration with OTel is its ability to automatically convert standard output (stdout) and standard error (stderr) logs into OTLP logs. We then route these logs to the OTel log stream, and export them to all the configured telemetry destinations, like the Heroku CLI or observability providers configured as drains.

Heroku Telemetry Collection

While Heroku infrastructure and application logs are converted to OTel logs, other OTel signals are also collected and exported from a Fir space. These signals include metrics on the health and usage of a dyno, routing traces and runtime OTel logs.

Spaces and apps configured with a drain automatically have these signals routed to the observability provider configured.

Application OTLP Telemetry Signal Collection

In addition to the telemetry generated by the Heroku ecosystem, it’s possible for an application to generate business-specific OTLP traces, metrics and logs. For example, you can capture a trace of a user’s product search journey, or the metrics associated with the time to purchase.

OpenTelemetry provides flexibility with a variety of language-specific SDKs. After configuring a trace and/or metrics provider to export OTLP telemetry signals, the rest of our tooling collects these signals and exports them to your configured drains in batches.

There are two types of OLTP transport protocols, gRPC and HTTP, which we discuss more later in this article. All OTLP signals exported from an application to the default OTLP ports on localhost: port 4317 for gRPC and 4318 for HTTP. Heroku telemetry services act as intermediaries, collecting and forwarding the telemetry data to a central OpenTelemetry Collector.

Data Residency

Telemetry generated by your apps and by Heroku’s infrastructure stay within the Fir Space itself. For example, if you deploy an application into a Fir space within the Tokyo region, all telemetry data generated by your application and the Heroku platform stays inside the Tokyo region.

OpenTelemetry Protocol Specification

The OpenTelemetry Protocol (OTLP) specification describes the encoding, transport, and delivery mechanism of telemetry data between telemetry sources, intermediate nodes such as collectors and telemetry backends.

Heroku’s Fir platform natively supports both OLTP transport mechanisms, gRPC and HTTP. Continue reading to learn about the differences between gRPC and HTTP or see the official OpenTelemetry OTLP protocol documentation.

In all cases, we recommend using the default protocol for your language, SDK, and observability provider.

gRPC

  • Performance and Efficiency: gRPC uses protocol buffers (Protobuf) over HTTP/2 for data serialization, which is more efficient than JSON over HTTP. This efficiency results in lower CPU usage and reduced network overhead.
  • Bi-directional Streaming: gRPC supports full-duplex streaming, allowing the client and server to send multiple messages independently in both directions. This streaming is beneficial for high-throughput and real-time applications.
  • Built-in Flow Control: gRPC provides built-in flow control mechanisms, which help manage backpressure and prevent overwhelming the server or client.
  • Compression Support: gRPC natively supports message compression, reducing the amount of data transmitted over the network.
  • Native Transport Protocol: OTLP over gRPC is the native transport protocol.

HTTP

  • Wider Compatibility: HTTP/1.1 is universally supported, ensuring better compatibility with existing network infrastructure and intermediaries.
  • Ease of Integration: Sending data over HTTP can be simpler, especially in environments where HTTP libraries are readily available and familiar.
  • Firewall and Proxy Friendly: HTTP traffic is commonly allowed through firewalls and proxies, reducing the likelihood of connectivity issues.
  • Easier Debugging: HTTP requests and responses can be easily inspected using standard tools like cURL, Postman, or browser developer consoles.

Additional Reading

  • Heroku Telemetry Drains
  • Working with Heroku Telemetry Drains

Keep reading

  • Monitoring & Metrics

Feedback

Log in to submit feedback.

Working with Heroku Telemetry Drains Production Check

Information & Support

  • Getting Started
  • Documentation
  • Changelog
  • Compliance Center
  • Training & Education
  • Blog
  • Support Channels
  • Status

Language Reference

  • Node.js
  • Ruby
  • Java
  • PHP
  • Python
  • Go
  • Scala
  • Clojure
  • .NET

Other Resources

  • Careers
  • Elements
  • Products
  • Pricing
  • RSS
    • Dev Center Articles
    • Dev Center Changelog
    • Heroku Blog
    • Heroku News Blog
    • Heroku Engineering Blog
  • Twitter
    • Dev Center Articles
    • Dev Center Changelog
    • Heroku
    • Heroku Status
  • Github
  • LinkedIn
  • heroku.com
  • Terms of Service
  • Privacy (日本語)
  • Cookies
  • Cookie Preferences
  • Your Privacy Choices
  • © 2025 Salesforce.com