Instrument for Cloud Trace

This document provides a brief overview on how to instrument your application for Cloud Trace. For detailed instructions on setting up Cloud Trace, see the language-specific setup pages.

Cloud Trace provides distributed tracing data for your applications. After instrumenting your application, you can inspect latency data for a single request and view the aggregate latency for an entire application in the Cloud Trace console.

When to instrument your application

When trace data isn't automatically captured, you need to instrument your application to collect this data.

You can instrument your application so that it collects application-specific information. Several open-source instrumentation frameworks let you collect metrics, logs, and traces from your application and send that data to any vendor, including Google Cloud. To instrument your application, we recommend that you use a vendor-neutral instrumentation framework that is open source, such as OpenTelemetry, instead of vendor- and product-specific APIs or client libraries.

For information about instrumenting your applications by using vendor-neutral instrumentation frameworks, see Instrumentation and observability.

How to instrument applications

To instrument your applications to collect trace data, you can do any of the following:

When to create spans

The Cloud Trace client libraries typically maintain a global trace context that holds information about the current span, including its trace ID and whether the trace is sampled. These libraries usually create spans on RPC boundaries. However, you might need to create spans if the default creation algorithm isn't sufficient for your needs.

The current active span can be accessed by the global trace context, which is sometimes wrapped in a Tracer object. You can add information relevant to your application by using custom annotations and tags to existing spans, or you can create new child spans with their own annotations and tags to trace the application's behavior with finer granularity. Because the context is global, multi-threaded applications that update the context must use appropriate isolation.

When to provide authentication credentials

You generally don't need to provide authentication credentials to your application or specify your Google Cloud project ID in your application when you are running on Google Cloud. For some languages, you do need to specify your Google Cloud project ID even if you are running on Google Cloud. Also, if you use Autopilot mode for Google Kubernetes Engine, or if you enable Workload Identity Federation for GKE, then you must configure your application to use Workload Identity Federation for GKE.

If you are running outside of Google Cloud, you need to provide authentication credentials to your application. You also need to specify your Google Cloud project ID in your application.

For details, go to the language-specific setup pages.

How to force a request to be traced

Unless your application always samples every span, it isn't possible, in general, to force a request to be traced end-to-end because each component in an end-to-end request makes its own sampling decision. However, you can influence the decision by adding to the trace header a sampled flag, with this flag set to true. This setting is a hint to child components to sample the request. For more information about trace headers, see Protocols for context propagation.

For downstream components whose code you own, you must determine whether your instrumentation logic respects the sampled flag. For example, when using OpenTelemetry for instrumentation, you can use the ParentBased sampler to ensure that the parent sampled flag is respected.

Google Cloud services that record tracing information to Cloud Trace typically accept the parent sampling flag as a hint; however, most services also rate-limit sampling. Each Google Cloud service determines whether it supports tracing, how the parent sampling flag is utilized, and the rate limit on sampling.

How to correlate metric and trace data

You can correlate distribution-valued metric data with traces by attaching exemplars to the metric data points. Provided you complete the necessary configuration steps, OpenTelemetry, which is the recommended instrumentation library, automatically adds these exemplars. For more information, see Correlate metrics and traces by using exemplars.

Configure your project and platform

  1. Ensure that the Cloud Trace API is enabled.

    By default, Google Cloud projects have the Cloud Trace API enabled and you don't need to take any action. However, security constraints defined by your organization might have disabled the API. For troubleshooting information, see Develop applications in a constrained Google Cloud environment.

    Enable the Cloud Trace API.

    Enable the API

  2. Configure your platform.

    You can use Cloud Trace on Google Cloud and other platforms.

    • Google Cloud: When your application is running on Google Cloud, you don't need to provide authentication credentials in the form of a service account to the client library. However, must ensure that your Google Cloud platform has the Cloud Trace API access scope enabled.

      For the following configurations, the default access-scope settings include the Cloud Trace API access scope:

      If you use custom access scopes, then you must ensure that Cloud Trace API access scope is enabled. For example, if you use the Google Cloud CLI to create a GKE cluster and if you specify the --scopes flag, then ensure the scope includes trace.append. The following command illustrates setting the --scopes flag:

      gcloud container clusters create example-cluster-name --scopes=https://www.googleapis.com/auth/trace.append

    • Running locally and elsewhere: If your application is running outside of Google Cloud, then you must provide authentication credentials in the form of a service account to the client library. The service account must be granted the Cloud Trace agent (roles/cloudtrace.agent) role. For information about roles, see Control access with IAM.

      Google Cloud client libraries use Application Default Credentials (ADC) to find your application's credentials. You can provide these credentials in one of three ways:

      • Run gcloud auth application-default login

      • Place the service account in a default path for your operating system. The following lists the default paths for Windows and Linux:

        • Windows: %APPDATA%/gcloud/application_default_credentials.json

        • Linux: $HOME/.config/gcloud/application_default_credentials.json

      • Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path to your service account:

        Linux/macOS

          export GOOGLE_APPLICATION_CREDENTIALS=path-to-your-service-accounts-private-key

        Windows

          set GOOGLE_APPLICATION_CREDENTIALS=path-to-your-service-accounts-private-key

        PowerShell:

          $env:GOOGLE_APPLICATION_CREDENTIALS="path-to-your-service-accounts-private-key"

What's next

For detailed configuration information, samples, and links to GitHub and other open source repositories, go to the setup page for your language.