Transformations

Uptrace allows to transform the ingested data using a YAML-based language. You can use it to rename/delete attribute keys, parse/change attribute values, sample/drop spans, etc.

Example

For example, to rename an attribute:

name: Rename service to service_name
scope: [spans, logs, events, datapoints]
type: rename_attr
old_key: service
new_key: service_name

You can use the YAML above to create an operation by following these steps:

1. In the navigation menu on the left, click "Project" -> "Transformations".
2. Click "New Operation" -> "New operation from YAML".
3. Paste the YAML and click "Create".

Scope

Each operation must have a scope field that limits the operation to a particular data type (aka signal). The scope is an array and must contain one of the following values:

• spans
• logs
• events
• datapoints

Conditions

You can further narrow down the telemetry data to be transformed using if expression:

name: Rename service to service_name
scope: [spans, logs, events, datapoints]
type: rename_attr
old_key: service
new_key: service_name
if: attr("deployment_environment") == "prod"

Uptrace uses Expr language for writing expressions.

Execution

Uptrace executes operations one by one after normalising attribute names, i.e. service.name is normalised to service_name so you should use the normalised name.

Execution will stop if the signal is dropped by a drop operation, but otherwise Uptrace will execute all operations regardless of errors.

To change the execution order, use the priority field.

name: Rename service to service_name
scope: [spans, logs, events, datapoints]
priority: 100
type: rename_attr
old_key: service
new_key: service_name

Uptrace executes transformations before it processes the data so you can't use certain attributes set by Uptrace, for example, the _display_name column is not available and instead you should use spanName(), eventName(), logName(), and log_message.

For the same reason, you also can't use attributes extracted from structured logs and SQL queries, such as db_arg_* attributes.

Operations

Rename attribute

To rename attr from service to service_name:

name: Rename service to service_name
scope: [spans, logs, events]
type: rename_attr
old_key: service
new_key: service_name

Set overwrite: true to overwrite the target attribute if it already exists:

name: Rename service to service_name
scope: [spans, logs, events]
type: rename_attr
old_key: service
new_key: service_name
overwrite: true

Delete attributes

To delete attrs:

name: Delete foo and bar
scope: [spans, logs, events, datapoints]
type: delete_attrs
keys:
  - foo
  - bar

To delete attrs using a regular expression:

name: Delete foo and bar
scope: [spans, logs, events, datapoints]
type: delete_attrs
regexp: ^(foo|bar)$

To delete attrs in the specific metric:

name: Delete foo and bar
scope: [datapoints]
type: delete_attrs
keys:
  - foo
  - bar
if: metricName() == "my_metric_name"

Keep attributes

To keep some attributes and delete others:

name: Keep foo and bar
scope: [spans, logs, events, datapoints]
type: keep_attrs
keys:
  - foo
  - bar

To keep attributes matching a regular expression:

name: Keep foo and bar
scope: [spans, logs, events, datapoints]
type: keep_attrs
regexp: ^(foo|bar)$

Drop

To drop logs that match if expression:

name: Drop hello logs
scope: [logs]
type: drop
if: attr("log_message") contains "hello"

To drop spans that match if expression:

name: Drop Redis evalsha
scope: [spans]
type: drop
if: spanName() == "evalsha" && spanStatusCode() != "error"

The if condition is required for the drop operation.

Sample

To sample 50% of logs that match if expression:

name: Sample 50% hello logs
scope: [logs]
type: sample
fraction: 0.5
if: attr("log_message") startsWith "hello"

The fraction must be between 0 and 1. For spans, sampling is based on the trace ID so all spans in the same trace are either kept or dropped together. The if condition is optional — when omitted, the sampling applies to all signals in scope.

Change attribute type

To convert an attribute value to a different type:

name: Convert elapsed_ms to float
scope: [spans]
type: change_attr_type
key: elapsed_ms
type: float

Supported types: string, int, float, bool.

Flatten Map

By default, Uptrace does not index attributes that contain maps. For example, the following attribute cannot be queried using the object.nested.foo path:

{
  "object": {
    "nested": {
      "foo": "bar",
      "hello": "world"
    }
  }
}

To flatten such attributes into dot-separated top-level attributes, use the flatten_map operation:

name: Flatten the object attribute
scope: [spans]
type: flatten_map

# Map attributes to flatten.
maps: ['object']

# Map paths to include.
include_paths: ['object.nested.foo']
include_regexp: '^object.nested\.'

# Map paths to exclude.
exclude_paths: ['object.nested.hello']
exclude_regexp: 'secret'

If you don't specify any paths, the operation will index the entire map but will stop after 32 paths. This limit exists to keep the number of indexed attributes low and improve performance.

IP Geo Attributes

The ip_geo_attrs operation enriches IP address attributes with GeoIP data. For each specified key, it looks up the IP address and adds {key}_country_code, {key}_country_name, and {key}_city attributes.

name: Enrich custom IP with geo data
scope: [spans, logs, events]
type: ip_geo_attrs
keys:
  - custom_ip_address

Text Index

The text_index operation collects selected attribute values into a full-text index column, enabling fast text search across attribute values using the *:value search syntax.

name: Index request attributes
scope: [spans, logs, events]
type: text_index
include:
  - log_message
  - exception_message
  - http.request.**
exclude:
  - http.request.headers.**

The include field accepts glob patterns with . as the separator:

The exclude field uses the same glob patterns to remove specific paths from indexing.

For attributes of type map or array, the value is serialized as JSON. The search is case-insensitive and token-based, so substring matches are not supported. The maximum indexed text size per span/log is 4096 bytes.

Once configured, you can search indexed content using *:value in the search bar:

*:hello
*:error|warning
-*:debug

Script

Uptrace allows to write simple scripts that can modify telemetry data, for example:

name: Update status code if span has exception* attributes
scope: [spans]
type: script
if: hasAttr("exception_type") && hasAttr("exception_message")
then:
  - setSpanStatusCode("error")
  - setSpanStatusMessage(attr("exception_message"))
else:
  - setSpanStatusCode("ok")

The if condition is required for the script operation. The then and else lists contain expressions that are executed when the condition is true or false respectively.

To parse a string attribute value as a float:

name: Parse elapsed_ms
scope: [spans]
type: script
if: hasAttr("elapsed_ms")
then:
  - setAttr("elapsed_ms", parseFloat(attr("elapsed_ms")))

To reduce cardinality of an attribute:

name: Reduce http_target cardinality
scope: [spans]
type: script
if: metricName() startsWith "http_server_" && hasAttr("http_target")
then:
  - setAttr("http_target", replaceGlob(attr("http_target"), "/user/*/list/*", "/user/{userId}/list/{listId}"))

To replace all numbers in a string attribute using a regular expression:

name: Replace numbers in foo
scope: [spans]
type: script
if: hasAttr("foo")
then:
  - setAttr("foo", replaceAllRegexp(attr("foo"), "(\\d+)", "<number>"))

To extract all numbers from a string attribute using a regular expression:

name: Extract numbers from foo
scope: [spans]
type: script
if: hasAttr("foo")
then:
  - setAttr("foo", extractAllRegexp(attr("foo"), "(\\d+)", "$1"))

To convert a log into a span:

name: Convert logs with elapsed_ms to spans
scope: [logs]
type: script
if: hasAttr("elapsed_ms")
then:
  - setSpanName("operation-name")
  - setLogName("") # must zero the log name
  - setSpanDuration(parseFloat(attr("elapsed_ms")) * 1e6)
  - setSpanStatusCode("ok")

Functions reference

In addition to built-in functions provided by Expr, Uptrace supports the following functions. All functions can be used in if conditions and script expressions.

Span, log, and event functions

These functions are available when the scope includes spans, logs, or events.

Getter functions

Note that spanName() and eventName() return the original value as reported by OpenTelemetry, e.g. eventName() usually contains only a single word like message, log, etc.

Setter functions

Attribute functions

Type parsing functions

Datapoint functions

These functions are available when the scope includes datapoints.

Getter functions

Setter functions

Attribute functions

Common functions

These functions are available for all signal types.

Related links

• Expr Language Definition