Skip to content

05 · Observability at Scale (Activity Tracker, Sysdig)

A handful of resources can be watched by hand in the console. A ROKS cluster with a dozen microservices and an Event Streams pipeline behind it cannot. This module wires up IBM Cloud Monitoring (Sysdig-based) for metrics and alerting, Log Analysis for centralized logs, and revisits Activity Tracker from an operations (not just compliance) angle.

Provision the monitoring instance

ibmcloud resource service-instance-create monitoring-mastery \
  sysdig-monitor graduated-tier us-south --resource-group-name mastery-path

Service instance monitoring-mastery is being created.
OK

Wire a ROKS cluster to send metrics

ibmcloud ob monitoring config create \
  --instance monitoring-mastery \
  --cluster roks-mastery

Configuring cluster 'roks-mastery' to monitoring instance 'monitoring-mastery'...
sysdig-agent daemonset deployed to namespace ibm-observe
OK

Confirm the agent pods are actually running before assuming metrics are flowing — a common early mistake is checking the Sysdig dashboard for data that never arrives because the daemonset failed to schedule (often an SCC issue, following on from Module 1: the agent needs a privileged SCC grant to read host-level metrics):

oc get pods -n ibm-observe

NAME                     READY   STATUS    RESTARTS
sysdig-agent-4x2kq       1/1     Running   0
sysdig-agent-9dvbn       1/1     Running   0
sysdig-agent-p7z1w       1/1     Running   0

Log Analysis: centralize container and platform logs

ibmcloud resource service-instance-create logs-mastery \
  logdna 7-day us-south --resource-group-name mastery-path

ibmcloud ob logging config create \
  --instance logs-mastery \
  --cluster roks-mastery

OK
Logging agent deployed to namespace ibm-observe

Query recent logs from the CLI without opening the console, useful for scripted health checks:

ibmcloud logging tail --instance logs-mastery --filter "app:frontend AND level:error"

Alerting on a metric, not just dashboards

A dashboard nobody watches at 2 a.m. doesn't page anyone. Define an alert policy instead:

ibmcloud ob monitoring alert-create \
  --instance monitoring-mastery \
  --name high-error-rate \
  --description "Frontend 5xx rate over 5% for 5 minutes" \
  --severity high \
  --condition 'avg(sysdig_http_error_rate{kube_deployment_name="frontend"}) > 0.05' \
  --duration 300 \
  --notification-channel pagerduty:orders-oncall

Alert policy high-error-rate created.

Gotcha: alert conditions are metric-name-sensitive to the exact Sysdig integration in use (kube-state-metrics label names change between agent versions) — always confirm the metric name exists first with ibmcloud ob monitoring metrics --instance monitoring-mastery | grep http_error before wiring a condition that will silently never fire.

Distributed tracing across services

Metrics show that the frontend is slow; tracing shows which downstream call caused it. IBM Cloud doesn't ship a dedicated managed tracing product distinct from Sysdig's APM features — the common pattern is OpenTelemetry instrumentation in-app, exporting to the Sysdig-compatible OTLP endpoint:

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'https://ingest.us-south.monitoring.cloud.ibm.com/otlp/v1/traces',
    headers: { 'IBM-Instance-ID': process.env.SYSDIG_INSTANCE_GUID,
               'Authorization': `Bearer ${process.env.SYSDIG_ACCESS_KEY}` },
  }),
});
sdk.start();

Every service in the request path needs this — a trace with a gap because one service wasn't instrumented is nearly as useless as no trace at all.

Correlate: Activity Tracker for the "who changed the config" question

When a dashboard shows a sudden behavior change (not an error spike, a behavior change — e.g. response times permanently doubled), the first question is often "did someone change something," which is Activity Tracker's job, not the monitoring instance's:

ibmcloud atracker events --target <cos-audit-target-id> \
  --start "2026-08-25T00:00:00Z" --end "2026-08-26T00:00:00Z" \
  | jq '.[] | select(.action | contains("worker-pool"))'

Terraform for the monitoring config

resource "ibm_resource_instance" "monitoring" {
  name              = "monitoring-mastery"
  service           = "sysdig-monitor"
  plan              = "graduated-tier"
  location          = "us-south"
  resource_group_id = data.ibm_resource_group.mastery_path.id
}

resource "ibm_ob_monitoring" "roks_monitoring" {
  cluster     = ibm_container_vpc_cluster.roks.id
  instance_id = ibm_resource_instance.monitoring.guid
}

terraform validate
# Success! The configuration is valid.

Gotchas

  • Sysdig agent needs privileged SCC on ROKS — expect the Module 1 lesson about SCC denials to resurface the first time you enable monitoring on a cluster.
  • 7-day log retention (7-day plan) is genuinely 7 days — export or archive anything needed for a longer audit window to Cloud Object Storage before it ages out.
  • Alert notification channels must be configured separately (Slack, PagerDuty, webhook) before an alert policy referencing them will actually notify anyone — a policy referencing a channel that doesn't exist creates silently and never fires.
  • Cost scales with ingested volume, both for logs and for high- cardinality custom metrics (e.g. one metric series per unique user ID) — cardinality explosions are the most common surprise monitoring bill.

How It Actually Works

  • The Sysdig agent daemonset needs privileged SCC because it reads metrics from the kernel and container runtime directly, not from an application-level API. It hooks system calls (via a kernel module or eBPF probe) on the host to capture per-process CPU, memory, network, and file-descriptor activity for every container on that node — capabilities a restricted-SCC pod simply isn't allowed to request, which is why this is a case where granting a broader SCC to a specific, known workload is the correct move rather than a shortcut around Module 1's security posture.
  • An alert condition referencing a metric name that doesn't exist yet creates successfully because alert policies are stored and evaluated independently of whether the metric currently has any data behind it. The alerting engine periodically runs the stored expression against whatever time series matches the label selector; a typo'd or renamed-by-agent-upgrade metric name simply matches zero series forever, which evaluates to "no data," not "true" or "false" — so the alert silently never fires instead of erroring, which is exactly why confirming the metric name first is the only reliable check.
  • OpenTelemetry spans are correlated into one trace using a shared trace ID propagated through request headers, not through Sysdig inferring causality after the fact. The originating service generates a trace ID and a root span ID, injects them as traceparent headers on outbound calls, and every downstream service that's instrumented reads that header and creates its child spans under the same trace ID before exporting to the OTLP endpoint. A service that isn't instrumented simply never reads or forwards that header, which is why one gap in instrumentation doesn't corrupt the trace — it just leaves an unexplained gap in the timeline where that hop should be.
  • Cardinality cost comes from each unique combination of metric name + label values becoming its own stored time series — a metric tagged with user_id doesn't add rows to one series, it creates one entirely separate series per distinct user ID the monitoring backend has ever seen, each with its own storage and query overhead. That multiplicative structure (metric × every label's cardinality) is why a single high-cardinality label can silently multiply the ingested-series count by orders of magnitude compared to what the dashboard visually suggests.

Cheat sheet

Task Command
Create monitoring instance ibmcloud resource service-instance-create <n> sysdig-monitor graduated-tier <region>
Attach cluster to monitoring ibmcloud ob monitoring config create --instance <n> --cluster <c>
Attach cluster to logging ibmcloud ob logging config create --instance <n> --cluster <c>
Tail logs with filter ibmcloud logging tail --instance <n> --filter "<query>"
Create alert ibmcloud ob monitoring alert-create --instance <n> --condition '<expr>'
List available metrics ibmcloud ob monitoring metrics --instance <n>

Exercise

  1. Attach an existing ROKS cluster to a new monitoring instance and a new logging instance, and confirm the agent pods reach Running.
  2. Write an alert policy on a real metric name you confirmed exists via ibmcloud ob monitoring metrics.
  3. Instrument a small Node.js service with OpenTelemetry exporting to the Sysdig OTLP endpoint and generate one trace.
  4. Use ibmcloud atracker events to find and explain one configuration change in your account's recent history.