Metrics, Logs and Traces: Knowing What Your System Is Doing

A note from the architecture desk to the team

Every few months someone on the team says “let’s just log it and build a dashboard from the logs.” It sounds pragmatic, and it’s how most of us started. But it is also one of the most expensive shortcuts in a production system. This article lays out how I want us to think about observability: what each signal is for, what it costs, and how we write the code so those decisions stay easy to review as the system evolves.

The short version: metrics tell you that something is wrong, traces tell you where, logs tell you why. Use each for the question it answers best.

Metrics vs logs: two different kinds of data

Metrics are numbers collected and aggregated over time. They map to a handful of instrument types:

• Counter: a value that only goes up (requests served, orders completed).
• Gauge / UpDownCounter: a value that moves both ways (queue length, active connections).
• Histogram: a distribution of values (request duration, database latency).

Metrics are highly structured. The only dimensions they carry are the tags (key-value pairs) we attach to them.

Logs are immutable, timestamped records of discrete events emitted by a running process. They can be free text or structured (JSON), and they carry rich context: messages, arguments, exception stack traces.

That difference in nature drives everything else.

	Metrics	Logs
What they are	Aggregated numbers	Individual events
Best at	Health, trends, capacity, alerting	Debugging, audit, forensics
Weakness	Cannot explain a single request	Expensive at volume
Cost driver	Tag cardinality	Volume, retention, I/O

The cost of each signal

This is the part that gets overlooked in design discussions.

Metrics are cheap because they are pre-aggregated inside the process. Incrementing a counter or updating a histogram bucket is a few nanoseconds, and what leaves the process is a small, fixed-size summary. If we get ten times the traffic, or a denial-of-service attack, the telemetry volume barely moves. The one thing that can hurt us is cardinality: the number of unique tag combinations. Put a user ID or a GUID in a tag and you create a new time series per value, which means memory exhaustion in the backend and a surprise bill.

Logs are expensive because every statement can trigger string formatting, serialization and physical I/O. Under load, synchronous logging directly degrades throughput and adds latency to the very transactions you are trying to observe. And since logs accumulate quickly, retention left unmanaged will fill disks or inflate cloud storage costs.

The key architectural point is that metrics scale with the number of series, logs scale with the number of events. Traffic grows faster than your tag set does.

What logs are actually for

Logs are not a performance tool. They are for four jobs:

1. Troubleshooting and root-cause debugging. Logs record the execution path, arguments and exceptions that answer “who did what, when, and why.” Across services we propagate a correlation ID so we can follow one failed request through the whole system.
2. Security monitoring and threat detection. SIEM systems correlate logs against baseline behavior to spot brute-force attempts, privilege elevation, or living-off-the-land techniques. After an incident, historical logs tell responders the scope and entry point of the compromise.
3. Auditing and compliance. Logs give us a record of administrative actions, configuration changes and sensitive data access. Frameworks like GDPR, HIPAA and PCI DSS expect this.
4. Operational visibility. INFO confirms that important business milestones happened (a payment was processed). WARN flags hazards before they become outages: retries, slow external APIs, resources approaching limits.

Notice what’s missing from that list: counting things and measuring durations at high frequency. That’s the metrics job.

How we track application performance

Our approach has three layers that work together.

1. Metrics for the quantitative picture

We organize our indicators around the Four Golden Signals:

• Latency: how long it takes to serve a request
• Traffic: how much demand the system is under
• Errors: the rate of failed requests
• Saturation: how full our resources are (CPU, memory, pools, queues)

For latency, always use histograms, not averages. Averages hide tail latency. We care about p95 and p99, because that’s where users feel the pain.

2. Distributed tracing for “where”

A metric tells you an endpoint is slow. A trace tells you where. Each request gets a Trace ID that propagates across service boundaries, and each operation creates a span with timing and context. Assembled, the spans show the full request flow and expose the bottleneck, whether that’s a slow query, a chatty downstream service or a serialization step.

3. Structured logs for “why”

We emit structured JSON logs enriched with the active Trace ID and Span ID. That lets you jump from a metric spike, to a trace, to the exact log lines emitted during that request. This correlation is what makes the three signals more than the sum of their parts.

Standardize on OpenTelemetry

We write vendor-neutral instrumentation with OpenTelemetry APIs and export over OTLP. Metrics are scraped into a time-series backend like Prometheus and visualized in Grafana; traces go to a backend like Jaeger. For quick local investigations, dotnet-counters shows live CPU, memory and custom counters without any infrastructure.

Can we derive metrics from logs?

Yes, log aggregation platforms can compute error counts per minute or latency percentiles from log attributes. It has real advantages: you can create a new metric retrospectively from historical logs without redeploying, and you can slice by rich context like specific error messages.

But the trade-offs are significant:

• Overhead. You pay the formatting, serialization and I/O cost for every event, just to count it.
• Cost. Indexing and querying high-volume logs to compute metrics on the fly is computationally heavy and drives storage cost up.
• No pre-aggregation. Native metric SDKs reduce data in memory before export. Log-derived metrics don’t, so telemetry volume spikes when traffic spikes, exactly when you least want it.

Our rule: native metrics for high-frequency health monitoring, dashboards and alerting. Logs for events, failures, audit and root-cause analysis. Log-derived metrics are fine for ad-hoc, retrospective questions, not as the foundation of our alerting.

Putting it into code

The design rules above only matter if the code makes them easy to follow. Here is how I’d like us to structure instrumentation in C#, consistent with how we already write code: small functions, side effects isolated in narrow methods, and composition through higher-order functions rather than a pile of cross-cutting framework magic.

Meters are static and defined once

Creating meters or instruments on a hot path allocates memory and hurts throughput. Define them once, as static fields (or singletons registered in DI).

using System.Diagnostics.Metrics;

public static class OrderTelemetry
{
    public const string MeterName = "Shop.Orders";

    private static readonly Meter Meter = new(MeterName, "1.0.0");

    public static readonly Counter<long> OrdersPlaced =
        Meter.CreateCounter<long>("orders.placed");

    public static readonly Histogram<double> PlacementDuration =
        Meter.CreateHistogram<double>("orders.placement.duration", unit: "ms");
}

Measurement as a higher-order function

The business logic should not know it is being measured. A small higher-order function wraps any operation and records its duration. The side effect (recording) lives in exactly one place, and it works with any Func<Task<T>>.

using System.Diagnostics;
using System.Diagnostics.Metrics;

public static class Measure
{
    public static async Task<T> TimedAsync<T>(
        Histogram<double> histogram,
        Func<Task<T>> operation,
        Func<T, string> classifyOutcome)
    {
        var start = Stopwatch.GetTimestamp();
        var outcome = "error";
        try
        {
            var result = await operation();
            outcome = classifyOutcome(result);
            return result;
        }
        finally
        {
            Record(histogram, start, outcome);
        }
    }

    // The only place where the side effect happens.
    private static void Record(Histogram<double> histogram, long start, string outcome) =>
        histogram.Record(
            Stopwatch.GetElapsedTime(start).TotalMilliseconds,
            new KeyValuePair<string, object?>("outcome", outcome));
}

Note the tag: outcome has a handful of possible values (success, rejected, error). That is bounded cardinality. An orderId tag would be a production incident waiting to happen.

Composing it with the use case is a one-liner at the composition root, with no base class or mediator pipeline involved:

Func<PlaceOrder, Task<OrderResult>> placeOrder = command =>
    Measure.TimedAsync(
        OrderTelemetry.PlacementDuration,
        () => handler.HandleAsync(command),
        result => result.IsSuccess ? "success" : "rejected");

Logs: structured, cheap, and where the IDs belong

High-cardinality identifiers like OrderId are perfectly fine in logs, because logs are per-event. Use the LoggerMessage source generator so we avoid allocation and string formatting on paths where the level is disabled.

public static partial class OrderLog
{
    [LoggerMessage(
        EventId = 1001,
        Level = LogLevel.Information,
        Message = "Order {OrderId} placed for customer {CustomerId}")]
    public static partial void OrderPlaced(
        this ILogger logger, Guid orderId, Guid customerId);

    [LoggerMessage(
        EventId = 1002,
        Level = LogLevel.Warning,
        Message = "Payment gateway slow for order {OrderId}: {ElapsedMs} ms")]
    public static partial void PaymentGatewaySlow(
        this ILogger logger, Guid orderId, double elapsedMs);
}

With OpenTelemetry’s logging integration, the active Trace ID and Span ID are attached automatically, which gives us the metric → trace → log pivot described above.

Observability is a behavior, so we test it

If an alert depends on a metric, that metric is a feature. We write the behavior test first, like everything else. MeterListener lets us observe measurements in-process without any backend.

using System.Diagnostics.Metrics;

public class MeasureTests
{
    [Fact]
    public async Task Records_duration_tagged_with_the_classified_outcome()
    {
        var outcomes = new List<string?>();

        using var listener = new MeterListener();
        listener.InstrumentPublished = (instrument, l) =>
        {
            if (instrument.Meter.Name == OrderTelemetry.MeterName)
                l.EnableMeasurementEvents(instrument);
        };
        listener.SetMeasurementEventCallback<double>((_, _, tags, _) =>
        {
            foreach (var tag in tags)
                if (tag.Key == "outcome")
                    outcomes.Add(tag.Value as string);
        });
        listener.Start();

        await Measure.TimedAsync(
            OrderTelemetry.PlacementDuration,
            () => Task.FromResult("ok"),
            _ => "success");

        Assert.Equal(new[] { "success" }, outcomes);
    }
}

Because measurement is isolated behind one small function, this single test covers the behavior for every use case that composes with it.

Rules of thumb for the team

1. Alerting and dashboards run on native metrics. Never on log queries.
2. Latency is a histogram. We look at p95 and p99, not the average.
3. Tags are bounded. No user IDs, GUIDs, raw URLs or query strings. If you need per-request context, use a log or a trace attribute. For linking a metric sample to a trace, use exemplars, never a Trace ID tag.
4. Instruments are created once. Static fields or DI singletons, never on a hot path.
5. Logs are structured and correlated. Every log carries the Trace ID, and cross-service flows carry a correlation ID.
6. Log levels mean something. INFO for business milestones, WARN for early warnings, ERROR for failures someone must look at.
7. No sensitive data in logs. Audit logs are valuable precisely because they are trustworthy, and privacy rules apply to them too.
8. Instrumentation is tested. If it feeds an alert, it has a behavior test.
9. Side effects stay small. Recording and writing live in narrow methods. Business logic stays pure and composable.

A note on revisiting these decisions

None of this is set in stone. Backends change, volumes change, and a rule that made sense at 100 requests per second may need rethinking at 10,000. Isolating telemetry behind small functions is what keeps those decisions cheap to revisit: swapping an exporter, changing a sampling strategy or tightening a tag set should touch one place, not fifty. If you find a case where these guidelines get in your way, bring it to the next architecture review. That is exactly what the review is for.

Summary

• Metrics answer “is something wrong, and how bad?” They are cheap, pre-aggregated and ideal for alerting. Watch cardinality.
• Traces answer “where is the time going?” They show the path of one request across services.
• Logs answer “what exactly happened, and why?” They are rich and indispensable for debugging, audit and security, but they cost real money and CPU at volume.
• Deriving metrics from logs works for retrospective questions. It is not a substitute for native metrics in the hot path.

Use the right signal for the right question, keep the side effects small, and test what you depend on.