Zero-Touch Observability: How to Debug Production .NET Microservices on AKS with Dynatrace
Struggling with transient 500s, slow Entity Framework queries, or mysterious ExitCode 137 OOM kills in your Kubernetes cluster? Here is an end-to-end guide to deep-tier .NET diagnostics on Azure Kubernetes Service (AKS) using Dynatrace.
Running .NET applications on Azure Kubernetes Service (AKS) provides elastic scalability, rapid deployment pipelines, and high infrastructure density. However, distributed containerized environments introduce a notoriously difficult problem: observability blind spots.
When an API call spikes from 40ms to 4 seconds, or a pod crashes intermittently during peak traffic, traditional logs and basic Kubernetes metrics (kubectl top pods) often leave engineers guessing. Is it Garbage Collection (GC) pressure? A missing database index? A connection pool starvation issue? Or an unhandled background thread exception?
In this post, we’ll walk through a production-ready solution design that integrates Dynatrace with AKS to instrument and debug .NET workloads—without code changes or deployment disruption.
1. The Architecture at a Glance
Rather than requiring developers to manually import NuGet packages, inject OpenTelemetry SDKs, and configure exporters across dozens of microservices, we utilize the Dynatrace Operator.
The Operator uses a Kubernetes Mutating Admission Webhook to inject Dynatrace OneAgent directly into the pod lifecycle.
Plaintext
┌───────────────────────────────┐
│ Dynatrace Platform │
│ (PurePath, Davis AI, Profiler)│
└──────────────▲────────────────┘
│ HTTPS (443)
┌────────────────────────────────────────────────┼────────────────┐
│ Azure Kubernetes Service (AKS) │ │
│ │ │
│ ┌──────────────────────────────┐ │ │
│ │ dynatrace Namespace │ │ │
│ │ • Dynatrace Operator │ │ │
│ │ • ActiveGate (Proxy/Cache) ├──────────────┘ │
│ └──────────────┬───────────────┘ │
│ │ Intercepts Pod Spec Creation │
│ ▼ │
│ ┌───────────────────────────────────────────────────────────┐ │
│ │ Production Namespace │ │
│ │ │ │
│ │ ┌───────────────────────────────────────────────────┐ │ │
│ │ │ .NET Workload Pod │ │ │
│ │ │ │ │ │
│ │ │ [OneAgent Init Container] │ │ │
│ │ │ └─► Injects Profiler Binaries │ │ │
│ │ │ │ │ │
│ │ │ [.NET App Container] │ │ │
│ │ │ • CORECLR_ENABLE_PROFILING=1 │ │ │
│ │ │ • Hooks into .NET CoreCLR Runtime Profiling │ │ │
│ │ │ • Live Traces, GC Heap, CPU, Thread Pools │ │ │
│ │ └───────────────────────────────────────────────────┘ │ │
│ └───────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
How the Magic Happens Under the Hood:
Admission Webhook Interception: When a deployment is scheduled in an instrumented namespace, the Dynatrace Operator injects an initContainer and mounts a shared emptyDir volume into your .NET pod.
CLR Profiling API Hooks: The environment variables CORECLR_ENABLE_PROFILING=1 and CORECLR_PROFILER are automatically injected.
Deep Instrumentation: The CoreCLR profiler hooks directly into the runtime engine, intercepting JIT compilation, method invocations, exceptions, thread allocations, and GC cycles in real time.
2. Deploying to AKS in 3 Steps
Step 1: Install the Dynatrace Operator
Install the operator into your AKS cluster via Helm:
Bash
helm repo add dynatrace https://raw.githubusercontent.com/Dynatrace/dynatrace-operator/master/config/helm/repos/stable
helm repo update
helm install dynatrace-operator dynatrace/dynatrace-operator \
--namespace dynatrace \
--create-namespace \
--set "installCRD=true"
Step 2: Configure the DynaKube Custom Resource
Generate an API token with Read/Write configuration and Data ingest permissions in your Dynatrace tenant, store it in a Kubernetes secret, and apply the DynaKube manifest:
YAML
# dynakube.yaml
apiVersion: dynatrace.com/v1beta1
kind: DynaKube
metadata:
name: aks-production-observability
namespace: dynatrace
spec:
apiUrl: "https://<your-tenant-id>.live.dynatrace.com/api"
tokens: "dynakube-tokens"
# Deploy OneAgent as a host DaemonSet with container injection
oneAgent:
cloudNativeFullStack: {}
# ActiveGate manages routing, TLS termination, and Kubernetes metadata
activeGate:
capabilities:
- routing
- kubernetes-monitoring
Bash
kubectl apply -f dynakube.yaml
Step 3: Enable Auto-Injection on Your Namespaces
Simply label the namespace where your .NET applications reside. Once labeled, trigger a rolling restart:
Bash
# Enable injection
kubectl label namespace ecommerce-api oneagent=true
# Restart existing workloads to pick up the injection webhook
kubectl rollout restart deployment/order-service -n ecommerce-api
3. Real-World Debugging Scenarios
Once the workloads are instrumented, you gain access to diagnostics that are virtually impossible to obtain with conventional logging alone.
Scenario A: Uncovering the "Hidden" Latency in Microservices (PurePath 4.0)
The Symptom: An API gateway returns high response times on /api/v1/checkout, but average CPU and memory consumption appear normal.
The Diagnostic: Dynatrace PurePath traces the request across process and container boundaries. In a single pane, you can see the thread transition into asynchronous execution (Task.WhenAll), drill straight into an Entity Framework Core query, and see the exact SQL statement, parameters, and number of round-trips causing the bottleneck (such as an inadvertent N+1 query pattern).
Scenario B: Diagnosing Silent OOM Kills (ExitCode 137)
The Symptom: AKS repeatedly restarts pods with OOMKilled. Kubernetes logs end abruptly with no stack trace.
The Diagnostic: By default, .NET Server Garbage Collection allocates memory aggressively, assuming it has the entire host available. If container memory limits in your pod spec are strict, the Linux kernel kills the container before the .NET runtime decides to trigger a full Gen 2 collection.
The Solution: Dynatrace correlates pod memory limits against .NET GC Heap metrics (Gen 0, Gen 1, Gen 2, and Large Object Heap / LOH). You can immediately verify whether the issue stems from unmanaged memory leaks, LOH fragmentation from large payload serializations, or misconfigured GC settings (DOTNET_GCHeapHardLimitPercent).
Scenario C: Debugging Live Microservices with Non-Breaking Breakpoints
The Symptom: A bug occurs only when certain tenant IDs submit a transaction under concurrent load. You cannot attach a live Visual Studio remote debugger to production pods without pausing the threads and causing timeout cascades.
The Diagnostic: Using Dynatrace Live Debugger, developers can define non-breaking breakpoints directly from Visual Studio Code or JetBrains Rider. When the execution path hits the line of code, Dynatrace captures a snapshot of local variables, call stacks, and context payloads without halting the execution thread.
4. Production Hardening & Best Practices
Upload Portable PDBs:
Ensure your CI/CD pipeline builds portable PDBs and pushes them into your Dynatrace tenant symbol store. This ensures your stack traces display exact file paths and source line numbers instead of generic IL offset pointers (MyService.ExecuteAsync+d__4.MoveNext()).
Adjust Kubernetes Probes:
The initial CoreCLR profiler attachment adds a brief initialization overhead to the cold start of a .NET runtime. Always verify that your livenessProbe and readinessProbe definitions have a reasonable initialDelaySeconds (e.g., 15–30 seconds) to prevent Kubernetes from killing the pod prematurely during startup.
Enforce Data Privacy Rules:
Enable automatic masking on captured HTTP request parameters and variable capture to ensure PII (Personally Identifiable Information), auth tokens, or PCI data never exit your AKS cluster.
Summary
Debugging containerized .NET applications no longer requires sprinkling ad-hoc log statements, SSHing into worker nodes, or running dotnet-dump inside transient pods. By pairing AKS with Dynatrace's operator-driven injection, development and platform teams gain code-level visibility, AI-driven root cause analysis, and non-intrusive live debugging—empowering teams to resolve incidents in minutes rather than hours.
Comments