Prometheus, Grafana, and OpenTelemetry with Aspire

Duration: 3 days

Hands-on workshop for operations, DevOps engineers, and developers to instrument, collect, visualize, and alert on logs, metrics, and traces using Prometheus, Grafana, Tempo/Jaeger, and Azure services. All labs run in Docker with Aspire to orchestrate collectors, exporters, and sample apps; Azure Log Analytics and Azure Container Apps are included for hybrid scenarios.

Audience

• Operations/DevOps/Developers/SREs building or operating distributed services (.NET-focused but language-agnostic)
• Teams adopting Prometheus/Grafana plus OpenTelemetry and Azure monitoring

Prerequisites

• Docker Desktop installed; basic container familiarity
• Basic understanding of HTTP/gRPC services
• Optional: Azure subscription for Log Analytics and Container Apps labs

Learning Goals

• Stand up Prometheus, Grafana, and tracing backends via Aspire + Docker
• Collect metrics, logs, and traces from .NET and polyglot services using OpenTelemetry exporters
• Build effective Grafana dashboards and alerts (RED/USE patterns)
• Integrate with Azure Log Analytics / Application Insights and run workloads on Azure Container Apps
• Optimize retention, recording rules, and scaling patterns for Prometheus/Grafana stacks
• Understand multiple observability stack options for logs/metrics/traces (Grafana stack, OpenTelemetry-centric stack, ELK-based stack)
• Evaluate cloud stacks (Azure- and AWS-managed observability)

Agenda

1. Architecture and setup
   • Prometheus/Grafana basics; exporters, jobs, labels, auth
   • Aspire + Docker composition for Prometheus, Grafana, Tempo/Jaeger, sample apps
2. Metric collection
   • Host metrics (Linux/Windows nodes), containers, SQL Server/PostgreSQL
   • App metrics (.NET, REST/gRPC); scraping and OTLP ingestion
   • Network/traffic analysis; querying local and cloud endpoints
   • Custom metrics (C#, PowerShell, Python) via OpenTelemetry
3. Dashboards in Grafana
   • Structure, variables, panels, templates
   • Visualizing KPIs and health; best practices for clarity
4. Alerting
   • Alert rules, routing, escalation, templates
   • Notifications (email, webhooks, OpsGenie, etc.)
5. Performance and request analysis
   • RED/USE methods; latency, errors, throughput
   • Correlations and root-cause analysis; service dependency graphs
   • Distributed tracing with Tempo/Jaeger; request-flow visualization
   • Load testing patterns and performance troubleshooting
6. Optimization and scaling
   • Retention policies, selective metric collection
   • Recording rules and query tuning
   • Long-term storage (Thanos/Cortex/VictoriaMetrics/cloud)
   • Horizontal scaling: federation and sharding; monitoring Prometheus itself
7. Azure integration
   • Shipping telemetry to Azure Log Analytics / Application Insights
   • Running services on Azure Container Apps with Prometheus/Grafana-compatible telemetry
8. Stack options (comparative)
   • Grafana stack: Prometheus, Loki, Tempo, Grafana, Promtail/Fluent Bit, OpenTelemetry Collector
   • OpenTelemetry-centric stack: OpenTelemetry Collector, Prometheus, Tempo/Jaeger, Loki, ELK
   • ELK + Jaeger + Prometheus stack: ELK, Prometheus, Jaeger, Grafana or Kibana
   • Optional Azure-managed: Azure Monitor/Log Analytics/App Insights; managed Prometheus/Grafana; Container Apps
   • Optional AWS-managed: CloudWatch/X-Ray, Amazon Managed Prometheus, Amazon Managed Grafana, OpenSearch, ADOT Collector

Hands-on Labs

• Launch Prometheus, Grafana, Tempo/Jaeger, and sample services via Aspire + Docker
• Scrape host/container/app metrics; expose custom metrics from a .NET service
• Build Grafana dashboards with variables, RED/USE panels, and health views
• Configure alert rules and route notifications
• Trace requests end-to-end with Tempo/Jaeger; view dependency graphs
• Apply recording rules and retention tweaks; observe impact
• Push telemetry to Azure Log Analytics and view in Azure portal
• Deploy the sample service to Azure Container Apps and validate metrics/traces

Outcomes

• A runnable Aspire+Doker stack for Prometheus/Grafana/Tempo/Jaeger and sample services
• Practical dashboards and alerts aligned to RED/USE patterns
• End-to-end tracing and correlation across services
• Patterns for hybrid monitoring with Azure Log Analytics and Container Apps