Imported from TheLobbi/Claude-m (azure-monitor/skills/azure-monitor/SKILL.md). Install upstream with npx skills add TheLobbi/Claude-m --skill azure-monitor. Copyright stays with the author.

Azure Monitor

Shared Workflow Routing

• Use the shared workflow spec for deterministic multi-plugin routing: workflows/multi-plugin-workflows.md.
• Apply the trigger phrases, handoff contracts, auth prerequisites, validation checkpoints, and stop conditions before escalating to the next plugin.

1. Azure Monitor Overview

Azure Monitor is the unified monitoring platform for all Azure resources and hybrid workloads. It collects, analyzes, and acts on telemetry from cloud and on-premises environments.

Data hierarchy:

Data sources:

Data sinks:

Key services in the Azure Monitor ecosystem:

• Log Analytics — Query engine and workspace for log data (KQL)
• Application Insights — APM for web applications (requests, dependencies, exceptions, traces)
• Metrics Explorer — Interactive charting for platform and custom metrics
• Alerts — Metric alerts, log alerts, activity log alerts, smart detection
• Workbooks — Rich interactive reports combining metrics, logs, and parameters
• Dashboards — Pinnable tiles in Azure Portal for at-a-glance views

2. Log Analytics Workspaces

A Log Analytics workspace is the central data store for Azure Monitor Logs. All KQL queries, log alerts, and workbooks run against one or more workspaces.

Create a workspace:

az monitor log-analytics workspace create \
  --resource-group monitoring-rg \
  --workspace-name prod-logs \
  --location eastus \
  --retention-time 90 \
  --sku PerGB2018

Pricing tiers:

Data retention:

• Default: 30 days (interactive retention, included in ingestion cost)
• Configurable: 30 to 730 days for interactive retention (charges apply beyond 31 days)
• Archive tier: Up to 12 years total (7 years archive), lower cost, requires search jobs or restore to query
• Per-table retention overrides: Set different retention per table (e.g., SecurityEvent = 365 days, Perf = 90 days)

# Set workspace-level retention
az monitor log-analytics workspace update \
  --resource-group monitoring-rg \
  --workspace-name prod-logs \
  --retention-time 180

# Set table-level retention
az monitor log-analytics workspace table update \
  --resource-group monitoring-rg \
  --workspace-name prod-logs \
  --table-name SecurityEvent \
  --retention-time 365 \
  --total-retention-time 730

Data collection rules (DCR): DCRs define what data to collect, how to transform it, and where to send it. They replace the legacy agent configuration model.

# Create a DCR for Windows performance counters
az monitor data-collection rule create \
  --resource-group monitoring-rg \
  --name "windows-perf-dcr" \
  --location eastus \
  --data-flows '[{"streams":["Microsoft-Perf"],"destinations":["prod-logs"]}]' \
  --destinations '{"logAnalytics":[{"workspaceResourceId":"/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/Microsoft.OperationalInsights/workspaces/prod-logs","name":"prod-logs"}]}' \
  --data-sources '{"performanceCounters":[{"name":"perfCounters","streams":["Microsoft-Perf"],"samplingFrequencyInSeconds":60,"counterSpecifiers":["\\Processor(_Total)\\% Processor Time","\\Memory\\Available MBytes","\\LogicalDisk(_Total)\\% Free Space"]}]}'

Cross-workspace queries:

// Query across multiple workspaces
union
  workspace("prod-logs").Heartbeat,
  workspace("staging-logs").Heartbeat
| summarize count() by Computer, _ResourceId
| order by count_ desc

Workspace design patterns:

3. KQL (Kusto Query Language)

KQL is the query language for Log Analytics, Application Insights, Azure Data Explorer, and Microsoft Sentinel. Queries are composed of tabular operators chained with the pipe (|) character.

Core Operators

Scalar Functions

Aggregation Functions

Time Series

// Create a time series of request counts per 5 minutes
requests
| make-series RequestCount=count() on timestamp from ago(24h) to now() step 5m
| render timechart

// Decompose time series into trend, seasonal, and residual components
requests
| make-series RequestCount=count() on timestamp from ago(7d) to now() step 1h
| extend (anomalies, score, baseline) = series_decompose_anomalies(RequestCount)
| mv-expand timestamp to typeof(datetime), RequestCount to typeof(long),
    anomalies to typeof(int), score to typeof(double), baseline to typeof(double)
| where anomalies != 0
| project timestamp, RequestCount, baseline, score

Practical Query Examples

1. Top 10 exceptions in the last 24 hours:

exceptions
| where timestamp > ago(24h)
| summarize Count=count() by outerMessage, problemId
| top 10 by Count desc
| project outerMessage, Count, problemId

2. P95 response time by operation:

requests
| where timestamp > ago(1h)
| where success == true
| summarize P95=percentile(duration, 95), Avg=avg(duration), Count=count()
    by operation_Name
| order by P95 desc
| render table

3. Failed dependency calls with error details:

dependencies
| where timestamp > ago(6h)
| where success == false
| summarize FailCount=count(), AvgDuration=avg(duration)
    by target, type, resultCode
| order by FailCount desc

4. VM CPU usage above 90% in the last hour:

Perf
| where TimeGenerated > ago(1h)
| where ObjectName == "Processor" and CounterName == "% Processor Time"
    and InstanceName == "_Total"
| summarize AvgCPU=avg(CounterValue), MaxCPU=max(CounterValue) by Computer
| where AvgCPU > 90
| order by AvgCPU desc

5. Sign-in failures by user (Azure AD logs):

SigninLogs
| where TimeGenerated > ago(24h)
| where ResultType != "0"
| summarize FailCount=count(), Reasons=make_set(ResultDescription) by UserPrincipalName, AppDisplayName
| order by FailCount desc
| take 20

6. HTTP 5xx errors with request details:

requests
| where timestamp > ago(4h)
| where resultCode startswith "5"
| project timestamp, name, url, resultCode, duration, operation_Id
| order by timestamp desc
| take 50

7. Slow database queries (dependencies):

dependencies
| where timestamp > ago(1h)
| where type == "SQL"
| where duration > 2000
| project timestamp, target, data, duration, operation_Id
| order by duration desc
| take 25

8. Availability test results:

availabilityResults
| where timestamp > ago(24h)
| summarize SuccessRate=countif(success == true) * 100.0 / count(),
    AvgDuration=avg(duration)
    by name, location
| order by SuccessRate asc

9. Custom event analysis:

customEvents
| where timestamp > ago(7d)
| where name == "FeatureUsed"
| extend Feature = tostring(customDimensions["featureName"])
| summarize Users=dcount(user_Id), Uses=count() by Feature
| order by Users desc
| render barchart

10. Resource health changes:

AzureActivity
| where TimeGenerated > ago(7d)
| where CategoryValue == "ResourceHealth"
| project TimeGenerated, ResourceGroup, Resource=_ResourceId,
    Status=ActivityStatusValue, Caller
| order by TimeGenerated desc

11. Container (AKS) pod restarts:

KubePodInventory
| where TimeGenerated > ago(24h)
| where PodRestartCount > 0
| summarize MaxRestarts=max(PodRestartCount), LastSeen=max(TimeGenerated)
    by Namespace, Name, ContainerName
| where MaxRestarts > 5
| order by MaxRestarts desc

12. Network Security Group flow logs:

AzureNetworkAnalytics_CL
| where TimeGenerated > ago(1h)
| where FlowStatus_s == "D"  // Denied flows
| summarize DeniedFlows=count() by SrcIP_s, DestIP_s, DestPort_d, NSGRule_s
| order by DeniedFlows desc
| take 20

4. Application Insights

Application Insights is the APM (Application Performance Management) feature of Azure Monitor. It monitors live web applications, detects anomalies, and provides deep diagnostics.

Create an Application Insights resource:

az monitor app-insights component create \
  --app my-web-app-insights \
  --location eastus \
  --resource-group monitoring-rg \
  --workspace "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/Microsoft.OperationalInsights/workspaces/prod-logs" \
  --kind web \
  --application-type web

Telemetry Types

Instrumentation (Node.js / TypeScript)

Auto-instrumentation with Azure Monitor OpenTelemetry:

npm install @azure/monitor-opentelemetry

import { useAzureMonitor } from "@azure/monitor-opentelemetry";

useAzureMonitor({
  azureMonitorExporterOptions: {
    connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING,
  },
  instrumentationOptions: {
    http: { enabled: true },
    azureSdk: { enabled: true },
    mongoDb: { enabled: true },
    mySql: { enabled: true },
    postgreSql: { enabled: true },
    redis: { enabled: true },
  },
});

// Must be called before importing other modules
import express from "express";
const app = express();

Manual telemetry with Application Insights SDK:

npm install applicationinsights

import * as appInsights from "applicationinsights";

appInsights.setup(process.env.APPLICATIONINSIGHTS_CONNECTION_STRING)
  .setAutoCollectRequests(true)
  .setAutoCollectDependencies(true)
  .setAutoCollectExceptions(true)
  .setAutoCollectPerformance(true, true)
  .setAutoCollectConsole(true, true)
  .setSendLiveMetrics(true)
  .start();

const client = appInsights.defaultClient;

// Track custom event
client.trackEvent({
  name: "OrderPlaced",
  properties: { orderId: "ORD-123", plan: "enterprise" },
  measurements: { orderValue: 499.99 },
});

// Track custom metric
client.trackMetric({
  name: "ActiveUsers",
  value: 142,
});

// Track exception
try {
  await processOrder(order);
} catch (err) {
  client.trackException({
    exception: err as Error,
    properties: { orderId: order.id },
  });
  throw err;
}

Sampling

Sampling reduces telemetry volume while preserving statistical accuracy.

appInsights.setup(connectionString)
  .setSendLiveMetrics(true);

// Fixed-rate sampling at 25%
appInsights.defaultClient.config.samplingPercentage = 25;

appInsights.start();

Sampling exclusions (always collect certain telemetry types):

const { SamplingTelemetryProcessor } = require("applicationinsights/out/TelemetryProcessors");

// Exclude exceptions and dependencies from sampling
appInsights.defaultClient.addTelemetryProcessor((envelope) => {
  if (envelope.data.baseType === "ExceptionData" ||
      envelope.data.baseType === "RemoteDependencyData") {
    return true; // Always send
  }
  return undefined; // Apply normal sampling
});

Live Metrics

Live Metrics Stream provides real-time (1-second latency) visibility into requests, dependencies, exceptions, CPU, and memory without affecting performance.

# Enable live metrics in SDK
appInsights.setup(connectionString).setSendLiveMetrics(true).start();

Access in Azure Portal: Application Insights > Live Metrics.

Application Map

Application Map automatically discovers the topology of your application — the web frontend, API, database, and external dependencies — and shows health, latency, and failure rates on each component.

Access in Azure Portal: Application Insights > Application map.

Availability Tests

Synthetic tests that probe your application from Azure datacenters worldwide.

# Create a URL ping availability test
az monitor app-insights web-test create \
  --resource-group monitoring-rg \
  --name "Homepage Ping" \
  --app-insights my-web-app-insights \
  --location "eastus" \
  --web-test-kind ping \
  --defined-web-test-name "HomepagePing" \
  --url "https://myapp.azurewebsites.net/health" \
  --frequency 300 \
  --timeout 120 \
  --expected-status-code 200 \
  --retry-enabled true

5. Metrics and Metric Alerts

Platform Metrics

Platform metrics are collected automatically for all Azure resources at 1-minute granularity with 93 days of retention.

Common metric namespaces:

Query metrics via CLI:

az monitor metrics list \
  --resource "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Web/sites/<app>" \
  --metric "HttpResponseTime" "Requests" "Http5xx" \
  --interval PT5M \
  --start-time 2024-01-15T00:00:00Z \
  --end-time 2024-01-15T12:00:00Z \
  --aggregation Average Count Sum

Custom Metrics

Send application-specific metrics to Azure Monitor Metrics for near real-time alerting.

// Via Application Insights SDK
client.trackMetric({ name: "OrderProcessingTime", value: 245 });
client.trackMetric({ name: "QueueDepth", value: 38 });

// Via REST API (custom metrics endpoint)
// POST https://<region>.monitoring.azure.com/<resourceId>/metrics
// Body: { "time": "...", "data": { "baseData": { "metric": "QueueDepth", "namespace": "Custom", "series": [{ "min": 38, "max": 38, "sum": 38, "count": 1 }] } } }

Metric Alert Rules

Create a metric alert (static threshold):

az monitor metrics alert create \
  --name "High CPU Alert" \
  --resource-group monitoring-rg \
  --scopes "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Web/sites/<app>" \
  --condition "avg Percentage CPU > 85" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --severity 2 \
  --action "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/microsoft.insights/actionGroups/ops-team" \
  --description "CPU usage exceeds 85% for 5 minutes"

Create a metric alert (dynamic threshold):

az monitor metrics alert create \
  --name "Response Time Anomaly" \
  --resource-group monitoring-rg \
  --scopes "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Web/sites/<app>" \
  --condition "avg HttpResponseTime > dynamic medium 4 of 5 since 2024-01-01T00:00:00Z" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --severity 3 \
  --action "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/microsoft.insights/actionGroups/ops-team"

Action Groups

Action groups define who gets notified and what automated actions to trigger.

az monitor action-group create \
  --name ops-team \
  --resource-group monitoring-rg \
  --short-name OpsTeam \
  --email-receiver name=Lead email=ops-lead@contoso.com \
  --email-receiver name=Backup email=ops-backup@contoso.com \
  --webhook-receiver name=PagerDuty uri="https://events.pagerduty.com/integration/<key>/enqueue" \
  --logic-app-receiver name=AutoRemediate \
    resource-id="/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/Microsoft.Logic/workflows/auto-scale-up" \
    callback-url="https://prod-00.eastus.logic.azure.com/workflows/<id>/triggers/manual/paths/invoke"

6. Log Alerts

Log alerts run KQL queries on a schedule against Log Analytics workspaces or Application Insights and fire when the query returns results matching a condition.

Create a scheduled query rule (log alert):

az monitor scheduled-query create \
  --name "High Error Rate" \
  --resource-group monitoring-rg \
  --scopes "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/Microsoft.OperationalInsights/workspaces/prod-logs" \
  --condition "count > 50" \
  --condition-query "requests | where resultCode startswith '5' | summarize ErrorCount=count() by bin(timestamp, 5m)" \
  --evaluation-frequency 5m \
  --window-size 5m \
  --severity 2 \
  --action-groups "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/microsoft.insights/actionGroups/ops-team" \
  --description "More than 50 HTTP 5xx errors in 5 minutes"

Dimensions in log alerts: Split alerts by dimension values so each unique combination fires independently:

az monitor scheduled-query create \
  --name "Per-Service Error Alert" \
  --resource-group monitoring-rg \
  --scopes "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/Microsoft.OperationalInsights/workspaces/prod-logs" \
  --condition "count > 10" \
  --condition-query "requests | where success == false | summarize ErrorCount=count() by cloud_RoleName, bin(timestamp, 5m)" \
  --evaluation-frequency 5m \
  --window-size 5m \
  --severity 3 \
  --action-groups "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/microsoft.insights/actionGroups/ops-team" \
  --dimension name=cloud_RoleName operator=Include values="*"

Severity levels:

Stateful alerts: Stateful alerts have a lifecycle (Fired -> Acknowledged -> Resolved). The alert auto-resolves when the condition is no longer met:

az monitor scheduled-query create \
  --name "Disk Space Low" \
  --resource-group monitoring-rg \
  --scopes "..." \
  --condition "count > 0" \
  --condition-query "Perf | where ObjectName == 'LogicalDisk' and CounterName == '% Free Space' and CounterValue < 10 | summarize MinFree=min(CounterValue) by Computer" \
  --evaluation-frequency 15m \
  --window-size 15m \
  --severity 2 \
  --auto-mitigate true \
  --action-groups "..."

Evaluation frequency and window combinations:

7. Diagnostic Settings

Diagnostic settings route platform logs and metrics from Azure resources to one or more destinations.

Common log categories by resource type:

Create diagnostic settings via CLI:

# Send App Service logs and metrics to Log Analytics
az monitor diagnostic-settings create \
  --name "appservice-to-loganalytics" \
  --resource "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Web/sites/<app>" \
  --workspace "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/Microsoft.OperationalInsights/workspaces/prod-logs" \
  --logs '[{"categoryGroup":"allLogs","enabled":true}]' \
  --metrics '[{"category":"AllMetrics","enabled":true}]'

# Send to multiple destinations (Log Analytics + Storage for archival)
az monitor diagnostic-settings create \
  --name "sql-diagnostics" \
  --resource "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Sql/servers/<server>/databases/<db>" \
  --workspace "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/Microsoft.OperationalInsights/workspaces/prod-logs" \
  --storage-account "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<storage>" \
  --logs '[{"category":"SQLInsights","enabled":true},{"category":"Errors","enabled":true},{"category":"Deadlocks","enabled":true}]' \
  --metrics '[{"category":"AllMetrics","enabled":true}]'

Bicep template for diagnostic settings:

resource diagnosticSettings 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = {
  name: 'send-to-loganalytics'
  scope: webApp
  properties: {
    workspaceId: logAnalyticsWorkspace.id
    logs: [
      {
        categoryGroup: 'allLogs'
        enabled: true
        retentionPolicy: {
          enabled: false
          days: 0
        }
      }
    ]
    metrics: [
      {
        category: 'AllMetrics'
        enabled: true
        retentionPolicy: {
          enabled: false
          days: 0
        }
      }
    ]
  }
}

Destination options:

Audit diagnostic settings across a subscription:

# List all resources without diagnostic settings
az resource list --query "[].id" -o tsv | while read resourceId; do
  settings=$(az monitor diagnostic-settings list --resource "$resourceId" --query "value[].name" -o tsv 2>/dev/null)
  if [ -z "$settings" ]; then
    echo "NO DIAGNOSTIC SETTINGS: $resourceId"
  fi
done

8. Dashboards and Workbooks

Azure Dashboards

Azure dashboards provide a portal-based, pinnable tile view for at-a-glance monitoring.

Dashboard JSON structure:

{
  "properties": {
    "lenses": [
      {
        "order": 0,
        "parts": [
          {
            "position": { "x": 0, "y": 0, "colSpan": 6, "rowSpan": 4 },
            "metadata": {
              "type": "Extension/Microsoft_OperationsManagementSuite_Workspace/PartType/LogsDashboardPart",
              "settings": {
                "content": {
                  "Query": "requests | where timestamp > ago(24h) | summarize count() by bin(timestamp, 1h) | render timechart",
                  "PartTitle": "Request Volume (24h)"
                }
              },
              "inputs": [
                {
                  "name": "resourceTypeMode",
                  "value": "workspace"
                },
                {
                  "name": "ComponentId",
                  "value": "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/Microsoft.OperationalInsights/workspaces/prod-logs"
                }
              ]
            }
          }
        ]
      }
    ]
  }
}

Create a dashboard via CLI:

az portal dashboard create \
  --resource-group monitoring-rg \
  --name "Production Monitoring" \
  --input-path dashboard.json \
  --location eastus

Azure Workbooks

Workbooks are interactive reports combining text, KQL queries, metrics, and parameters into rich visualizations.

Key features:

Visualization types:

Parameterized workbook query example:

// Uses workbook parameter {TimeRange} and {ServiceName}
requests
| where timestamp {TimeRange}
| where cloud_RoleName == "{ServiceName}"
| summarize Requests=count(), Errors=countif(success == false),
    P95=percentile(duration, 95)
    by bin(timestamp, {TimeRange:grain})
| render timechart

Create a workbook from template via CLI:

az monitor app-insights workbook create \
  --resource-group monitoring-rg \
  --name "Service Health Dashboard" \
  --location eastus \
  --kind shared \
  --category workbook \
  --serialized-data @workbook-template.json

9. Distributed Tracing

Distributed tracing follows a single user request across multiple services, showing exactly where time is spent and where failures occur.

OpenTelemetry Integration

Azure Monitor supports OpenTelemetry as the standard for distributed tracing.

npm install @azure/monitor-opentelemetry @opentelemetry/api

import { useAzureMonitor } from "@azure/monitor-opentelemetry";
import { trace, context, SpanKind, SpanStatusCode } from "@opentelemetry/api";

// Initialize Azure Monitor with OpenTelemetry
useAzureMonitor({
  azureMonitorExporterOptions: {
    connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING,
  },
});

// Create custom spans
const tracer = trace.getTracer("my-service");

async function processOrder(orderId: string) {
  return tracer.startActiveSpan("processOrder", { kind: SpanKind.INTERNAL }, async (span) => {
    try {
      span.setAttribute("order.id", orderId);

      // Nested span for validation
      await tracer.startActiveSpan("validateOrder", async (validationSpan) => {
        await validateOrder(orderId);
        validationSpan.end();
      });

      // Nested span for payment
      await tracer.startActiveSpan("chargePayment", async (paymentSpan) => {
        paymentSpan.setAttribute("payment.method", "credit_card");
        await chargePayment(orderId);
        paymentSpan.end();
      });

      span.setStatus({ code: SpanStatusCode.OK });
    } catch (error) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: (error as Error).message });
      span.recordException(error as Error);
      throw error;
    } finally {
      span.end();
    }
  });
}

Correlation IDs

Azure Monitor uses W3C Trace Context for correlation across services:

• Operation ID (operation_Id): Root trace ID, shared by all telemetry in one user request
• Parent ID (operation_ParentId): Span ID of the calling service
• Span ID: Unique ID for each operation within the trace

Query correlated telemetry:

// Find all telemetry for a specific operation
union requests, dependencies, exceptions, traces
| where operation_Id == "<operation-id>"
| project timestamp, itemType, name, duration, success, message,
    operation_ParentId, itemId
| order by timestamp asc

Performance Profiler

Application Insights Profiler captures .NET call stacks during requests, showing exactly which code paths consume time. Enable it in Application Insights > Performance > Profiler.

For Node.js applications, use the distributed tracing view in Application Insights > Transaction search to drill into request timelines.

Snapshot Debugger

Snapshot Debugger captures memory snapshots when exceptions occur, allowing you to inspect variable values at the time of failure without redeploying.

# Enable snapshot debugger (via portal or ARM)
az monitor app-insights component update \
  --app my-web-app-insights \
  --resource-group monitoring-rg \
  --set properties.DisableIpMasking=false

Access in Azure Portal: Application Insights > Failures > Select an exception > Open Debug Snapshot.

10. Alerting Best Practices

Prevent Alert Fatigue

Severity Guidelines

Action Group Design

Alert Processing Rules

Alert processing rules (formerly action rules) modify alert behavior without changing the alert rule itself.

# Suppress all alerts during a maintenance window
az monitor alert-processing-rule create \
  --name "maintenance-window" \
  --resource-group monitoring-rg \
  --rule-type RemoveAllActionGroups \
  --scopes "/subscriptions/<sub>/resourceGroups/production-rg" \
  --schedule-recurrence-type Weekly \
  --schedule-recurrence "Sunday" \
  --schedule-start-time "02:00:00" \
  --schedule-end-time "06:00:00" \
  --schedule-time-zone "Eastern Standard Time"

Suppression Rules

Prevent duplicate notifications for the same issue:

• Mute period: After an alert fires, suppress repeat notifications for a specified duration (e.g., 1 hour)
• Resolution required: Only re-fire after the alert has auto-resolved and re-triggered
• Daily digest: Use Logic App to batch low-severity alerts into a daily summary

11. Cost Management

Log Analytics Cost Drivers

Data Ingestion Optimization

Use Basic logs for high-volume, low-query tables: Basic logs cost ~$0.65/GB (vs $2.76/GB analytics) but have limited query capabilities (8-day retention, no alerts, limited KQL).

# Set a table to Basic logs plan
az monitor log-analytics workspace table update \
  --resource-group monitoring-rg \
  --workspace-name prod-logs \
  --table-name ContainerLogV2 \
  --plan Basic

Tables suitable for Basic logs: ContainerLogV2, AppTraces, custom tables with high volume and infrequent queries.

Data Collection Rule (DCR) transforms for filtering:

{
  "transformKql": "source | where SeverityLevel >= 2 | project-away RawData, AdditionalContext"
}

This KQL runs at ingestion time, dropping debug/verbose logs and unnecessary columns before they reach the workspace.

Workspace Commitment Tiers

# Check current daily ingestion
az monitor log-analytics workspace show \
  --resource-group monitoring-rg \
  --workspace-name prod-logs \
  --query "properties.sku"

# Usage query to estimate daily volume
# Run in Log Analytics:
# Usage | where TimeGenerated > ago(30d) | summarize DailyGB=sum(Quantity)/1024 by bin(TimeGenerated, 1d) | summarize AvgDaily=avg(DailyGB), P95Daily=percentile(DailyGB, 95)

Cost Monitoring Queries

// Daily ingestion by table (last 30 days)
Usage
| where TimeGenerated > ago(30d)
| summarize IngestedGB = sum(Quantity) / 1024 by DataType, bin(TimeGenerated, 1d)
| summarize AvgDailyGB = avg(IngestedGB), TotalGB = sum(IngestedGB) by DataType
| order by TotalGB desc

// Billable vs non-billable data
Usage
| where TimeGenerated > ago(30d)
| summarize TotalGB = sum(Quantity) / 1024 by IsBillable, DataType
| order by TotalGB desc

12. Common Patterns

Pattern 1: Full-Stack Monitoring for Web App + SQL + Functions

Set up comprehensive monitoring for a typical Azure web application.

Step 1: Create shared monitoring infrastructure:

# Create resource group and Log Analytics workspace
az group create --name monitoring-rg --location eastus
az monitor log-analytics workspace create \
  --resource-group monitoring-rg --workspace-name prod-logs \
  --location eastus --retention-time 90

# Create Application Insights (workspace-based)
az monitor app-insights component create \
  --app prod-app-insights --location eastus \
  --resource-group monitoring-rg \
  --workspace "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/Microsoft.OperationalInsights/workspaces/prod-logs" \
  --kind web --application-type web

# Create action group
az monitor action-group create \
  --name ops-team --resource-group monitoring-rg --short-name OpsTeam \
  --email-receiver name=Lead email=ops@contoso.com

Step 2: Enable diagnostic settings for all resources:

# App Service
az monitor diagnostic-settings create \
  --name "appservice-diag" \
  --resource "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Web/sites/<app>" \
  --workspace "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/Microsoft.OperationalInsights/workspaces/prod-logs" \
  --logs '[{"categoryGroup":"allLogs","enabled":true}]' \
  --metrics '[{"category":"AllMetrics","enabled":true}]'

# SQL Database
az monitor diagnostic-settings create \
  --name "sql-diag" \
  --resource "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Sql/servers/<server>/databases/<db>" \
  --workspace "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/Microsoft.OperationalInsights/workspaces/prod-logs" \
  --logs '[{"category":"SQLInsights","enabled":true},{"category":"Errors","enabled":true},{"category":"Deadlocks","enabled":true}]' \
  --metrics '[{"category":"AllMetrics","enabled":true}]'

# Azure Functions
az monitor diagnostic-settings create \
  --name "functions-diag" \
  --resource "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Web/sites/<function-app>" \
  --workspace "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/Microsoft.OperationalInsights/workspaces/prod-logs" \
  --logs '[{"categoryGroup":"allLogs","enabled":true}]' \
  --metrics '[{"category":"AllMetrics","enabled":true}]'

Step 3: Configure Application Insights in the app:

import { useAzureMonitor } from "@azure/monitor-opentelemetry";

useAzureMonitor({
  azureMonitorExporterOptions: {
    connectionString: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING,
  },
});

Step 4: Create alerts:

# High error rate
az monitor metrics alert create \
  --name "High-5xx-Rate" --resource-group monitoring-rg \
  --scopes "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Web/sites/<app>" \
  --condition "total Http5xx > 10" --window-size 5m --evaluation-frequency 1m \
  --severity 1 --action "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/microsoft.insights/actionGroups/ops-team"

# SQL DTU saturation
az monitor metrics alert create \
  --name "SQL-DTU-High" --resource-group monitoring-rg \
  --scopes "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Sql/servers/<server>/databases/<db>" \
  --condition "avg dtu_consumption_percent > 85" --window-size 10m --evaluation-frequency 5m \
  --severity 2 --action "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/microsoft.insights/actionGroups/ops-team"

# Availability test
az monitor app-insights web-test create \
  --resource-group monitoring-rg --name "Health Check" \
  --app-insights prod-app-insights --location eastus \
  --web-test-kind ping --defined-web-test-name "HealthCheck" \
  --url "https://myapp.azurewebsites.net/health" \
  --frequency 300 --timeout 120 --expected-status-code 200

Pattern 2: KQL Dashboard for Error Analysis

Build a workbook-style query set for triaging production errors.

// 1. Error overview tile
requests
| where timestamp > ago(24h)
| summarize TotalRequests=count(), FailedRequests=countif(success == false),
    ErrorRate=round(countif(success == false) * 100.0 / count(), 2)
| project TotalRequests, FailedRequests, ErrorRate

// 2. Errors by status code over time
requests
| where timestamp > ago(24h)
| where success == false
| summarize Count=count() by resultCode, bin(timestamp, 1h)
| render timechart

// 3. Top failing operations
requests
| where timestamp > ago(24h)
| where success == false
| summarize ErrorCount=count(), P95Latency=percentile(duration, 95) by operation_Name
| order by ErrorCount desc
| take 10

// 4. Exception breakdown with stack traces
exceptions
| where timestamp > ago(24h)
| summarize Count=count(), LastSeen=max(timestamp) by type, outerMessage
| order by Count desc
| take 10

// 5. Dependency failures impacting requests
dependencies
| where timestamp > ago(24h)
| where success == false
| join kind=inner (
    requests
    | where timestamp > ago(24h)
    | where success == false
    | project operation_Id
) on operation_Id
| summarize ImpactedRequests=dcount(operation_Id), AvgLatency=avg(duration)
    by target, type, resultCode
| order by ImpactedRequests desc

Pattern 3: Custom Availability Monitoring with Multi-Step Tests

Implement programmatic availability tests for complex user flows.

import * as appInsights from "applicationinsights";

appInsights.setup(process.env.APPLICATIONINSIGHTS_CONNECTION_STRING).start();
const client = appInsights.defaultClient;

async function runAvailabilityTest() {
  const testName = "Checkout Flow";
  const startTime = new Date();
  let success = true;
  let message = "All steps passed";
  let duration = 0;

  try {
    // Step 1: Load product page
    const step1Start = Date.now();
    const productRes = await fetch("https://myapp.com/api/products/1");
    if (!productRes.ok) throw new Error(`Product page failed: ${productRes.status}`);
    const step1Duration = Date.now() - step1Start;

    // Step 2: Add to cart
    const step2Start = Date.now();
    const cartRes = await fetch("https://myapp.com/api/cart", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ productId: "1", quantity: 1 }),
    });
    if (!cartRes.ok) throw new Error(`Add to cart failed: ${cartRes.status}`);
    const step2Duration = Date.now() - step2Start;

    // Step 3: Validate cart
    const step3Start = Date.now();
    const validateRes = await fetch("https://myapp.com/api/cart/validate");
    if (!validateRes.ok) throw new Error(`Cart validation failed: ${validateRes.status}`);
    const step3Duration = Date.now() - step3Start;

    duration = step1Duration + step2Duration + step3Duration;
    message = `Step1: ${step1Duration}ms, Step2: ${step2Duration}ms, Step3: ${step3Duration}ms`;
  } catch (error) {
    success = false;
    message = (error as Error).message;
    duration = Date.now() - startTime.getTime();
  }

  client.trackAvailability({
    name: testName,
    success,
    duration,
    runLocation: "Custom-EastUS",
    message,
    time: startTime,
    id: `${testName}-${startTime.toISOString()}`,
  });
}

// Run every 5 minutes
setInterval(runAvailabilityTest, 5 * 60 * 1000);
runAvailabilityTest();

Pattern 4: Cost-Optimized Logging with DCR Filtering

Use Data Collection Rules to filter noisy telemetry at ingestion time, reducing costs.

Step 1: Identify high-volume tables:

Usage
| where TimeGenerated > ago(30d)
| summarize DailyGB = sum(Quantity) / 1024.0 / 30.0 by DataType
| order by DailyGB desc
| take 10

Step 2: Create a DCR with transform that filters verbose logs:

{
  "properties": {
    "dataCollectionEndpointId": "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/Microsoft.Insights/dataCollectionEndpoints/prod-dce",
    "streamDeclarations": {
      "Custom-AppLogs": {
        "columns": [
          { "name": "TimeGenerated", "type": "datetime" },
          { "name": "Level", "type": "string" },
          { "name": "Message", "type": "string" },
          { "name": "ServiceName", "type": "string" }
        ]
      }
    },
    "dataSources": {
      "logFiles": [
        {
          "streams": ["Custom-AppLogs"],
          "filePatterns": ["/var/log/myapp/*.log"],
          "format": "json",
          "name": "appLogs"
        }
      ]
    },
    "dataFlows": [
      {
        "streams": ["Custom-AppLogs"],
        "destinations": ["prod-logs"],
        "transformKql": "source | where Level in ('Error', 'Warning', 'Critical') or (Level == 'Information' and Message !has 'health check')",
        "outputStream": "Custom-AppLogs_CL"
      }
    ],
    "destinations": {
      "logAnalytics": [
        {
          "workspaceResourceId": "/subscriptions/<sub>/resourceGroups/monitoring-rg/providers/Microsoft.OperationalInsights/workspaces/prod-logs",
          "name": "prod-logs"
        }
      ]
    }
  }
}

Step 3: Switch high-volume tables to Basic plan:

# ContainerLogV2 is often the largest table in AKS environments
az monitor log-analytics workspace table update \
  --resource-group monitoring-rg \
  --workspace-name prod-logs \
  --table-name ContainerLogV2 \
  --plan Basic

# AppTraces for verbose application logs
az monitor log-analytics workspace table update \
  --resource-group monitoring-rg \
  --workspace-name prod-logs \
  --table-name AppTraces \
  --plan Basic

Step 4: Set up cost alert:

# Create a budget alert for the Log Analytics workspace
az consumption budget create \
  --budget-name "log-analytics-budget" \
  --amount 500 \
  --time-grain Monthly \
  --start-date 2024-01-01 \
  --end-date 2025-12-31 \
  --resource-group monitoring-rg \
  --category Cost \
  --notifications '{"actual_GreaterThan_80":{"enabled":true,"operator":"GreaterThan","threshold":80,"contactEmails":["ops@contoso.com"]}}'

Knowledge references

• references/operational-knowledge.md — compact API surface map, prerequisite matrix, deterministic failure remediation, limits/quotas and pagination/throttling guidance, and safe-default read-first/apply-second pattern.

Progressive Disclosure — Reference Files