Observability Engineer

Position Overview
The Observability Engineer builds and improves the capabilities used to understand the health, performance, and reliability of Goosehead applications and services. This hands-on role focuses primarily on application and service observability and partners with Cloud & Infrastructure on underlying infrastructure telemetry.

The engineer works with Software Engineering, DevOps, NOC, Incident Management, and Production Support to instrument services, improve telemetry, establish service-health and alerting standards, and accelerate production diagnosis. New Relic is a key platform, but the role is broader than administration of any single tool.

Core outcome: Make production health visible, alerts actionable, and failures easier to diagnose.

Key Responsibilitie s
·  Design, implement, and improve observability for production applications, APIs, integrations, and critical services using OpenTelemetry for instrumenting metrics, logs, traces, and custom telemetry.
·  Partner with Engineering to instrument applications using OpenTelemetry for structured logging, distributed tracing, and custom metrics.
·  Create and maintain service-health dashboards, alerts, service maps, and diagnostic views that make availability, latency, errors, dependencies, and customer impact clear.
·  Establish practical standards for instrumentation, monitoring coverage, alerting, dashboards, Service Level Indicators (SLIs), and Service Level Objectives (SLOs).
·  Improve alert quality by reducing noise, false positives, duplicates, and non-actionable alerts; ensure alerts support effective NOC triage and escalation.
·  Support Incident Management and Production Support during significant events by correlating telemetry, deployments, dependencies, and system behavior to speed diagnosis.
·  Partner with Problem Management to identify observability gaps after recurring or significant incidents and implement preventive improvements.
·  Partner with DevOps and Cloud & Infrastructure to connect application observability with CI/CD, deployments, cloud, infrastructure, and platform telemetry.
·  Build scripts and automation using PowerShell, Python, Bash, APIs, or similar technologies to improve telemetry, configuration, alerting, and operational workflows.
·  Help integrate observability and alert/event data with ITSM and incident workflows, including future ServiceNow capabilities.
·  Maintain concise technical standards, reusable patterns, and operational guidance for Engineering and Technology Operations teams.
·  Maintain service-based error budgets which our development teams use to dictate feature work.

Qualifications
·  Minimum 3 years of hands-on experience in Observability Engineering, SRE, DevOps, Application Performance Monitoring, Production Engineering, Application Support Engineering, or a closely related role; 4-5 years preferred.
·  Hands-on experience with application performance monitoring, metrics, centralized logging, distributed tracing, dashboards, alerting, and production troubleshooting.
·  Experience instrumenting applications or services using OpenTelemetry for custom metrics, structured logs, and distributed tracing.
·  Strong understanding of APIs, integrations, distributed applications, service dependencies, and common performance and reliability issues.
·  Ability to correlate metrics, logs, traces, and change/deployment information to troubleshoot production issues.
·  Hands-on scripting or automation experience using PowerShell, Python, Bash, APIs, or comparable technologies.
·  Working knowledge of SLIs, SLOs, availability, latency, error rates, alert design, and service-health concepts.
·  Working knowledge of cloud environments and the relationship between application and infrastructure telemetry.
·  Strong troubleshooting, analytical, communication, and cross-team collaboration skills.
·  Associate or bachelor's degree in Information Technology, Computer Science, Engineering, or related field preferred; equivalent technical experience considered.

Preferred Qualifications
·  Hands-on experience with New Relic, including OpenTelemetry services, logs, dashboards, alerting, distributed tracing, or service maps.
·  Experience with comparable observability platforms such as Datadog, Dynatrace, Splunk Observability, AppDynamics, Azure Monitor/Application Insights, Grafana, or Prometheus.
·  Practical OpenTelemetry implementation experience and experience defining reusable observability or instrumentation standards.
·  Experience establishing SLIs/SLOs or service-health measures for production applications.
·  Experience integrating observability platforms with incident, event, notification, or ITSM workflows; ServiceNow experience is a plus.
·  Experience supporting Microsoft Azure or other public cloud environments and integrating observability with CI/CD or deployment processes.
·  Familiarity with ITIL Incident Management and Problem Management practices.