Threat Investigation Engineer

About 10a Labs: 10a Labs is the safety and threat-intelligence layer trusted by frontier AI labs, AI unicorns, Fortune 10 companies, and leading global technology platforms. Our adversarial red teaming, model evaluations, and intelligence collection enable engineering, safety, and security teams to stay ahead of evolving threats and deploy AI systems safely.

Location & Travel Expectations

We are primarily seeking candidates based in the Washington, DC, New York City, San Francisco Bay Area, and Denver metropolitan areas. Candidates hired in these locations will be expected to work from a 10a Labs office at least two days per week.

All new hires will participate in in-person onboarding in Washington, DC, New York City, or San Francisco during their first week. Occasional travel may also be required for interviews, in-person meetings, and other business needs.

About the Role

We are seeking a Threat Investigation Engineer to build, improve, and support AI-powered systems used in high-stakes safety and threat investigations.

This is a hands-on software engineering role combining backend development, production troubleshooting, and platform reliability. You'll work closely with investigators and engineering teams to develop practical solutions, improve existing workflows, and ensure critical investigation systems operate reliably.

The ideal candidate is a strong Python engineer with experience building and maintaining production applications, troubleshooting complex systems, and integrating AI-powered capabilities. You should be comfortable navigating ambiguous technical challenges, working directly with end users, and taking ownership of solutions from initial problem definition through deployment and ongoing support.

This position includes participation in a rotating on-call schedule, including weekend and holiday coverage, to support critical investigation workflows.

What You'll Do

- Design, develop, test, and maintain backend services and AI-assisted applications that support threat investigation workflows.

- Partner directly with investigators to understand operational challenges, identify opportunities for improvement, and translate requirements into practical engineering solutions.

- Build and integrate AI-powered features, APIs, and services that improve the efficiency, accuracy, and reliability of investigation processes.

- Troubleshoot and resolve production issues, including failed workflows, processing delays, data inconsistencies, and system errors.

- Participate in rotating on-call coverage, including weekends and holidays, to diagnose issues, implement fixes, and escalate when necessary.

- Improve system reliability and observability through monitoring, logging, automated testing, documentation, and operational tooling.

- Investigate complex technical issues across backend services, data pipelines, and AI-assisted workflows, identifying root causes and implementing sustainable solutions.

- Collaborate with engineering and platform teams to deploy improvements, maintain system stability, and support evolving operational needs.

- Own engineering tasks end-to-end, from understanding user requirements and designing solutions to implementation, testing, deployment, and maintenance.

What We're Looking For

- Strong software engineering experience, particularly in Python and backend application development .

- Experience with platform engineering, site reliability engineering (SRE), or production operations.

- Experience building, deploying, debugging, and maintaining production software systems.

- Familiarity with REST APIs, backend services, asynchronous processing, and system integrations.

- Practical experience working with AI-powered applications, LLM APIs, or model-assisted development tools.

- Strong troubleshooting and debugging skills, with the ability to investigate unfamiliar systems and resolve complex production issues.

- Experience supporting production environments, including monitoring, incident response, and reliability improvements.

- Ability to independently scope technical problems, prioritize work, and deliver maintainable solutions.

- Strong communication skills and experience collaborating with technical and nontechnical stakeholders.

- Sound judgment when handling sensitive information, operational incidents, and time-sensitive technical challenges.

- Availability to participate in scheduled on-call coverage, including weekends and holidays.

Nice to Have

- Experience with asynchronous job processing, background workers, task queues, or distributed systems.

- Familiarity with cloud infrastructure and services such as AWS, GCP, or Azure.

- Experience building internal tools, workflow automation, or investigator-facing applications.

- Familiarity with observability tools, structured logging, alerting, and incident management practices.

- Experience working in cybersecurity, trust and safety, threat intelligence, or other high-stakes operational environments.

Additional Information

- Employment Type: Full-Time

- Compensation: $160,000 - $200,000

- Location: Remote

- Schedule: Includes rotating on-call coverage, with weekend and holiday availability required.

This role supports sensitive safety and threat investigations and may involve exposure to potentially distressing case information.