Principal AI Architect/Engineer

• Observability Platform Engineering & OTEL Integration (25%):
• Implement OpenTelemetry (OTEL) instrumentation within assigned agent frameworks or platforms — including custom exporters, span enrichers, semantic conventions, and context propagation hooks.
• Build and maintain telemetry pipeline components (collectors, processors, exporters) that route metrics, logs, traces, and semantic signals to central observability backends.
• Develop and maintain observability dashboards, alerting rules, and SLO/SLA definitions for the assigned sub-domain, ensuring signal quality and low false-positive rates.
• Participate in on-call rotations and incident response for the observability platform — contributing to RCA documentation and runbook improvement.
• Write unit, integration, and end-to-end tests for all telemetry components; maintain >80% test coverage across owned services.
• Instrument safety-critical signal capture within assigned pipelines — including guardrail trigger rates, policy violation events, prompt injection detections, and hallucination flags.
• Support red team exercises by building observability hooks that capture adversarial test results, attack surface telemetry, and behavioral deviation signals in real time.
• Implement secure trace handling for sensitive AI decision events — applying data masking, PII redaction, and audit-log retention policies as defined by the security architecture.
• Assist in maintaining the Security Observability Playbook — documenting findings, updating escalation paths, and contributing to incident classification procedures.
• Monitor agent-to-agent protocol traffic (A2A, UCP, AP2) for anomalous communication patterns and flag deviations for review by the L11 architect and security team.
• Responsible AI (RAI) & Governance Signal Instrumentation (10%):
• Implement RAI signal collectors within assigned agent workflows — capturing fairness indicators, bias detection outputs, explainability scores, and content safety classifications.
• Maintain RAI telemetry pipelines and ensure data quality, completeness, and timeliness of governance signals feeding into compliance dashboards.
• Contribute to audit-readiness work by ensuring all AI decision traces within the assigned domain include required governance metadata and are retained per policy.
• Support gap analyses by comparing current RAI signal coverage against governance framework requirements and flagging coverage gaps to the L11.
• Quality Engineering for Agentic Solutions — Post Go-Live & Continuous QE (15%):
• Build and maintain quality gate components within CI/CD pipelines — using observability data to detect performance regressions, behavioral drift, and SLA breaches before they reach production.
• Instrument and monitor Skill Evaluations (evals) across the Memory, Skills, and MCP harness stack — collecting eval results, tracking pass/fail trends, and alerting on regression thresholds.
• Implement continuous quality monitoring for post-go-live agentic solutions — tracking agent success rate, tool-call fidelity, latency distributions, and user outcome proxies.
• Conduct structured testing of new agent capabilities using standardized eval harnesses — documenting results and feeding findings into quality improvement cycles.
• Develop automated quality reports and quality metric dashboards for stakeholder review, surfacing trends and anomalies in agent behavior over time.
• Memory, Skills, MCP & Harness Engineering Observability (10%):
• Instrument agent memory operations (read/write latency, cache hit rates, memory drift) across episodic, semantic, and working memory backends within the assigned scope.

None flagged.