- Jarmin
- San Francisco, CA
- Full-Time
- 42 days ago
- $130,000 – $180,000
Member of Technical Staff (Evals).
Before you go
Before you leave us, sign up for our email alerts
We don't do job spam, just the best digital jobs delivered straight to your inbox.
Member of Technical Staff (Evals): our view in 3 lines...
- The Role:This role is for a technical staff engineer focused on evaluating how AI employees perform on the job.
- The Person:The person will build evaluation systems, feedback loops, and autonomous failure analysis, and define what good, degraded, and failed runs look like for alerting.
- Requirements:The ideal candidate has built evaluation or measurement systems, has strong software engineering fundamentals, and has shipped LLM systems to production.
About the role
As a Member of Technical Staff on Evals, you'll own how we measure AI Employee performance on the job.
Areas you may work in:
- Evaluation systems: datasets, offline replay, scorers, and regression alerts
- Feedback loops for models and agents
- Autonomous failure analysis
- Defining what good, degraded, and failed runs look like, and alerting on them
You may be a good fit if you:
- Have built evaluation or measurement systems, such as AI evals, experimentation, ranking, or search quality
- Can turn ambiguous quality questions into concrete metrics and decisions
- Have strong software engineering fundamentals and ship production systems
- Are comfortable where there are few established patterns
- Want to work in person in San Francisco
Even better:
- You've shipped LLM systems to production and learned where they break
- You have strong data instincts and work well with researchers
- You contribute to open source projects
Jarmin.ai builds AI employees for human-level roles. It learns its responsibilities, understands your data, embeds into your systems, and becomes part of your team's day-to-day workflow. Completely hand over full initiatives or individual tasks.
Hire AI ML Engineers, Account Executives, Paralegals, Financial analysts, Compliance analysts, and more.
Jarmin’s founding team is from Meta SuperIntelligence Labs (Staff Engineers), Apple (Machine Learning Engineer), AWS (scaling infra), Lockheed-Martin (Research), and JPMorganChase (SWE) with experience building AI/ML and automating complex work of Researchers, Data Scientists, ML Engineers with AI.
