Job Description for Senior Backend Engineer at Instrumentl

About the role

• Own AI features end to end: from rapid prototype to production and evaluation.
• Build APIs, tool-using agents, and RAG pipelines for grant discovery, application drafting, and research tools.
• High-ownership role on a small team with fast customer impact.

What you'll do

• Ship AI to production:
• Build tool-using LLM agents with task planning, function/tool calling, multi-step workflows, guardrails.
• Turn prototypes into resilient, observable services with SLAs, rollback/fallback strategies, cost/latency budgets.
• Establish evaluation and observability for AI safety and cost-effectiveness.
• Build trustworthy backends:
• Write high-quality, tested backend code and data pipelines for retrieval and evaluation.
• Contribute to reliability practices: alerts, dashboards, incident response.
• Collaborate and raise the bar:
• Partner with Product, Design, GTM on scoping, UX, measurement.
• Run experiments (A/B tests), interpret results, iterate.
• Improve engineering standards via maintainable code, tests, docs, reviews.

What we're looking for

Required

• 7+ years building/shipping production backend systems in Python (FastAPI, Celery or equivalent).
• Experience building LLM features in production: tool/function calling, multi-step agent workflows, guardrails/evals.
• Strong data fundamentals: SQL, schema design, pipelines for retrieval/evaluation.
• Thrives in fast startup environment with high ownership and bias for action.

Nice to have

• TypeScript/Node experience; familiarity or willingness to learn Ruby on Rails.
• Experience with AWS/GCP, Docker, CI/CD, observability (logs/metrics/traces).
• RAG depth: document ingestion/chunking/windowing/embeddings/hybrid search/re-ranking/citations.
• Experience with re-rankers/cross-encoders/hybrid retrieval tuning/search/recommendation systems.
• Evaluation mindset: designing eval suites using automated/human-in-the-loop methods; familiarity with Ragas/DeepEval/OpenAI Evals.
• Orchestration frameworks: LangChain/LangGraph/LlamaIndex/Semantic Kernel/custom orchestration.

Compensation & Benefits

• US salary range: $175,000 - $220,000 USD plus equity; final depends on experience/location/etc.
• Canada compensation varies by province; shared early by recruiter.

Benefits

• 100% covered health/dental/vision insurance for employees (50% for dependents).
• Generous PTO including parental leave.
• 401(k).
• Company laptop and home-office stipend.
• Biannual company retreats.

Additional Information

Instrumentl is an equal opportunity employer committed to inclusion. Reasonable accommodations available during application/interview process. AI tools may be used in hiring process but final decisions made by humans.