Job Description

Trullion is transforming the accounting world with an AI-powered platform built for accountants and auditors. Our technology automates complex financial workflows, boosts efficiency, reduces costly errors, and fosters real collaboration. Founded in 2019, Trullion is on a mission to make accounting and audit technology people truly love, driving a more efficient, transparent, and human-oriented industry. From rapid onboarding to a seamless managed service model, we deliver a customer-first experience at every step.

Headquartered in New York, with offices in Tel Aviv and London, we are backed by top investors, including Aleph, Third Point Ventures, Greycroft, StepStone Group, as well as leading global CFOs and accounting firms.

As our company continues to grow, an exciting opportunity awaits for a dynamic individual to join our R&D team as an LLMOps Engineer. In this role, you will lead Trullion’s evaluation (evals) efforts across all AI-powered products, and help build and maintain the LLM infrastructure that powers production-grade systems.You will work closely with AI engineers, backend engineers, and product stakeholders to define what “good” means for our AI, measure it continuously, and create reliable deployment and monitoring workflows. We’re looking for someone with strong engineering fundamentals, real-world experience scaling AI systems, and a passion for turning LLM innovation into dependable customer value.

Responsibilities

• Build and maintain LLM evaluation pipelines (offline benchmarks + online quality monitoring), ensuring measurable and repeatable AI releases
• Improve AI infrastructure and production readiness, including model/prompt versioning, deployment automation, observability, and incident response readiness
• Partner cross-functionally with Product, AI, and Engineering teams to align evaluation outcomes with user impact and business KPIs
• Lead Trullion’s evaluation strategy across all AI products, including success metrics, datasets, tooling, and continuous regression testing
• Drive best practices for LLM reliability, including robustness, latency/cost optimization, drift detection, and guardrails where needed

Requirements

• 5+ years of experience in MLOps / ML Engineering / Platform Engineering / Backend Engineering in production environments.
• Proven hands-on experience building evaluation frameworks for ML/LLM systems (benchmarks, regression tests, quality metrics, automated pipelines).
• Strong software engineering capabilities (high proficiency in Python) and experience designing maintainable infrastructure/tooling.
• Experience working with LLM-based systems (RAG, embeddings, vector databases, prompt orchestration, model routing, etc.)
• Comfortable owning cross-team processes and driving alignment through measurable outcomes and clear technical standards

About us

We embrace our core values in everything that we do:

• Think BIG. We are ambitious, proactive, and we love challenging the status quo.
• We live by truth. We uphold the highest standards of accuracy and accountability.
• On Your Side. We are on this journey together and want to collaborate and empower each other to succeed.
• Committed. Committed to execution. Committed to delivery. Committed to excellence.