Job Description

At Lilly, we unite caring with discovery to make life better for people around the world. We are a global healthcare leader headquartered in Indianapolis, Indiana. Our employees around the world work to discover and bring life-changing medicines to those who need them, improve the understanding and management of disease, and give back to our communities through philanthropy and volunteerism. We give our best effort to our work, and we put people first. We’re looking for people who are determined to make life better for people around the world.

This role sits at the intersection of GenAI and pharmaceutical learning — a rare opportunity to own AI avatar behavior in a highly regulated, high-stakes environment.

The Conversational AI Platform Owner is the single point of operational accountability for the LillyAvatarAssistant system. The role spans four connected disciplines: prompt and configuration management across the full prompt suite (Conversation, Evaluation, Compliance, and Drug Information); LLM model configuration and versioning; publishing and release operations through the AI Avatar Publishing Platform; and production support including incident response and rollback execution.

This is not a backend development role. It requires technical depth to manage prompts, model parameters, multi-language configurations, compliance-sensitive scoring logic, and structured release processes — while operating within a validated, change-controlled pharmaceutical environment.

Key Responsibilities

1. Prompt Suite Ownership

• Own and maintain the full prompt suite — Conversation Prompts, Evaluation Prompts, Compliance Prompts, and Drug Information Prompts — including their associated criteria and scoring logic
• Maintain persona definitions, system messages, tone alignment, and behavioral reliability across all avatar configurations

2. LLM Model Configuration & Versioning

• Manage LLM model configurations including inference parameters (temperature, top-p, max tokens), safety/guardrail settings, and content filter configurations
• Maintain a model inventory (“model cards”) documenting model name/version, primary use, key settings, known limitations, and failure modes
• Follow a disciplined model/prompt change process: define intent, curate golden test sets, validate, document evidence, confirm rollback path

3. Publishing Platform Operations

• Create, edit, test, publish, and unpublish Avatars through the full Publishing Platform lifecycle
• Manage Avatar Characteristics, Site Content, prompt configurations, and Test & Review workflows
• Manage Drafts and Published Avatars pages including edit, test, publish/unpublish operations

4. Testing & Validation

• Execute layered testing: developer checks, integration tests, LLM evaluation harnesses, manual smoke tests, and full UAT regression (per the established UAT checklist)
• Develop and maintain golden conversation test sets for behavioral validation after model or prompt changes
• Collect and document evidence: screenshots, transcript snippets, before/after comparisons, CI run links
• Run UAT regression before releases or major configuration changes

5. Release Discipline & Stakeholder Sign-Offs

• Drive the stage-gated release process: implement in non-prod, smoke test, deploy to draft, execute UAT, collect sign-offs, promote to production
• Coordinate Business Stakeholder and System Owner acknowledgements per the established sign-off framework
• Ensure change request documentation and evidence packages are complete per change management policy
• Own post-release monitoring: confirm error rate, latency, and key user journeys are within expected thresholds

6. Production Support & Incident Response

• Triage production issues using observability tools (logs, traces, metrics) to isolate avatar session or evaluation failures
• Execute emergency response: unpublish problematic Avatars, coordinate platform-wide rollbacks with Development Leads
• Document incident timeline, root cause, and preventive actions
• Maintain a list of “known-good” Avatars for regression testing and track platform dependencies (LLM provider status, auth/SSO, email service)

7. Documentation & Governance

• Maintain prompt libraries, model cards, runbooks, architecture references, and release checklist templates
• Follow governance practices including approvals, change tracking, and audit trails
• Capture knowledge artifacts that reduce bus factor: architecture diagrams, data flows, monitoring dashboard links, alerting thresholds

8. Cross-Functional Collaboration

• Partner with Product Owner, Tech Lead, Program Management, Development Leads, and vendors through iteration and release cycles
• Support issue resolution by interpreting API responses and troubleshooting configuration and publishing errors in collaboration with Tech teams
• Understand how avatar configurations connect to backend data stores (e.g., Azure Cosmos DB) and how API endpoints are managed (e.g., Azure APIM) to support troubleshooting and change impact assessment

Required Skills

Prompt & Content

Must Have

Strong writing and editing skills for AI avatar copy (dialogue, tone, persona, system messages)

Solid understanding of prompt engineering fundamentals: tone control, accuracy, consistency, behavioral alignment, and prompt layering conventions (system prompt → avatar-level → turn-level)

Ability to own evaluation, compliance, and drug information prompt logic — including criteria management and scoring alignment

Familiarity with conversational design principles and voice/tone systems

Experience writing structured content for chat-based experiences (assistants, guided flows, role-play personas)

Familiarity with localization workflows: managing translation content, pronunciation guidance, and locale-specific access control across multiple languages

Platform & Operations

Must Have

Comfort executing UAT cycles and validating outputs using structured test cases and edge-case thinking

Experience with production support processes: incident triage, severity assessment, escalation paths, rollback execution, and root cause documentation

Comfort navigating observability and monitoring tools (e.g., Splunk, Datadog, App Insights, Grafana) to locate logs/traces and diagnose production issues

Understanding of release discipline: stage-gated deployments, evidence collection, stakeholder sign-off coordination, and change management compliance

Exposure to compliance-oriented content workflows and approval processes in regulated environments

Technical Foundations

Must Have

Working knowledge of LLM inference settings (temperature, top-p, max tokens, frequency/presence penalties) and their behavioral impact on conversation quality and scoring accuracy

Strong working knowledge of Markdown and HTML fundamentals for publishing, content rendering, and documentation formatting

Comfort working with structured configurations (JSON/YAML) and API-based workflows (e.g., Postman, cURL)

Understanding of versioning principles: change logs, audit trails, rollback procedures, and configuration version management

Understanding of role-based access control principles as applied to publishing platforms and content governance

High attention to detail with ability to prevent regressions and unintended behavior shifts

Nice to Have

• Experience working with Conversational AI platforms (chatbots, virtual assistants, AI avatars, or agent-based systems)
• Experience with LLM output evaluation methods: golden-set testing, before/after behavioral comparison, and scoring validation using evaluation harnesses
• Basic understanding of Retrieval-Augmented Generation (RAG) concepts including document ingestion, embeddings, and search behavior
• Exposure to cloud-based data integrations (Azure preferred); willingness to learn platform-specific configurations
• Experience with pharmaceutical or healthcare compliance frameworks

Education & Qualifications

Required

• Bachelor’s degree in Computer Science, Information Technology, Computational Linguistics, Human-Computer Interaction, Digital Media, or a related technical/interdisciplinary field
• 7–10 years of professional experience across one or more of: conversational AI, prompt engineering, NLP/NLU platforms, content operations for AI systems, or technical product/platform operations

Preferred

• Master’s degree in AI/Machine Learning, Computational Linguistics, Cognitive Science, Learning Technologies, or a related advanced discipline
• Professional certifications in cloud platforms (e.g., Azure AI Fundamentals, Azure AI Engineer Associate) or AI/ML (e.g., Google Professional ML Engineer, AWS Machine Learning Specialty)
• Formal training or certification in prompt engineering, conversational design, or LLM application development
• ITIL or similar certification demonstrating familiarity with service management, change control, and incident management frameworks

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form ( https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

#WeAreLilly