Job Description

At Johnson & Johnson, we believe health is everything. Our strength in healthcare innovation empowers us to build a world where complex diseases are prevented, treated, and cured, where treatments are smarter and less invasive, and solutions are personal. Through our expertise in Innovative Medicine and MedTech, we are uniquely positioned to innovate across the full spectrum of healthcare solutions today to deliver the breakthroughs of tomorrow, and profoundly impact health for humanity. Learn more at jnj.com

As guided by Our Credo, Johnson & Johnson is responsible to our employees who work with us throughout the world. We provide an inclusive work environment where each person is considered as an individual. At Johnson & Johnson, we respect the diversity and dignity of our employees and recognize their merit.

Job Function:

Career Programs

Job Sub Function:

Non-LDP Intern/Co-Op

Job Category:

Career Program

All Job Posting Locations:

Titusville, New Jersey, United States of America

About Innovative Medicine

Our expertise in Innovative Medicine is informed and inspired by patients, whose insights fuel our science-based advancements. Visionaries like you work on teams that save lives by developing the medicines of tomorrow.

Join us in developing treatments, finding cures, and pioneering the path from lab to life while championing patients every step of the way.

Learn more at https://www.jnj.com/innovative-medicine

At J&J we are developing Generative AI systems to support scientific work across discovery and translational research, including reports, hypotheses, summaries, and analyses. Many qualities that matter in these outputs — such as scientific plausibility, reasoning quality, insightfulness, novelty, and usefulness for future research — are partly subjective. Expert reviewers may reasonably disagree, and there may be no single ground-truth answer.

This internship will explore how AI judges can be calibrated to different scientific users or reviewer groups so they better reflect expert judgment, uncertainty, and disagreement while remaining grounded in evidence and scientific standards.

The role is intended for a current PhD student with research experience in subjective alignment, human preference modeling, disagreement modeling, LLM-as-judge methods, or related evaluation methods who wants to apply that expertise to a real pharmaceutical R&D problem.

Key Responsibilities

• Investigate how methods for modeling subjective or pluralistic human judgment can be adapted to scientific evaluation contexts.
• Explore approaches for building AI judges that can be calibrated to different scientific users, reviewer groups, or evaluation styles.
• Help define scientific quality beyond correctness, including how an AI judge might assess plausibility, reasoning quality, novelty, usefulness, and research value.
• Compare approaches for representing evaluator uncertainty, disagreement, or multiple valid interpretations.
• Help define a practical use case grounded in accessible data and available expert feedback.
• Build small benchmark datasets or annotation samples that capture expert disagreement on scientific report or hypothesis quality.
• Support the design of evaluation protocols that preserve evaluator uncertainty rather than collapsing feedback into a single score.
• Analyze where AI judge outputs align or diverge from expert reviewers.
• Work with scientific and data science colleagues to gather qualitative feedback and translate it into testable evaluation criteria.
• Participate in reviews with the Evaluation & Standards team to check assumptions against how scientific experts reason about output quality.
• Communicate research tradeoffs, limitations, and findings to both technical and non-technical collaborators.
• Document methods, findings, and open questions so the team can build on the work after the internship.
• Contribute to reusable evaluation assets, such as a calibrated AI judge approach, a disagreement-modeling method, an annotation guide, or a readiness rubric.
• Prepare a final readout summarizing the research approach, findings, limitations, and recommended next steps.

Qualifications

Education

• Currently enrolled in a PhD program in NLP, machine learning, computer science, biomedical informatics, data science, human-centered AI, or a related field.

Required

• Research experience, publications, or active PhD work in one or more of the following areas: subjective alignment, human preference modeling, disagreement modeling, pluralistic alignment, LLM-as-judge methods, model evaluation, or evaluation under uncertainty.
• Proficiency in Python and standard NLP, machine learning, or data science tooling.
• Ability to translate qualitative, disagreement-heavy feedback into testable evaluation criteria, datasets, or analysis plans.
• Ability to analyze model outputs, compare reviewer judgments, and identify patterns of agreement or disagreement.
• Clear written and verbal communication skills, including the ability to explain research framing to scientific collaborators outside NLP or machine learning.
• Interest in applying AI evaluation research to scientific or pharmaceutical R&D problems.

Preferred

• Familiarity with LLM-as-judge approaches, calibration methods, preference modeling, reward modeling, or representation-learning methods for qualitative attributes.
• Exposure to retrieval-augmented generation, grounding, citation evaluation, or context-aware evaluation frameworks.
• Interest or prior exposure to biomedical, scientific, clinical, regulatory, or other expert domains where evaluation requires judgment.
• Experience designing expert review workflows, annotation guides, adjudication processes, or inter-rater agreement analyses.
• Experience working with qualitative feedback, rubric design, benchmark construction, or human evaluation studies.

The expected pay range for this position is between $23.50 per hour and $52.50 per hour but will be based on candidate's program year, discipline, degree and/or experience. Co-Ops/Interns are eligible to participate in Company sponsored employee medical benefits in accordance with the terms of the plan. Co-Ops and Interns are eligible for the following sick time benefits: up to 40 hours per calendar year; for employees who reside in the State of Washington, up to 56 hours per calendar year. Co-Ops and Interns are eligible to participate in the Company’s consolidated retirement plan (pension).

For additional general information on Company benefits, please go to: https://www.careers.jnj.com/employee-benefits

This job posting is anticipated to close on 08/26/2026. The Company may however extend this time-period, in which case the posting will remain available on https://www.careers.jnj.com to accept additional applications.

Ineligible for severance.

If you are under 18 years of age, you (the candidate) may need to obtain the necessary working papers or other documentation required by state law to start the assignment, as well as get a parent’s consent for the background check.

Johnson & Johnson is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, age, national origin, disability, protected veteran status or other characteristics protected by federal, state or local law. We actively seek qualified candidates who are protected veterans and individuals with disabilities as defined under VEVRAA and Section 503 of the Rehabilitation Act.

Johnson & Johnson is committed to providing an interview process that is inclusive of our applicants’ needs. If you are an individual with a disability and would like to request an accommodation, please contact us via https://www.jnj.com/contact-us/careers or contact AskGS to be directed to your accommodation resource.

Required Skills:

Preferred Skills:

The anticipated base pay range for this position is :

$23.50 per hour - $52.50 per hour

Additional Description for Pay Transparency:

The expected pay range for this position is between $23.50 per hour and $52.50 per hour but will be based on candidate's program year, discipline, degree and/or experience. Co-Ops/Interns are eligible to participate in Company sponsored employee medical benefits in accordance with the terms of the plan. Co-Ops and Interns are eligible for the following sick time benefits: up to 40 hours per calendar year; for employees who reside in the State of Washington, up to 56 hours per calendar year. Co-Ops and Interns are eligible to participate in the Company’s consolidated retirement plan (pension). For additional general information on Company benefits, please go to: https://www.careers.jnj.com/employee-benefits This job posting is anticipated to close on 08/26/2026. The Company may however extend this time-period, in which case the posting will remain available on https://www.careers.jnj.com to accept additional applications.