Making every stage of medical device document assessment with large language models deterministic

Key points

• To show that a medical device complies with regulation, experts judge whether documents and records meet each requirement and record the judgment for others to verify.1
• For each assessment item that experts define, large language models could prepare a preliminary assessment: a label such as Satisfied or Not satisfied, a reason and quoted evidence.1
• Reducing reasoning where the interface allows it did not reduce the variation, and prompts of retrieved passages removed it only for the smallest models.1
• The same open-weight model was reproducible or not depending only on how it was served.1

Sources (1)

• [1]Making every stage of medical device document assessment with large language models deterministic
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 05:45 AM

  A pipeline we built to be deterministic at every stage -- text extraction and optical character recognition (OCR), chunking, embedding, indexing, retrieval and inference -- returned byte-identical output at each stage across reruns, interruptions, edits, machines of one graphics processing unit (GPU) model and days, for blood glucose meter and infusion pump documents.

  To show that a medical device complies with regulation, experts judge whether documents and records meet each requirement and record the judgment for others to verify.

Extractive summary: sentences quoted from the sources.