Building AI that can't poison anyone: a dual-layer safety pattern for health apps

A consumer nutrition app generates personalized recipes with AI. Some of its users have anaphylactic allergies. Read those two sentences together and the engineering problem becomes plain: a language model that hallucinates one ingredient is not producing a bug, it is producing a medical event. “We told the model about the allergy in the prompt” is not a safety story anyone should accept.

The short answer

Defense in depth, two independent layers:

1. Pre-check (the instructed layer). The user’s restrictions, with severity levels, are injected into the generation prompt. The model is told what to avoid and why. This layer produces mostly safe output and better recipes, because the model designs around the restriction instead of patching it.
2. Post-check (the deterministic layer). Every ingredient in every generated recipe is string-matched against a static allergen database: seventeen restriction categories covering the FDA’s major allergens and common sensitivities, expanded into hundreds of aliases. Not just “peanut” but derivative ingredients, processing terms, and brand names that contain the allergen. A match blocks or flags the recipe with the specific offending ingredient and severity displayed.

The layers fail independently. The model can ignore its instructions entirely and the post-check still catches the violation, because dictionary lookups do not hallucinate.

Why the alias database is the real work

Anyone can block the word “milk.” The dangerous cases are casein, whey, ghee, and the dozen processed forms that appear in ingredient lists without the trigger word. Building the alias map means thinking like a label reader: derivatives, chemical names, regional terms, brand names. It is unglamorous data work, and it is where the actual safety lives. The database is versioned, reviewed like code, and deliberately static; a safety list that updates itself is a safety list you cannot reason about.

The principle that generalizes

This pattern is not about food. Any AI feature whose failure hurts someone follows the same shape:

• Let the model do what models are good at: generation, adaptation, tone, creativity.
• Let plain code do what code is good at: verification against rules that must hold every single time.
• Never let the model check its own work for the property that matters. A second AI pass is a stronger suggestion, not a guarantee.

In a CRM context the same structure protects pricing promises in AI-drafted emails. In finance, payment instructions. In healthcare, dosage and contraindications. The model proposes; deterministic code disposes.

What to ask of any AI vendor or builder

One question separates production-grade AI from demos: “what happens when the model gets it wrong?” If the answer involves prompt quality, model choice, or testing, keep asking. The only fully satisfying answer names a verification layer that contains no AI at all.