NutriTwin — Feed-Ingredient Ontology & Knowledge Graph (WP2)

A tool for the NutriTwin WP2 work package (pig nutrition, cSBO / Flanders' FOOD) that turns heterogeneous feed-ingredient data into a single, editable knowledge graph and exports it as RDF/Turtle. It ingests two sources — the CVB Feed Table 2023 (parsed deterministically from a PDF) and feedtables.com / INRA (2024) (HTML scrape) — canonicalizes their nutrients and units, lets an operator confirm cross-source ingredient matches, builds an editable graph, and ships it as .ttl with a SPARQL playground. The ontology is aligned to the Hestia Earth schema.

Everything runs locally through one Flask console (web/app.py) plus a few standalone CLI modules.

 ① PARSE (offline CLI)      ② MAP  (web /mapper)             ③ EDITABLE KG (web /kg)    ④ SHIP (web /ship)
 raw PDF / web → *.pkl  →   Nutrients → Units → Matching  →  kg.pkl {nodes,edges}   →   kg.ttl + SPARQL
 (READ-ONLY parsed data)    (operator confirms suggestions)  (merge / delete / edit)    (pkl→ttl via rdflib)

---

Quick start

# dependencies (no lockfile is committed)
pip install flask rdflib pypdf openai requests sentence-transformers numpy

# run the console (default http://0.0.0.0:5000)
python web/app.py
python web/app.py --port 8080 --debug     # --host / --debug also available

Then open http://localhost:5000. Parsing, mapping, KG editing, and read-only browsing all work without any API key; only the AI helpers need one (see AI features).

Git LFS: data/processed/kg.ttl (and any *.owl) is stored via Git LFS — run git lfs install after cloning, or that file will be a pointer stub.

---

Repository layout

NT_01_01_Parser_CVB.py                 ① CVB Feed Table 2023 PDF → data/raw/raw_results/cvb.pkl  (pypdf)
NT_01_01_Parser_CVB_Abbreviations.py   CVB abbreviation → (full name, base unit) lookup + enrich_entry()
NT_01_02_Parser_Feedtables.py          ① feedtables.com scrape → data/raw/raw_results/feedtables.pkl
NT_01_03_Source_Descriptions.py        ① ingredient descriptions → data/processed/source_descriptions.json
NT_02_01_Mapper_Nutrients.py           ② canonical nutrient names → data/processed/nutrient_canonical.json
NT_02_02_Unit_Canonicalization.py      ② unit-conversion helpers (imported by the web app)
NT_02_03_Ingredient_Match.py           ② cross-source ingredient similarity (imported by the web app)

web/
  app.py                               single-file Flask console — all of stages ②③④ + the HTTP API
  static/{app.js, style.css}           app.js = shared AI model picker; style.css = UGent design tokens
  templates/{base,pipeline,mapper,kg,vocab,ship}.html

data/
  raw/                                 SOURCE — read-only, never edited
    extracted-cvb-feed-table-20232.pdf   the upstream CVB document
    raw_results/{cvb,feedtables}.pkl     parsed records (the "Parsed Data" ground truth)
  processed/                           DERIVED — regenerable, committed so the shipped KG is versioned
    nutrient_canonical.json              canonical nutrient-name map          (Nutrients mapper)
    unit_rules.json                      unit-conversion rules                (Units mapper)
    ingredient_matches.json              confirmed cross-source pairs         (Matching)
    source_descriptions.json             editable ingredient descriptions     (Mapper — not the raw pkls)
    sources.json                         source id → display label
    kg.pkl                               the editable working KG {nodes, edges, predicates}
    kg.ttl                               Turtle export of kg.pkl              (Git LFS)
    kg_vocabulary.json                   controlled statement vocabulary (allowed predicates + object classes)

api_key.txt                            OpenAI key (gitignored; or use the OPENAI_API_KEY env var)

Naming convention: every root Python module is NT_<stage>_<order>_<Name>.py — 01_* are parsers, 02_* are the three Mapper steps. The web app is kept to a single file.

---

The four stages

① Parse — offline, deterministic

Run the parsers directly; they write the read-only Parsed Data layer:

python NT_01_01_Parser_CVB.py          # PDF → raw_results/cvb.pkl (uses the abbreviations lookup)
python NT_01_02_Parser_Feedtables.py   # web scrape → raw_results/feedtables.pkl
python NT_01_03_Source_Descriptions.py # → data/processed/source_descriptions.json  (--force, --limit)

CVB ships no explicit units, so NT_01_01_Parser_CVB_Abbreviations.py resolves each abbreviation to a full name and a section-aware base unit. NT_01_03 fetches ingredient descriptions (feedtables: the AFZ prose on each page; CVB: a placeholder — currently the ingredient name — until the CVB description API exists; the two are kept strictly separate). These .pkl files are the authoritative input to everything downstream and are inspected — never edited — through the web reader at / (a.k.a. /pipeline).

② Map — three steps (web /mapper)

• Nutrients (NT_02_01_Mapper_Nutrients.py): assigns every nutrient across both sources one flat canonical name (e.g. CVB LYS + feedtables Lysine → lysine_sid_pig_as_fed) by parsing facets (method / species / basis / metric / year) from the section keys. Offline sentence-transformer embeddings only suggest merges for a human to confirm. Output: nutrient_canonical.json. Re-run automatically on source import; edited via the Nutrients tab.
• Units (NT_02_02_Unit_Canonicalization.py): converts each value into its canonical's unit. Mass fractions convert exactly (% ↔ g/kg ↔ mg/kg); energy pairs (MJ ↔ kcal) are deliberately not converted. Rules live in unit_rules.json, editable in the Units tab.
• Matching (NT_02_03_Ingredient_Match.py): finds cross-source ingredient matches. Each ingredient becomes a z-score-normalized vector over canonical nutrients; similarity is exp(-rms) over the co-measured overlap, with an optional description-similarity component on top. Results are decision support — the operator confirms each match. Confirmed pairs go to ingredient_matches.json.

③ Editable KG (web /kg)

Confirmed matches and imports build an editable graph in data/processed/kg.pkl ({nodes, edges, predicates}: ingredients, per-source instances, object classes, statements, reuses, distinguishers). It is edited live in the browser — add/remove triples, merge or delete nodes, rename classes/predicates, manage skeletons, and accept AI-suggested statements. Statements draw from a controlled vocabulary (/vocab → kg_vocabulary.json): predicates and object classes are chosen from a fixed list; neither the operator nor the AI may invent new terms.

④ Ship (web /ship)

Serializes kg.pkl → data/processed/kg.ttl with rdflib (namespace nt: <https://nutritwin.ugent.be/kg#>, one node per instance, camelCase identifiers). The Ship page also provides a SPARQL playground (run SELECT queries over the shipped graph) and a natural-language → SPARQL helper.

---

Web app

All pages extend base.html (UGent-branded top nav). Routes:

Page	Purpose
/ (/pipeline)	Parsed Data reader — inspect the read-only *.pkl records
/mapper	Mapper — Nutrients / Units / Matching tabs
/kg	Editable KG — triples, instances, skeletons, AI statement suggestions
/vocab	Controlled statement vocabulary editor
/ship	Ship — pkl→ttl, SPARQL playground, NL→SPARQL

HTTP API groups (JSON): /api/parsed/* (read parsed pkls), /api/canon/* (nutrient mapper), /api/units/*, /api/match/* (ingredient matching + nutrient comparison), /api/sources/*, /api/kg/* (edit the pkl graph), /api/ship/* (ttl / sparql / ai-sparql), and /api/ai/models.

The frontend is Flask + Jinja + vanilla JS (no framework). web/static/style.css defines UGent design tokens (primary blue #1E64C8, yellow #FFD200) used throughout; the app is single (light) theme.

---

AI features

• Four in-page "easy" AI buttons (Mapper nutrient describe / merge-suggest, KG describe / statement-suggest) prefer a local LM Studio server (OpenAI-compatible, no token spend) and fall back to OpenAI only if it's unreachable. LM Studio is auto-probed at localhost:1234 and the WSL2 default gateway; override with LMSTUDIO_BASE_URL / LMSTUDIO_MODEL.
• Ship NL→SPARQL (/api/ship/ai-sparql) uses OpenAI only (the hardest task); the schema brief is derived from the KG. Every AI button has a ▾ model picker; the choice is remembered per browser.
• Key: put it in api_key.txt at the repo root (preferred) or set OPENAI_API_KEY. It is only needed for NL→SPARQL and the OpenAI fallback; the rest of the pipeline runs without one. api_key.txt is gitignored.

---

Notes

• Semantic warm-up: on launch the app preloads the all-MiniLM-L6-v2 sentence-transformer (in a background thread) so the first Matching click is instant. If the model is missing, semantic matching silently stays off.
• Derived files are committed (data/processed/*) so the shipped KG is versioned, even though they can be regenerated from the parsers and Mapper.
• No automated test suite ships with the repo; changes are verified by driving the running app.
• Requires Python 3.10+. The live AI client is openai (also used to talk to LM Studio) — there is no anthropic dependency.