What the AI is asked to do
The model acts as an educational assessment expert applying the Thering Framework:
it rates each of the seven criteria on a 1–3 scale
(1 = Ready for Updating, 2 = Partially Aligned & Engaging, 3 = Strongly Aligned & Engaging)
and must reply with structured output: the seven scores, a written explanation per criterion, and a confidence score.
All providers (OpenAI, LM Studio, BullsAI) receive exactly the same prompt — only the model answering it differs.
What information the model sees
- The rating rubric — the seven Thering criteria and the 1–3 scale, embedded in the prompt.
-
The Knowledge Wiki — only if the checkbox is selected.
When “Include LLM Knowledge Wiki in prompt” is checked (it is by default), the server adds
the wiki schema, the wiki index, and every page under
wiki/pages/ to the prompt,
up to a character budget (about 28,000 characters; pages later in the alphabet may be cut if the budget runs out).
The model is told this context informs its reasoning but the rubric remains authoritative.
If unchecked, none of this is sent.
- The assignment itself — the extracted text of your uploaded file (or the description if there is no file content).
- The model's own training knowledge — inherent to any LLM, not controlled by this app.
Note on the wiki: raw PDFs in wiki/raw/ are never sent — only the
human-reviewed synthesis pages written during ingestion. Any new page added to wiki/pages/ is
automatically included in future ratings.
The model does not see: the Faculty Handbook, the Framework page, other assignments or their
ratings, files in wiki/raw/, or anything from the internet at rating time.
How the response is processed
- The server extracts the structured ratings from the model's reply.
- Each rating is validated and clamped to 1–3; a missing or malformed rating defaults to 2.
- The confidence score is clamped to 0–100%; if missing it defaults to 50%.
- If the reply cannot be parsed at all, a text-scanning fallback estimates the ratings and flags itself with 30% confidence.
The result is stored with the provider and model used, and shown alongside manual ratings.
The confidence score is the model's own self-assessment, not a statistical measure —
and the same assignment can get somewhat different ratings on different runs, so the written reasoning
matters more than the exact numbers.