Test Assignment SCM

What the AI is asked to do

The model acts as an educational assessment expert applying the Thering Framework: it rates each of the seven criteria on a 1–3 scale (1 = Ready for Updating, 2 = Partially Aligned & Engaging, 3 = Strongly Aligned & Engaging) and must reply with structured output: the seven scores, a written explanation per criterion, and a confidence score. All providers (OpenAI, LM Studio, BullsAI) receive exactly the same prompt — only the model answering it differs.

What information the model sees

1. The rating rubric — the seven Thering criteria and the 1–3 scale, embedded in the prompt.
2. The Knowledge Wiki — only if the checkbox is selected. When “Include LLM Knowledge Wiki in prompt” is checked (it is by default), the server adds the wiki schema, the wiki index, and every page under wiki/pages/ to the prompt, up to a character budget (about 28,000 characters; pages later in the alphabet may be cut if the budget runs out). The model is told this context informs its reasoning but the rubric remains authoritative. If unchecked, none of this is sent.
3. The assignment itself — the extracted text of your uploaded file (or the description if there is no file content).
4. The model's own training knowledge — inherent to any LLM, not controlled by this app.

The model does not see: the Faculty Handbook, the Framework page, other assignments or their ratings, files in wiki/raw/, or anything from the internet at rating time.

How the response is processed

• The server extracts the structured ratings from the model's reply.
• Each rating is validated and clamped to 1–3; a missing or malformed rating defaults to 2.
• The confidence score is clamped to 0–100%; if missing it defaults to 50%.
• If the reply cannot be parsed at all, a text-scanning fallback estimates the ratings and flags itself with 30% confidence.

The result is stored with the provider and model used, and shown alongside manual ratings. The confidence score is the model's own self-assessment, not a statistical measure — and the same assignment can get somewhat different ratings on different runs, so the written reasoning matters more than the exact numbers.