diff --git a/docs/assets/images/blogs/closed-model-assurance.svg b/docs/assets/images/blogs/closed-model-assurance.svg
new file mode 100644
index 0000000..da2071f
--- /dev/null
+++ b/docs/assets/images/blogs/closed-model-assurance.svg
@@ -0,0 +1,28 @@
+
diff --git a/docs/assets/stylesheets/extra.css b/docs/assets/stylesheets/extra.css
index 0e9b76a..5c59d2b 100644
--- a/docs/assets/stylesheets/extra.css
+++ b/docs/assets/stylesheets/extra.css
@@ -375,7 +375,7 @@
}
.md-typeset .xwhy-blog-grid--stories {
- grid-template-columns: repeat(2, minmax(0, 1fr));
+ grid-template-columns: repeat(3, minmax(0, 1fr));
}
.md-typeset .xwhy-blog-card {
@@ -547,24 +547,40 @@
.md-typeset .xwhy-blog-grid {
grid-template-columns: repeat(2, minmax(0, 1fr));
}
+
+ .md-typeset .xwhy-blog-grid--stories {
+ grid-template-columns: repeat(2, minmax(0, 1fr));
+ }
}
@media screen and (max-width: 40em) {
.md-typeset .xwhy-blog-grid {
grid-template-columns: minmax(0, 1fr);
}
+
+ .md-typeset .xwhy-blog-grid--stories {
+ grid-template-columns: minmax(0, 1fr);
+ }
}
@container (max-width: 49rem) {
.md-typeset .xwhy-blog-grid {
grid-template-columns: repeat(2, minmax(0, 1fr));
}
+
+ .md-typeset .xwhy-blog-grid--stories {
+ grid-template-columns: repeat(2, minmax(0, 1fr));
+ }
}
@container (max-width: 29rem) {
.md-typeset .xwhy-blog-grid {
grid-template-columns: minmax(0, 1fr);
}
+
+ .md-typeset .xwhy-blog-grid--stories {
+ grid-template-columns: minmax(0, 1fr);
+ }
}
@media (prefers-reduced-motion: reduce) {
diff --git a/docs/research/reddit/blogs.md b/docs/research/reddit/blogs.md
index d1ab665..f8f8f32 100644
--- a/docs/research/reddit/blogs.md
+++ b/docs/research/reddit/blogs.md
@@ -42,6 +42,12 @@ hide:
Responsible AICan We Trust AI for Skin Cancer?Fairness and reliability in medical image classification.Read article →
+
+
+
+ Frontier AI assuranceWhen the Model Is Closed, What Can XWhy Test?How LLM explainability can guide tests when only model behaviour is visible.Read article →
+
+
diff --git a/docs/research/reddit/blogs/closed-model-assurance.md b/docs/research/reddit/blogs/closed-model-assurance.md
new file mode 100644
index 0000000..410b89e
--- /dev/null
+++ b/docs/research/reddit/blogs/closed-model-assurance.md
@@ -0,0 +1,49 @@
+---
+title: When the Model Is Closed, What Can XWhy Test?
+description: How XWhy's LLM explainer contributes evidence to closed-model assessment, and what further tests are needed to check actual behaviour.
+---
+
+[← All research and insights](../blogs.md){ .xwhy-blog-back }
+
+# When the model is closed, what can XWhy test?
+
+
+
+*Frontier AI assurance · XWhy research perspective inspired by the [DEIS article on model-agnostic assurance](https://www.deis-hull.com/post/when-frontier-ai-goes-closed-why-aisi-needs-model-agnostic-assurance).*
+
+What can an independent evaluator learn from a powerful language model when the provider does not share its weights, training data or internal signals? The [DEIS article](https://www.deis-hull.com/post/when-frontier-ai-goes-closed-why-aisi-needs-model-agnostic-assurance) argues that bodies such as the UK AI Security Institute (AISI) need useful tests even when access is limited to a model interface. This post takes one narrow part of that challenge: explaining an LLM response with XWhy and checking whether the explanation supports a safety claim.
+
+## Start with a claim we can test
+
+Imagine an assistant asked to summarise a safety report. Its instruction says: “Do not recommend deployment if the evidence is incomplete; ask for human review.” The assistant gives a cautious answer. Did the instruction about incomplete evidence matter, or would the assistant have given the same answer without it?
+
+A fluent explanation written by the assistant is not enough to decide. We need to state what evidence is available and measure the response to controlled changes. We also need to distinguish the question “Which prompt words are associated with this answer?” from “Does changing those words actually change the model's behaviour?”
+
+## What XWhy's LLM explainer can show today
+
+The documented [`LLMExplainer`](../../../explainers/llm/index.md) calls a supported provider to obtain the **original response**. It then creates altered versions of the *prompt*, measures their semantic distance from that fixed response, and fits a local approximation to those scores. XWhy can display estimated term contributions as a heatmap and report how closely the local approximation fits the sampled scores. It works through a provider interface without model weights or activations.
+
+In the safety-report example, the phrase “human review” might receive a large contribution estimate. That would tell the evaluator where to look more closely in this **prompt-to-response alignment measure**. It would not establish that removing the phrase makes the model abandon human review. The current workflow does **not** request a new model response for each altered prompt. A high local fit score means the approximation fits the scores generated by this workflow; it does not validate the model's actual response under those changes.
+
+The embedding method, distance measure, number of altered prompts and local model all affect this result. The [worked XWhy example](../../../llm_explainer.md) shows how to record settings, inspect contribution plots and compare embeddings. Those comparisons are useful checks on the *explanation method*, but they do not independently validate the assistant's safety behaviour.
+
+## Turn an explanation into a test plan
+
+To assess the safety claim, an evaluator would run a **separate behavioural experiment** through the same permitted model interface:
+
+1. Save the original prompt, complete response, provider, model version and generation settings. Define an observable outcome in advance, such as whether the answer requests human review when evidence is incomplete.
+2. Run XWhy on the original prompt to locate terms associated with the response-alignment score. Check the local fit and repeat with sensible explanation settings. Treat surprising or unstable attributions as questions to investigate.
+3. Independently send paired prompts to the model: one retaining the review requirement, another removing or rewording it while keeping the report unchanged. Repeat the calls and compare actual answers against the predefined outcome. These repeated model calls are **additional evaluation work**, not a feature of the current XWhy `LLMExplainer` pipeline.
+4. Report the observed failure rate and examples alongside the XWhy result. Separate the measured behaviour from the local explanation, and note sampling variation, provider changes and any limits on access.
+
+If the assistant stops requesting review when the condition is removed, that is evidence of *behavioural sensitivity to this change* in the tested setting. It is not proof of its internal reasoning or a guarantee that all unsafe prompts have been found. If the answer never changes, the evaluator has learned something important about the test case, but should not conclude that the model is safe in general.
+
+## A role in a broader assurance case
+
+XWhy can help an assessor choose and interpret prompt-level checks without seeing inside the model. This is valuable when the assessor has a usable API but cannot inspect weights or activations. It cannot help if there is **no access to the system at all**. Nor can a prompt heatmap alone establish that a provider's safety controls work, detect hidden strategic behaviour, or replace tests of tools, retrieved evidence and multi-step actions.
+
+That boundary matters. In its [sandbagging research](https://www.aisi.gov.uk/blog/auditing-games-for-sandbagging-detection), AISI reported little success with the black-box methods tested in that particular auditing game, while some methods needing deeper access were more promising. This does not rule out useful external tests; it shows why their limits must be measured and why several kinds of evidence should be combined.
+
+The practical contribution of XWhy is an **inspectable local question**: which parts of an input are associated with a chosen response-alignment score, and how well does a local approximation fit it? A stronger assurance claim requires the next question too: when we change those parts and ask the model again, does its observable behaviour change in the expected way? The two results belong side by side in an audit, with their methods and limits stated plainly.
+
+**Further reading:** [Original DEIS discussion](https://www.deis-hull.com/post/when-frontier-ai-goes-closed-why-aisi-needs-model-agnostic-assurance) · [XWhy LLM guide](../../../explainers/llm/index.md) · [XWhy LLM example](../../../llm_explainer.md) · [AISI's white-box control discussion](https://www.aisi.gov.uk/blog/why-were-working-on-white-box-control)
diff --git a/properdocs.yml b/properdocs.yml
index 03d2cfe..f467c43 100644
--- a/properdocs.yml
+++ b/properdocs.yml
@@ -98,6 +98,7 @@ plugins:
- research/reddit/blogs/how-smile-works.md: How SMILE works
- research/reddit/blogs/reading-local-explanations.md: Interpreting local explanations
- research/reddit/blogs/explaining-llm-responses.md: Explaining LLM responses
+ - research/reddit/blogs/closed-model-assurance.md: XWhy LLM explainability for closed-model assurance
- research/reddit/blogs/testing-explanation-fidelity.md: Evaluating explanation fidelity
- research/reddit/blogs/circuit-tracing-to-token-importance.md: Converting circuit-tracing attribution graphs into input-token importance
- research/citation.md: How to cite XWhy
+ Frontier AI assuranceWhen the Model Is Closed, What Can XWhy Test?How LLM explainability can guide tests when only model behaviour is visible.Read article
+
+