full bug sweep fix

EVALUATIONS.md

@@ -470,7 +470,7 @@ client.evaluations.run(
 ).execute(my_agent_fn)
 ```
 
-or on a live trace from real production traffic, scored continuously by a self-host Online Evaluator:
+or on a live trace from real production traffic, scored continuously by a judge scorer's online profile (self-host):
 
 ```python
 with client.tracer.trace("support-agent", metadata={"promptName": prompt.name}) as span:
@@ -480,7 +480,7 @@ with client.tracer.trace("support-agent", metadata={"promptName": prompt.name})
 From the self-host dashboard: Governance > Manage > **Prompts** > a prompt's row menu > **Suggest
 improvement**. It merges both kinds of evidence - deliberate eval runs (defaulting to the *current
 published version only*, auto-widening to every version if there isn't enough recent evidence yet)
-and worst-scoring Online Evaluator ratings from a recent time window - feeds the worst-rated
+and the worst-scoring ratings a judge scorer's online profile produced in a recent time window - feeds the worst-rated
 examples to a judge, and shows a full rewrite plus reasoning. **Nothing is saved until a human
 approves it as a new version.** The same propose loop is scriptable: `prompts.examples(prompt.id)`
 returns the evidence, `prompts.propose(prompt.id)` asks the judge for a rewrite (returns

TRACING.md

@@ -112,7 +112,7 @@ All parameters work in both decorator and context-manager form - the decorator f
 | `monitor` | `bool` | - | `True` checks this trace against Monitor patterns immediately; `False` opts out of every ingest-time check. Default (`None`) leaves the server's standard behavior. See [Monitor](#monitor) |
 | `pattern_ids` | `list[str]` | - | With `monitor=True`: restrict detection to exactly these pattern ids |
 | `agent_id` | `str` | - | Pin this trace to a known agent id instead of resolving by `name` - a disambiguator for when the name alone isn't enough |
-| `span_kind` | `str` | - | What kind of step this span is (`"agent"`, `"llm"`, `"tool"`, `"retrieval"`, ...), stated instead of left to the backend's classification fallback |
+| `span_kind` | `str` | - | What kind of step this span is (`"agent"`, `"llm"`, `"tool"`, `"retrieval"`, `"memory"`, ...), stated instead of left to the backend's classification fallback |
 
 ### `_TraceSpan` methods and attributes (context manager form)
 
@@ -369,6 +369,19 @@ with tracer.trace("rag-agent") as span:
 
 `tracer.record_retrieval(name, query=..., output=..., duration_ms=...)` is the after-the-fact form. Custom names like `"kb_search"` work - the span carries an explicit retrieval marker, not a name heuristic.
 
+### Memory operations
+
+The memory twins mark a long-term-memory operation (a Mem0/Zep/Letta-style recall or store) as a `span_kind="memory"` child span of the active span. Memory is deliberately NOT retrieval: retrieval spans feed the RAG judges' `{context}` (knowledge grounding), while memory is recalled state.
+
+```python
+with tracer.trace("support-agent") as span:
+    with tracer.trace_memory("user prefs", operation="read", query=user_id) as m:
+        m.output = memory.search(user_id, question)
+    span.output = answer
+```
+
+`tracer.record_memory(name, operation=..., query=..., output=..., duration_ms=...)` is the after-the-fact form. `operation` is free text - conventionally `"read"` or `"write"` - carried in the span's metadata, while the kind itself stays one value so dashboards and scorers can select all memory activity at once. With no active span, both forms queue the record and merge it into the next trace this tracer sends (the patched-client flow where the memory op runs just before a standalone completions call) instead of silently dropping it.
+
 ---
 
 ## Session grouping
@@ -672,6 +685,6 @@ Constructing the client makes no network call; `client.ping()` is the fail-fast
 ## Delivery behavior and limits
 
 - **Queueing** - traces are enqueued (up to 500 in flight) and drained by a background daemon thread. On overflow, or when retries are exhausted, the trace is dropped **with a logged warning** (first drop, then every 50th, with a cumulative count) - never silently.
-- **Retries** - each queued trace is retried up to 3 times with backoff on connection errors, 429, and 5xx responses; a 429's `Retry-After` header is honored. `sync=True` sends block once with a 10s timeout and do not retry - a failed sync send just means `span.trace_id` stays `None`.
+- **Retries** - each queued trace walks the full backoff schedule (up to 3 retries after the first attempt) on connection errors, 429, and 5xx responses alike; a 429's `Retry-After` header is honored in place of the schedule's next wait. `sync=True` sends block with a 10s timeout and retry only briefly - up to 2 bounded retries on 429/503, honoring `Retry-After` (capped at 5s per wait); span ids make redelivery idempotent server-side. A sync send that still fails means the trace was **not stored** (nothing is persisted locally or retried in the background), so `span.trace_id` stays `None`.
 - **Payload truncation** - `input`, `output`, and `metadata` are serialized best-effort before sending: nesting deeper than 4 levels, dicts/lists beyond 30 entries, and unserializable objects are truncated/stringified (long fallback strings cut to 200 chars) to keep payloads bounded.
 - **First failure warns** - the first delivery failure per client logs at WARNING with a hint (bad key vs. bad URL); repeats log at DEBUG. `client.ping()` at startup fails fast instead.

agentx/__init__.py

@@ -12,11 +12,10 @@
     CIGateFailure,
 )
 
-logging.basicConfig(
-    level=logging.INFO,
-    format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
-    datefmt="%Y-%m-%d %H:%M:%S %Z",
-)
+# Library logging hygiene: a library must never call logging.basicConfig - it hijacks the
+# host application's root logger (format AND level) and turns the app's own later basicConfig
+# into a no-op. Consumers opt into our logs with logging.getLogger("agentx").setLevel(...).
+logging.getLogger("agentx").addHandler(logging.NullHandler())
 
 __all__ = [
     "AgentX",

agentx/evaluations/client.py

@@ -348,7 +348,9 @@ def init_run(
             payload["scorerGroupId"] = scorer_group_id
         if split:
             payload["split"] = split
-        data = self._request("POST", "/runs", json=self._with_workspace(payload))
+        # Server-side write: a timeout after the run row was created would be
+        # retried into a duplicate run, so no transport retry.
+        data = self._request("POST", "/runs", json=self._with_workspace(payload), retry=False)
         return EvaluationRun(**data)
 
     def append_results(
@@ -432,8 +434,12 @@ def analyze_run(
 
         if not self._analysis_on_dashboard_router:
             try:
+                # The self-host route runs the analysis SYNCHRONOUSLY (engine
+                # routes/evaluations.ts) - a short timeout with retries re-billed the whole
+                # multi-judge analysis up to 4x while the first was still running. Full
+                # analysis timeout, no transport retry.
                 return self._request(
-                    "POST", f"/runs/{run_id}/analyze", json=payload, timeout=30
+                    "POST", f"/runs/{run_id}/analyze", json=payload, timeout=1800, retry=False
                 )
             except AgentXEvaluationsError as exc:
                 if not self._note_missing_analysis_route(exc, "analyze"):
@@ -554,13 +560,17 @@ def _report_from_dashboard(self, run_id: str) -> Report:
         if isinstance(dataset_id, dict):  # populated reference, not a bare id
             dataset_id = dataset_id.get("_id") or dataset_id.get("id")
 
-        return Report(
-            runId=run_id,
-            datasetId=dataset_id or "",
-            status=envelope.get("status") or "completed",
-            statistics=envelope.get("statistics"),
+        # Built as one merged dict (explicit keys last, so they win) - passing the
+        # explicit keys as keyword arguments alongside **body raises "got multiple
+        # values" whenever the analysis body itself carries runId/datasetId/status/
+        # statistics.
+        return Report(**{
             **body,
-        )
+            "runId": run_id,
+            "datasetId": dataset_id or "",
+            "status": envelope.get("status") or "completed",
+            "statistics": envelope.get("statistics"),
+        })
 
     # ------------------------------------------------------------------
     # Prompt improvement loop (examples -> propose -> publish). These ride the engine's
@@ -589,8 +599,11 @@ def publish_prompt_version(
             payload["reasoning"] = reasoning
         if based_on_version is not None:
             payload["basedOnVersion"] = based_on_version
+        # Server-side write: a timeout after the version was stored would be
+        # retried into a duplicate version, so no transport retry.
         return self._request(
-            "POST", f"/evaluate/prompts/{prompt_id}/versions", base=self._api_root, json=payload
+            "POST", f"/evaluate/prompts/{prompt_id}/versions", base=self._api_root, json=payload,
+            retry=False,
         )
 
     # ------------------------------------------------------------------
@@ -628,7 +641,7 @@ def compare_pairwise(
             payload["judgeModel"] = judge_model
         if both_orders:
             payload["bothOrders"] = True
-        response = self._request("POST", "/evaluate/runs/pairwise", json=payload, base=self._api_root)
+        response = self._request("POST", "/evaluate/runs/pairwise", json=payload, base=self._api_root, timeout=900, retry=False,)
         return PairwiseComparison(**response["comparison"])
 
     def get_pairwise(self, batch_id: str) -> PairwiseComparison:
@@ -662,7 +675,8 @@ def create_tool_schema(self, *, name: str, definition: str, description: Optiona
         payload: dict = {"name": name, "definition": definition}
         if description is not None:
             payload["description"] = description
-        return self._request("POST", "/evaluate/tool-schemas", base=self._api_root, json=payload)
+        # Server-side write - no transport retry (see init_run's comment).
+        return self._request("POST", "/evaluate/tool-schemas", base=self._api_root, json=payload, retry=False)
 
     def get_tool_schema_examples(self, tool_schema_id: str, window: Optional[str] = None) -> dict:
         params = {"window": window} if window else None
@@ -686,8 +700,10 @@ def publish_tool_schema_version(
             payload["reasoning"] = reasoning
         if based_on_version is not None:
             payload["basedOnVersion"] = based_on_version
+        # Server-side write - no transport retry (see init_run's comment).
         return self._request(
-            "POST", f"/evaluate/tool-schemas/{tool_schema_id}/versions", base=self._api_root, json=payload
+            "POST", f"/evaluate/tool-schemas/{tool_schema_id}/versions", base=self._api_root, json=payload,
+            retry=False,
         )
 
     # ------------------------------------------------------------------

agentx/evaluations/datasets.py

@@ -16,7 +16,13 @@
 
 
 class DatasetBuilder:
-    """Fluent builder for creating a Custom Agent Evaluations dataset."""
+    """Fluent builder for creating a Custom Agent Evaluations dataset.
+
+    ``judge_prompt``/``judge_model`` are LLM-as-judge overrides for the dataset's grading
+    config. NOTE (self-host): the engine's dataset-create route currently ignores both -
+    set them on a judge scorer / the evaluation settings instead. ``sovereignty_models``
+    is accepted on the wire but not acted on by the self-host engine.
+    """
 
     def __init__(
         self,
@@ -50,6 +56,8 @@ def __init__(
         # LLM-as-judge overrides for this dataset's own grading config. Omit either to keep the
         # server default (raw prompt template / gpt-5.6-luna, see EVALUATIONS.md). judge_model
         # must be one of client.evaluations.list_models() (OpenAI or Anthropic).
+        # Self-host: the dataset-create route currently IGNORES judgePrompt/judgeModel - set
+        # them on a judge scorer / the evaluation settings instead (see class docstring).
         if judge_prompt is not None:
             self._payload["judgePrompt"] = judge_prompt
         if judge_model is not None:
@@ -84,7 +92,8 @@ def __init__(
         if rouge_score:
             self._payload["rougeScore"] = {"enabled": True}
         # Sovereignty & Portability - the models to compare on this dataset (use
-        # client.evaluations.list_models() to discover valid ids).
+        # client.evaluations.list_models() to discover valid ids). Self-host: accepted on
+        # the wire but not acted on by the engine (see class docstring).
         if sovereignty_models:
             self._payload["sovereigntyIndex"] = {
                 "enabled": True,
@@ -354,10 +363,13 @@ def import_dataset(self, source: Any, name: Optional[str] = None) -> Dataset:
             "acceptanceCriteria",
             "rejectionCriteria",
             "evaluationCriteria",
+            "judgePrompt",
+            "judgeModel",
             "vectorSimilarity",
             "jaccardSimilarity",
             "bleuScore",
             "rougeScore",
+            "sovereigntyIndex",
             "codeScorers",
         ):
             if wire.get(key) is not None:

agentx/evaluations/reporting.py

@@ -1,5 +1,7 @@
 from __future__ import annotations
 
+from typing import Optional
+
 from agentx.evaluations.models import Report
 from agentx.evaluations._term import (
     bold,
@@ -21,14 +23,14 @@
 _PRI_COLORS = {"high": red, "medium": yellow, "low": dim}
 
 
-def _rating_badge(rating: str | None) -> str:
+def _rating_badge(rating: Optional[str]) -> str:
     icon = _RATING_ICONS.get(rating or "", "·")
     color = _RATING_COLORS.get(rating or "", dim)
     label = (rating or "").upper()
     return color(f"{icon} {label}") if label else dim(icon)
 
 
-def _section(title: str, rating: str | None = None) -> None:
+def _section(title: str, rating: Optional[str] = None) -> None:
     badge = f"  {_rating_badge(rating)}" if rating else ""
     print(f"\n{bold(title)}{badge}")
     print(dim(_THIN))

agentx/evaluations/runner.py

@@ -3,6 +3,8 @@
 import logging
 import os
 import time
+
+import requests
 import uuid
 from typing import Any, Callable, Dict, Iterator, List, Optional, Set, Union
 
@@ -304,6 +306,12 @@ def _flush_batch(self, batch: List[EvaluationResult]) -> None:
                         resp.failed_validation,
                     )
                     return
+                except requests.Timeout as exc:
+                    # A read timeout means the engine may STILL be scoring this batch - a
+                    # retry re-POSTs it and double-bills every judge call (idempotency keys
+                    # protect rows already inserted, not judge work mid-flight). Fail loud.
+                    last_exc = exc
+                    break
                 except Exception as exc:
                     last_exc = exc
                     if attempt == 1:

agentx/integrations/anthropic.py

@@ -274,7 +274,7 @@ def __iter__(self_inner):
                 return iter(ctx)
 
             def __aiter__(self_inner):
-                return aiter(ctx)
+                return ctx.__aiter__()  # aiter() builtin is 3.10+; python_requires is >=3.9
 
             def __getattr__(self_inner, item):
                 return getattr(ctx, item)

agentx/integrations/autogen.py

@@ -167,6 +167,9 @@ def _summarize_messages(self, messages: List[Any], run_start: float) -> tuple:
                         "end_time": end_t,
                         "input": pending["input"] if pending else None,
                         "output": f"ERROR: {output}" if is_error else (str(output) if output is not None else None),
+                        # The engine's failure test is success === false; without
+                        # this a failed tool call would read as passing.
+                        "success": not is_error,
                     })
                 continue
 

agentx/integrations/crewai.py

@@ -127,6 +127,16 @@ def _start_task_timing_capture(self):
         except ImportError:
             return task_timings, lambda: None
 
+        # Double-instrumentation guard (bus-keyed latch, the same idea as the
+        # other integrations' _agentx_patched flag): the event bus is a global
+        # singleton, so a notebook re-run or an overlapping kickoff that
+        # already has AgentX listeners registered would otherwise get a second
+        # set and duplicate every task span. When already attached, this
+        # kickoff just falls back to the evenly-divided timing approximation.
+        if getattr(crewai_event_bus, "_agentx_attached", False):
+            return task_timings, lambda: None
+        crewai_event_bus._agentx_attached = True
+
         def on_task_started(source: Any, event: Any) -> None:
             task_id = getattr(event, "task_id", None)
             if task_id is None:
@@ -154,9 +164,14 @@ def on_task_failed(source: Any, event: Any) -> None:
         crewai_event_bus.on(TaskFailedEvent)(on_task_failed)
 
         def unregister() -> None:
-            crewai_event_bus.off(TaskStartedEvent, on_task_started)
-            crewai_event_bus.off(TaskCompletedEvent, on_task_completed)
-            crewai_event_bus.off(TaskFailedEvent, on_task_failed)
+            try:
+                crewai_event_bus.off(TaskStartedEvent, on_task_started)
+                crewai_event_bus.off(TaskCompletedEvent, on_task_completed)
+                crewai_event_bus.off(TaskFailedEvent, on_task_failed)
+            finally:
+                # Clear the latch even if .off() raises, so a later kickoff
+                # can re-attach instead of being locked out forever.
+                crewai_event_bus._agentx_attached = False
 
         return task_timings, unregister