Alto 5590

README.md

@@ -25,7 +25,7 @@
 
 Official Python SDK for the [Behavioral Signals API](https://behavioralsignals.readme.io/).
 
-Analyze human behavior and detect deepfake speech using batch and real-time audio APIs. Experimental video deepfake detection is also available in batch mode.
+Estimate behavioral signals from speech and detect deepfake speech using batch and real-time audio APIs. Experimental video deepfake detection is also available in batch mode.
 
 [Python SDK Documentation](https://behavioralsignals.readme.io/docs/behavioral-signals-python-sdk) ·
 [Examples](examples/) ·
@@ -34,7 +34,7 @@ Analyze human behavior and detect deepfake speech using batch and real-time audi
 
 ## Features
 
-- **Behavioral Analysis** — analyze human behavior from speech in batch and real-time streaming modes
+- **Behavioral Analysis** — estimate behavioral signals from speech in batch and real-time streaming modes
 - **Deepfake Detection** — detect synthetic or manipulated speech in batch and real-time streaming modes
 - **Video Deepfake Detection (Experimental, Batch Only)** — analyze both the video frames and audio track of supported video files
 - **Core Speech Attributes (Batch Only)** — automatic speech recognition (ASR), speaker diarization, and language identification
@@ -96,7 +96,7 @@ print_results(result.results)
 
 `upload_audio()` returns a process with a unique process ID (`pid`). `wait_for_result()` polls until processing completes and returns the analysis result.
 
-`print_results()` prints one line per result: the start and end time in seconds, the task, and the top label with its probability.
+`print_results()` prints one line per result: the start and end time in seconds, the task, and the final label with its probability.
 Continuous tasks such as `intensity` have no label, only a score, and `asr` and `diarization` have a label without a probability.
 For a 10-second clip of one speaker, the output starts like this:
 
@@ -122,6 +122,52 @@ process = client.behavioral.upload_audio(
 )
 ```
 
+### Behavioral outputs
+
+The Behavioral API returns **24 signals** across 8 dimensions: 18 behavioral signals and 6 speaker signals. Each label is its own signal with its own probability.
+All eight dimensions are scored together on every utterance longer than 1 second, so one utterance can be *sad + weak + negative + slow + hesitating + withdrawn*, with a voice estimated as female and aged 31 - 45. That gives 4 × 3 × 3 × 3 × 2 × 3 × 2 × 4 = **5,184 possible profiles** per utterance.
+
+| Task | What it measures | Labels | # |
+|---|---|---|---|
+| `emotion` | Basic emotion in the voice | `happy`, `angry`, `sad`, `neutral` | 4 |
+| `strength` | Arousal: energy in the voice | `strong`, `weak`, `neutral` | 3 |
+| `positivity` | Valence: sentiment of the tone | `positive`, `negative`, `neutral` | 3 |
+| `speaking_rate` | How fast the speaker talks, compared to speakers in general, not to their own pace | `fast`, `slow`, `normal` | 3 |
+| `hesitation` | Signs of hesitation | `yes`, `no` | 2 |
+| `engagement` | Involved or detached tone | `engaged`, `withdrawn`, `neutral` | 3 |
+| `gender` | Sex of the speaker, estimated from the voice | `female`, `male` | 2 |
+| `age` | Age range of the speaker, estimated from the voice | `18 - 22`, `23 - 30`, `31 - 45`, `46 - 65` | 4 |
+| **Total** | | | **24** |
+
+Plus `intensity`: how intense the emotion is, as one score between 0 and 1. It is not a score for any one emotion, even when `emotion` is `angry`. It has no labels and is not counted in the 24.
+In batch results, utterances of 1 second or less return only `diarization`, `asr` and `language`. When streaming, every segment gets all 24 signals and `intensity`, however short it is.
+In batch results, `gender` and `age` are given per speaker: every utterance of a speaker longer than 1 second gets the same values, averaged over the file. When streaming, `gender` and `age` are estimated on each segment.
+The other tasks have open-ended values, not fixed labels: `diarization` (a speaker ID such as `SPEAKER_00`), `asr` (the transcript) and `language`. With `embeddings=True`, `features` holds the behavioral embedding.
+All results of one utterance share its `id`, `startTime` and `endTime`. To split results by speaker, use the `diarization` label of each utterance. Speaker IDs are generic: the API does not say who is the agent and who is the customer.
+
+`neutral` means something different in each task, so always read a label together with its task: `emotion: neutral` means no clear emotion, `strength: neutral` normal energy, `positivity: neutral` neither positive nor negative, and `engagement: neutral` neither engaged nor withdrawn.
+
+`finalLabel` gives one answer per task. For behavioral tasks it is the label with the highest probability.
+When you need a score instead of a label, use the probabilities in `prediction`. They are strings, so convert them with `float()`.
+For each task in the table they add up to 1, so each label's probability is a score from 0 to 1 that you can track over time, average over a call, or compare against your own threshold.
+For example, the `angry` probability of `emotion` is a score for anger. `intensity` is not: it does not say which emotion is intense.
+
+`positivity` is valence and `strength` is arousal, the two dimensions often used in emotion research. The API gives them as probabilities from 0 to 1. To put them on the usual scale from -1 to 1, subtract the probabilities of the two opposite labels:
+
+```python
+for item in result.results:
+    if item.task in ("positivity", "strength"):
+        scores = {p.label: float(p.posterior) for p in item.prediction}
+        if item.task == "positivity":
+            print(item.st, "valence", scores["positive"] - scores["negative"])
+        else:
+            print(item.st, "arousal", scores["strong"] - scores["weak"])
+```
+
+All probabilities are model estimates: use them to compare utterances and follow trends, not as exact measurements.
+
+See [Definition of behaviors](https://behavioralsignals.readme.io/docs/definition-of-behaviors) for what each signal means.
+
 ## Deepfake Detection
 
 ### Audio
@@ -383,8 +429,8 @@ The server exposes four tools:
 | Tool | What it does |
 |---|---|
 | `analyze_behavior` | Uploads an audio file or S3 presigned URL for behavioral analysis and returns the results |
-| `detect_deepfake` | Uploads an audio or video file, or an S3 presigned URL, for deepfake detection and returns the results |
-| `get_result` | Returns the results of a process, in pages, optionally filtered to specific tasks |
+| `detect_deepfake` | Uploads an audio or video file, or an S3 presigned URL, for deepfake detection (video is experimental) and returns the results |
+| `get_result` | Returns the results of a process, in pages, optionally filtered to specific tasks. Each row has the final label and the probability of every label |
 | `list_processes` | Lists processes, newest first, including the failure reason when available |
 
 Notes:

src/behavioralsignals/mcp_server.py

@@ -36,7 +36,7 @@
 
 DEFAULT_WAIT = 45
 DEFAULT_LIMIT = 300
-RESULT_COLUMNS = ["start", "end", "task", "label", "confidence"]
+RESULT_COLUMNS = ["start", "end", "task", "label", "probabilities"]
 PROCESS_COLUMNS = ["pid", "name", "status", "duration", "created", "reason"]
 READ_ONLY = ToolAnnotations(read_only_hint=True)
 
@@ -71,7 +71,7 @@ def detect_deepfake(
     generator_detection: bool = False,
     wait_seconds: WaitSeconds = DEFAULT_WAIT,
 ) -> str:
-    """Detects whether speech (media="audio") or a video (media="video") is a deepfake.
+    """Detects whether speech (media="audio") or a video (media="video", experimental) is a deepfake.
 
     `source` is an absolute path to a local file, or an S3 presigned URL. With
     generator_detection=True, the result also names the likely generator (experimental). Each
@@ -100,6 +100,9 @@ def get_result(
     `analysis` is the kind of job: "behavioral" (analyze_behavior), "deepfake_audio" or
     "deepfake_video" (detect_deepfake). Results come as tab-separated rows, `limit` rows from
     `offset`. `tasks` keeps only those tasks, e.g. ["emotion"]; the header lists the task names.
+    Each row has the final label and the probability of every label of the task. Use the
+    probabilities as scores. positivity is valence and strength is arousal; for a scale from -1
+    to 1, use P(positive) - P(negative) and P(strong) - P(weak).
     """
     with _tool_errors(), _api(analysis) as api:
         return _result_text(api, analysis, pid, wait_seconds, tasks, offset, limit)
@@ -238,8 +241,15 @@ def _result_row(item: ResultItem) -> list[str] | None:
     label = item.finalLabel or (top.score if top else None)
     if not label:
         return None
-    confidence = top.posterior if top else None
-    return [_cell(value) for value in (item.startTime, item.endTime, item.task, label, confidence)]
+    values = (item.startTime, item.endTime, item.task, label, _probabilities(item))
+    return [_cell(value) for value in values]
+
+
+def _probabilities(item: ResultItem) -> str:
+    """Returns every label of the item with its probability, e.g. "sad=0.70, neutral=0.30"."""
+    return ", ".join(
+        f"{p.label}={float(p.posterior):.2f}" for p in item.prediction or [] if p.posterior
+    )
 
 
 def _format_result(

src/behavioralsignals/models.py

@@ -215,15 +215,17 @@ def model_dump_json(self, **kwargs):
 class ModelPredictions(_SerializableModel):
     label: str | None = Field(None, description="The name of the class", examples=["happy"])
     posterior: str | None = Field(
-        None, description="The probability of this class being present", examples=["0.754"]
+        None,
+        description="The probability of this class, as a string. The posteriors of one result add up to about 1, except for language, which lists only the top 3 languages. Use float(posterior) as a score when finalLabel is not enough",
+        examples=["0.754"],
     )
     score: str | None = Field(
         None,
         description="The regression score for continuous tasks (e.g. intensity), bounded in (0,1)",
         examples=["0.62"],
     )
     dominantInSegments: list[int] | None = Field(
-        None, description="The segments in which this class is dominant"
+        None, description="Internal field, usually empty. You can ignore it."
     )
 
 
@@ -237,22 +239,24 @@ class ResultItem(_SerializableModel):
     )
     task: str | None = Field(
         None,
-        description="The behavioral attribute. Can be one of diarization, deepfake, visual_deepfake, asr, gender, age, language, features, emotion, strength, positivity, speaking_rate, hesitation, politeness. "
+        description="The task. Behavioral: emotion, strength, positivity, speaking_rate, hesitation, engagement, intensity. Speaker: gender, age. Other: diarization, asr, language, features. Deepfakes API: deepfake, visual_deepfake, generator. "
         "Consider visiting the guides in behavioralsignals.readme.io for the latest examples.",
         examples=["emotion"],
     )
     prediction: list[ModelPredictions] | None = None
     finalLabel: str | None = Field(
-        None, description="The dominant value of the behavioral attribute", examples=["happy"]
+        None,
+        description="The label the API picks for this task. For behavioral tasks it is the label with the highest posterior.",
+        examples=["happy"],
     )
     level: str | None = Field(
         None,
-        description="Whether this result corresponds to a segment/utterance",
+        description="'utterance' in batch results. Streaming can also return 'segment'.",
         examples=["utterance"],
     )
     embedding: str | None = Field(
         None,
-        description="The corresponding embedding (present in diarization or features). It's a stringified array of length 728.",
+        description="The embedding as a stringified array: the speaker embedding (192 values) on the diarization row of a speaker's first utterance, or the behavioral embedding (768 values) on the features row.",
         examples=["[11.614513397216797, -15.228992462158203, -4.92175817489624, ...]"],
     )
 

src/behavioralsignals/utils.py

@@ -27,7 +27,7 @@ def make_audio_stream(file_path: str, chunk_size: float = 0.25) -> tuple[Iterato
 
 
 def print_results(items: Iterable[ResultItem] | None) -> None:
-    """Print one line per result: start time, end time, task, top label and its probability.
+    """Print one line per result: start time, end time, task, final label and its probability.
 
     Continuous tasks (e.g. intensity) have no label, so their score is printed instead.
     Rows without a label or score (e.g. the features row, which carries embeddings) are skipped.
@@ -40,5 +40,6 @@ def print_results(items: Iterable[ResultItem] | None) -> None:
         label = item.finalLabel or (top.score if top else None)
         if not label:
             continue
-        confidence = f" ({float(top.posterior):.1%})" if top and top.posterior else ""
+        posterior = next((p.posterior for p in item.prediction or [] if p.label == label), None)
+        confidence = f" ({float(posterior):.1%})" if posterior else ""
         print(f"{item.st} {item.et} {item.task} {label.strip()}{confidence}")

tests/test_mcp_server.py

@@ -111,13 +111,25 @@ async def test_formats_labelled_rows_and_drops_the_rest(api):
     assert not is_error
     assert text == (
         "pid 7 completed. Rows 1-3 of 3. Tasks: asr, emotion, intensity\n"
-        "start\tend\ttask\tlabel\tconfidence\n"
+        "start\tend\ttask\tlabel\tprobabilities\n"
         "0.5\t1.5\tasr\tHello there.\t\n"
-        "0.5\t1.5\temotion\tsad\t0.706\n"
+        "0.5\t1.5\temotion\tsad\tsad=0.71\n"
         "0.5\t1.5\tintensity\t0.62\t"
     )
 
 
+async def test_rows_show_the_probability_of_every_label(api):
+    prediction = [
+        ModelPredictions(label="sad", posterior="0.5"),
+        ModelPredictions(label="neutral", posterior="0.4"),
+    ]
+    api.responses["wait_for_result"] = ResultResponse(
+        pid=7, results=[ResultItem(task="emotion", finalLabel="neutral", prediction=prediction)]
+    )
+    _, text = await call("get_result", pid=7, analysis="behavioral")
+    assert text.endswith("\temotion\tneutral\tsad=0.50, neutral=0.40")
+
+
 async def test_tabs_and_newlines_in_a_cell_become_spaces(api):
     api.responses["wait_for_result"] = ResultResponse(
         pid=7, results=[item("asr", "Hi\tthere.\nBye.")]
@@ -134,9 +146,9 @@ async def test_video_rows_have_a_track_column(api):
     )
     _, text = await call("get_result", pid=7, analysis="deepfake_video")
     assert text.splitlines()[1:] == [
-        "track\tstart\tend\ttask\tlabel\tconfidence",
-        "audio\t0.5\t1.5\tdeepfake\tbonafide\t0.9",
-        "video\t0.5\t1.5\tvisual_deepfake\tspoofed\t0.8",
+        "track\tstart\tend\ttask\tlabel\tprobabilities",
+        "audio\t0.5\t1.5\tdeepfake\tbonafide\tbonafide=0.90",
+        "video\t0.5\t1.5\tvisual_deepfake\tspoofed\tspoofed=0.80",
     ]
 
 

tests/test_utils.py

@@ -59,6 +59,12 @@ def test_print_results_prints_one_line_per_result(capsys):
     )
 
 
+def test_print_results_shows_the_probability_of_the_final_label(capsys):
+    prediction = [{"label": "sad", "posterior": "0.5"}, {"label": "neutral", "posterior": "0.4"}]
+    print_results([_item("emotion", prediction, "neutral")])
+    assert capsys.readouterr().out == "0.487 3.001 emotion neutral (40.0%)\n"
+
+
 def test_print_results_accepts_none(capsys):
     print_results(None)
     assert capsys.readouterr().out == ""