diff --git a/README.md b/README.md index 2c91e63..8484629 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ Official Python SDK for the [Behavioral Signals API](https://behavioralsignals.readme.io/). -Analyze human behavior and detect deepfake speech using batch and real-time audio APIs. Experimental video deepfake detection is also available in batch mode. +Estimate behavioral signals from speech and detect deepfake speech using batch and real-time audio APIs. Experimental video deepfake detection is also available in batch mode. [Python SDK Documentation](https://behavioralsignals.readme.io/docs/behavioral-signals-python-sdk) · [Examples](examples/) · @@ -34,7 +34,7 @@ Analyze human behavior and detect deepfake speech using batch and real-time audi ## Features -- **Behavioral Analysis** — analyze human behavior from speech in batch and real-time streaming modes +- **Behavioral Analysis** — estimate behavioral signals from speech in batch and real-time streaming modes - **Deepfake Detection** — detect synthetic or manipulated speech in batch and real-time streaming modes - **Video Deepfake Detection (Experimental, Batch Only)** — analyze both the video frames and audio track of supported video files - **Core Speech Attributes (Batch Only)** — automatic speech recognition (ASR), speaker diarization, and language identification @@ -96,7 +96,7 @@ print_results(result.results) `upload_audio()` returns a process with a unique process ID (`pid`). `wait_for_result()` polls until processing completes and returns the analysis result. -`print_results()` prints one line per result: the start and end time in seconds, the task, and the top label with its probability. +`print_results()` prints one line per result: the start and end time in seconds, the task, and the final label with its probability. Continuous tasks such as `intensity` have no label, only a score, and `asr` and `diarization` have a label without a probability. For a 10-second clip of one speaker, the output starts like this: @@ -122,6 +122,52 @@ process = client.behavioral.upload_audio( ) ``` +### Behavioral outputs + +The Behavioral API returns **24 signals** across 8 dimensions: 18 behavioral signals and 6 speaker signals. Each label is its own signal with its own probability. +All eight dimensions are scored together on every utterance longer than 1 second, so one utterance can be *sad + weak + negative + slow + hesitating + withdrawn*, with a voice estimated as female and aged 31 - 45. That gives 4 × 3 × 3 × 3 × 2 × 3 × 2 × 4 = **5,184 possible profiles** per utterance. + +| Task | What it measures | Labels | # | +|---|---|---|---| +| `emotion` | Basic emotion in the voice | `happy`, `angry`, `sad`, `neutral` | 4 | +| `strength` | Arousal: energy in the voice | `strong`, `weak`, `neutral` | 3 | +| `positivity` | Valence: sentiment of the tone | `positive`, `negative`, `neutral` | 3 | +| `speaking_rate` | How fast the speaker talks, compared to speakers in general, not to their own pace | `fast`, `slow`, `normal` | 3 | +| `hesitation` | Signs of hesitation | `yes`, `no` | 2 | +| `engagement` | Involved or detached tone | `engaged`, `withdrawn`, `neutral` | 3 | +| `gender` | Sex of the speaker, estimated from the voice | `female`, `male` | 2 | +| `age` | Age range of the speaker, estimated from the voice | `18 - 22`, `23 - 30`, `31 - 45`, `46 - 65` | 4 | +| **Total** | | | **24** | + +Plus `intensity`: how intense the emotion is, as one score between 0 and 1. It is not a score for any one emotion, even when `emotion` is `angry`. It has no labels and is not counted in the 24. +In batch results, utterances of 1 second or less return only `diarization`, `asr` and `language`. When streaming, every segment gets all 24 signals and `intensity`, however short it is. +In batch results, `gender` and `age` are given per speaker: every utterance of a speaker longer than 1 second gets the same values, averaged over the file. When streaming, `gender` and `age` are estimated on each segment. +The other tasks have open-ended values, not fixed labels: `diarization` (a speaker ID such as `SPEAKER_00`), `asr` (the transcript) and `language`. With `embeddings=True`, `features` holds the behavioral embedding. +All results of one utterance share its `id`, `startTime` and `endTime`. To split results by speaker, use the `diarization` label of each utterance. Speaker IDs are generic: the API does not say who is the agent and who is the customer. + +`neutral` means something different in each task, so always read a label together with its task: `emotion: neutral` means no clear emotion, `strength: neutral` normal energy, `positivity: neutral` neither positive nor negative, and `engagement: neutral` neither engaged nor withdrawn. + +`finalLabel` gives one answer per task. For behavioral tasks it is the label with the highest probability. +When you need a score instead of a label, use the probabilities in `prediction`. They are strings, so convert them with `float()`. +For each task in the table they add up to 1, so each label's probability is a score from 0 to 1 that you can track over time, average over a call, or compare against your own threshold. +For example, the `angry` probability of `emotion` is a score for anger. `intensity` is not: it does not say which emotion is intense. + +`positivity` is valence and `strength` is arousal, the two dimensions often used in emotion research. The API gives them as probabilities from 0 to 1. To put them on the usual scale from -1 to 1, subtract the probabilities of the two opposite labels: + +```python +for item in result.results: + if item.task in ("positivity", "strength"): + scores = {p.label: float(p.posterior) for p in item.prediction} + if item.task == "positivity": + print(item.st, "valence", scores["positive"] - scores["negative"]) + else: + print(item.st, "arousal", scores["strong"] - scores["weak"]) +``` + +All probabilities are model estimates: use them to compare utterances and follow trends, not as exact measurements. + +See [Definition of behaviors](https://behavioralsignals.readme.io/docs/definition-of-behaviors) for what each signal means. + ## Deepfake Detection ### Audio @@ -383,8 +429,8 @@ The server exposes four tools: | Tool | What it does | |---|---| | `analyze_behavior` | Uploads an audio file or S3 presigned URL for behavioral analysis and returns the results | -| `detect_deepfake` | Uploads an audio or video file, or an S3 presigned URL, for deepfake detection and returns the results | -| `get_result` | Returns the results of a process, in pages, optionally filtered to specific tasks | +| `detect_deepfake` | Uploads an audio or video file, or an S3 presigned URL, for deepfake detection (video is experimental) and returns the results | +| `get_result` | Returns the results of a process, in pages, optionally filtered to specific tasks. Each row has the final label and the probability of every label | | `list_processes` | Lists processes, newest first, including the failure reason when available | Notes: diff --git a/src/behavioralsignals/mcp_server.py b/src/behavioralsignals/mcp_server.py index 5b87b1c..8894a88 100644 --- a/src/behavioralsignals/mcp_server.py +++ b/src/behavioralsignals/mcp_server.py @@ -36,7 +36,7 @@ DEFAULT_WAIT = 45 DEFAULT_LIMIT = 300 -RESULT_COLUMNS = ["start", "end", "task", "label", "confidence"] +RESULT_COLUMNS = ["start", "end", "task", "label", "probabilities"] PROCESS_COLUMNS = ["pid", "name", "status", "duration", "created", "reason"] READ_ONLY = ToolAnnotations(read_only_hint=True) @@ -71,7 +71,7 @@ def detect_deepfake( generator_detection: bool = False, wait_seconds: WaitSeconds = DEFAULT_WAIT, ) -> str: - """Detects whether speech (media="audio") or a video (media="video") is a deepfake. + """Detects whether speech (media="audio") or a video (media="video", experimental) is a deepfake. `source` is an absolute path to a local file, or an S3 presigned URL. With generator_detection=True, the result also names the likely generator (experimental). Each @@ -100,6 +100,9 @@ def get_result( `analysis` is the kind of job: "behavioral" (analyze_behavior), "deepfake_audio" or "deepfake_video" (detect_deepfake). Results come as tab-separated rows, `limit` rows from `offset`. `tasks` keeps only those tasks, e.g. ["emotion"]; the header lists the task names. + Each row has the final label and the probability of every label of the task. Use the + probabilities as scores. positivity is valence and strength is arousal; for a scale from -1 + to 1, use P(positive) - P(negative) and P(strong) - P(weak). """ with _tool_errors(), _api(analysis) as api: return _result_text(api, analysis, pid, wait_seconds, tasks, offset, limit) @@ -238,8 +241,15 @@ def _result_row(item: ResultItem) -> list[str] | None: label = item.finalLabel or (top.score if top else None) if not label: return None - confidence = top.posterior if top else None - return [_cell(value) for value in (item.startTime, item.endTime, item.task, label, confidence)] + values = (item.startTime, item.endTime, item.task, label, _probabilities(item)) + return [_cell(value) for value in values] + + +def _probabilities(item: ResultItem) -> str: + """Returns every label of the item with its probability, e.g. "sad=0.70, neutral=0.30".""" + return ", ".join( + f"{p.label}={float(p.posterior):.2f}" for p in item.prediction or [] if p.posterior + ) def _format_result( diff --git a/src/behavioralsignals/models.py b/src/behavioralsignals/models.py index f941cd4..e6b07ea 100644 --- a/src/behavioralsignals/models.py +++ b/src/behavioralsignals/models.py @@ -215,7 +215,9 @@ def model_dump_json(self, **kwargs): class ModelPredictions(_SerializableModel): label: str | None = Field(None, description="The name of the class", examples=["happy"]) posterior: str | None = Field( - None, description="The probability of this class being present", examples=["0.754"] + None, + description="The probability of this class, as a string. The posteriors of one result add up to about 1, except for language, which lists only the top 3 languages. Use float(posterior) as a score when finalLabel is not enough", + examples=["0.754"], ) score: str | None = Field( None, @@ -223,7 +225,7 @@ class ModelPredictions(_SerializableModel): examples=["0.62"], ) dominantInSegments: list[int] | None = Field( - None, description="The segments in which this class is dominant" + None, description="Internal field, usually empty. You can ignore it." ) @@ -237,22 +239,24 @@ class ResultItem(_SerializableModel): ) task: str | None = Field( None, - description="The behavioral attribute. Can be one of diarization, deepfake, visual_deepfake, asr, gender, age, language, features, emotion, strength, positivity, speaking_rate, hesitation, politeness. " + description="The task. Behavioral: emotion, strength, positivity, speaking_rate, hesitation, engagement, intensity. Speaker: gender, age. Other: diarization, asr, language, features. Deepfakes API: deepfake, visual_deepfake, generator. " "Consider visiting the guides in behavioralsignals.readme.io for the latest examples.", examples=["emotion"], ) prediction: list[ModelPredictions] | None = None finalLabel: str | None = Field( - None, description="The dominant value of the behavioral attribute", examples=["happy"] + None, + description="The label the API picks for this task. For behavioral tasks it is the label with the highest posterior.", + examples=["happy"], ) level: str | None = Field( None, - description="Whether this result corresponds to a segment/utterance", + description="'utterance' in batch results. Streaming can also return 'segment'.", examples=["utterance"], ) embedding: str | None = Field( None, - description="The corresponding embedding (present in diarization or features). It's a stringified array of length 728.", + description="The embedding as a stringified array: the speaker embedding (192 values) on the diarization row of a speaker's first utterance, or the behavioral embedding (768 values) on the features row.", examples=["[11.614513397216797, -15.228992462158203, -4.92175817489624, ...]"], ) diff --git a/src/behavioralsignals/utils.py b/src/behavioralsignals/utils.py index 18054cb..ee0c232 100644 --- a/src/behavioralsignals/utils.py +++ b/src/behavioralsignals/utils.py @@ -27,7 +27,7 @@ def make_audio_stream(file_path: str, chunk_size: float = 0.25) -> tuple[Iterato def print_results(items: Iterable[ResultItem] | None) -> None: - """Print one line per result: start time, end time, task, top label and its probability. + """Print one line per result: start time, end time, task, final label and its probability. Continuous tasks (e.g. intensity) have no label, so their score is printed instead. Rows without a label or score (e.g. the features row, which carries embeddings) are skipped. @@ -40,5 +40,6 @@ def print_results(items: Iterable[ResultItem] | None) -> None: label = item.finalLabel or (top.score if top else None) if not label: continue - confidence = f" ({float(top.posterior):.1%})" if top and top.posterior else "" + posterior = next((p.posterior for p in item.prediction or [] if p.label == label), None) + confidence = f" ({float(posterior):.1%})" if posterior else "" print(f"{item.st} {item.et} {item.task} {label.strip()}{confidence}") diff --git a/tests/test_mcp_server.py b/tests/test_mcp_server.py index 5686e91..66a1365 100644 --- a/tests/test_mcp_server.py +++ b/tests/test_mcp_server.py @@ -111,13 +111,25 @@ async def test_formats_labelled_rows_and_drops_the_rest(api): assert not is_error assert text == ( "pid 7 completed. Rows 1-3 of 3. Tasks: asr, emotion, intensity\n" - "start\tend\ttask\tlabel\tconfidence\n" + "start\tend\ttask\tlabel\tprobabilities\n" "0.5\t1.5\tasr\tHello there.\t\n" - "0.5\t1.5\temotion\tsad\t0.706\n" + "0.5\t1.5\temotion\tsad\tsad=0.71\n" "0.5\t1.5\tintensity\t0.62\t" ) +async def test_rows_show_the_probability_of_every_label(api): + prediction = [ + ModelPredictions(label="sad", posterior="0.5"), + ModelPredictions(label="neutral", posterior="0.4"), + ] + api.responses["wait_for_result"] = ResultResponse( + pid=7, results=[ResultItem(task="emotion", finalLabel="neutral", prediction=prediction)] + ) + _, text = await call("get_result", pid=7, analysis="behavioral") + assert text.endswith("\temotion\tneutral\tsad=0.50, neutral=0.40") + + async def test_tabs_and_newlines_in_a_cell_become_spaces(api): api.responses["wait_for_result"] = ResultResponse( pid=7, results=[item("asr", "Hi\tthere.\nBye.")] @@ -134,9 +146,9 @@ async def test_video_rows_have_a_track_column(api): ) _, text = await call("get_result", pid=7, analysis="deepfake_video") assert text.splitlines()[1:] == [ - "track\tstart\tend\ttask\tlabel\tconfidence", - "audio\t0.5\t1.5\tdeepfake\tbonafide\t0.9", - "video\t0.5\t1.5\tvisual_deepfake\tspoofed\t0.8", + "track\tstart\tend\ttask\tlabel\tprobabilities", + "audio\t0.5\t1.5\tdeepfake\tbonafide\tbonafide=0.90", + "video\t0.5\t1.5\tvisual_deepfake\tspoofed\tspoofed=0.80", ] diff --git a/tests/test_utils.py b/tests/test_utils.py index bae75ec..ab44139 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -59,6 +59,12 @@ def test_print_results_prints_one_line_per_result(capsys): ) +def test_print_results_shows_the_probability_of_the_final_label(capsys): + prediction = [{"label": "sad", "posterior": "0.5"}, {"label": "neutral", "posterior": "0.4"}] + print_results([_item("emotion", prediction, "neutral")]) + assert capsys.readouterr().out == "0.487 3.001 emotion neutral (40.0%)\n" + + def test_print_results_accepts_none(capsys): print_results(None) assert capsys.readouterr().out == ""