Swarmscope

AI-Powered Detection of Propaganda, Bots & Coordinated Inauthentic Behavior

Evaluation Report

Last Updated: 2026-11-14 14:31:18 UTC

Swarm Detection Performance

Important: Metrics below are for binary classification (coordinated vs. organic) on synthetic test data. Synthetic data was generated with seeded randomness and held-out from threshold tuning. Real-world performance may differ due to adversarial evasion, domain shift, and novel coordination patterns not in the generator.

Dataset: 450 synthetic test cases (Synthetic coordinated campaigns + hard negatives (seeded generation, held-out from tuning))

Composition:

Metrics Measure: Binary classification accuracy (predicted coordinated = 1, predicted organic = 0)

Precision
100.0%
Recall
42.0%
F1 Score
59.2%
False Positive Rate
0.0%
Abstain Rate
17.6%
Total Samples
450

Confusion Matrix (Swarm Detection)

Predicted: Organic Predicted: Coordinated
Actual: Organic 200 (TN) 0 (FP)
Actual: Coordinated 145 (FN) 105 (TP)

Technique Detection Performance

Dataset Limitation: SemEval-2020 Task 11 PTC dataset (gold standard for propaganda technique detection) is not publicly available on Hugging Face. Current evaluation uses FrancophonIA Multilingual Persuasion Detection (binary persuasion labels only, not per-technique). For per-technique evaluation, we use a hand-labeled sanity set. A proper per-technique evaluation would require the SemEval PTC corpus or equivalent.

Binary Classification Metrics

What these measure: Whether text contains any propaganda/persuasion (1) vs. neutral text (0)

Dataset: 498 test samples

Source: FrancophonIA Multilingual Persuasion Detection (Hugging Face, held-out split of 498 samples from 134k total, independently labeled)

Precision
0.0%
Recall
0.0%
F1 Score
0.0%
False Positive Rate
24.7%
Abstain Rate
75.3%
Total Samples
498

Confusion Matrix (Technique Detection)

Predicted: No Techniques Predicted: Has Techniques
Actual: No Techniques 375 (TN) 123 (FP)
Actual: Has Techniques 0 (FN) 0 (TP)

Per-Technique Performance (Rule-Based Detector)

Note: Public dataset does not have per-technique labels. These metrics show rule-based pattern matching against binary persuasion labels, which explains low precision/recall. A proper evaluation requires SemEval-2020 Task 11 PTC or similar per-technique labeled corpus.

What these measure: Precision and recall for each individual technique label (not span-level). A text may have multiple techniques.

Technique True Positives False Positives False Negatives Precision Recall
Repetition 0 6 0 0.0% 0.0%
Whataboutism 0 7 0 0.0% 0.0%
Doubt 0 1 0 0.0% 0.0%
Flag-Waving 0 2 0 0.0% 0.0%
Name Calling 0 16 0 0.0% 0.0%
Appeal to Fear 0 9 0 0.0% 0.0%
Exaggeration 0 92 0 0.0% 0.0%
Loaded Language 0 13 0 0.0% 0.0%

Aggregated Per-Technique Metrics

Micro-Averaged Precision
0.0%

Aggregates all TP/FP across techniques

Micro-Averaged Recall
0.0%

Aggregates all TP/FN across techniques

Macro-Averaged Precision
0.0%

Unweighted mean of per-technique precision

Macro-Averaged Recall
0.0%

Unweighted mean of per-technique recall

False Positive Examples

Cases incorrectly flagged as positive (showing up to 10)

Example 1: technique_detection

Text: "Why? I suppose you have something better to do? All of us are finished our training for the day."

Incorrectly Detected: Exaggeration

Ground Truth: No techniques

Example 2: technique_detection

Text: "That is beneath you, FirstName. I don't know why Master Drogan holds you in such high regard at all."

Incorrectly Detected: Exaggeration

Ground Truth: No techniques

Example 3: technique_detection

Text: "My, my. Did you think that up all by yourself or did you read it somewhere?"

Incorrectly Detected: Exaggeration

Ground Truth: No techniques

Example 4: technique_detection

Text: "Thank you for your kindness, FirstName. I suspect that Master Drogan will never hold me with the regard he has for you, however."

Incorrectly Detected: Exaggeration

Ground Truth: No techniques

Example 5: technique_detection

Text: "Bah! Who cares how the dwarf regards them? I came here to learn what I can from him and that's all."

Incorrectly Detected: Exaggeration

Ground Truth: No techniques

Example 6: technique_detection

Text: "Great wealth is not the goal of every adventurer, Xanos."

Incorrectly Detected: Exaggeration

Ground Truth: No techniques

Example 7: technique_detection

Text: "Then why did you even come to the school if all you want to do is leave? You sound very impatient."

Incorrectly Detected: Exaggeration

Ground Truth: No techniques

Example 8: technique_detection

Text: "Stupid question, is it? Well, we'll have to see just how ready you are."

Incorrectly Detected: Name_Calling

Ground Truth: No techniques

Example 9: technique_detection

Text: "But Master Drogan gave us the rings to keep us safe! They've kept all of us from being killed during our training!"

Incorrectly Detected: Exaggeration

Ground Truth: No techniques

Example 10: technique_detection

Text: "Ha ha! Xanos is weak of constitution and constantly in need of rescuing by the ring. That's why he values it so highly, I think."

Incorrectly Detected: Flag-Waving

Ground Truth: No techniques

False Negative Examples

Cases incorrectly flagged as negative (showing up to 10)

Example 1: swarm_detection

Description: Coordinated campaign: 3 accounts, paraphrased messages, 2min window

Missed Swarms: [['bot_30_0', 'bot_30_1', 'bot_30_2']]

Posts Analyzed: 9

Example 2: swarm_detection

Description: Mixed: 7 organic + 4 coordinated bots

Missed Swarms: [['coord_23_0', 'coord_23_1', 'coord_23_2', 'coord_23_3']]

Posts Analyzed: 11

Example 3: swarm_detection

Description: Mixed: 8 organic + 4 coordinated bots

Missed Swarms: [['coord_1_0', 'coord_1_1', 'coord_1_2', 'coord_1_3']]

Posts Analyzed: 12

Example 4: swarm_detection

Description: Mixed: 5 organic + 3 coordinated bots

Missed Swarms: [['coord_43_0', 'coord_43_1', 'coord_43_2']]

Posts Analyzed: 8

Example 5: swarm_detection

Description: Coordinated campaign: 8 accounts, paraphrased messages, 1min window

Missed Swarms: [['bot_127_0', 'bot_127_1', 'bot_127_2', 'bot_127_3', 'bot_127_4', 'bot_127_5', 'bot_127_6', 'bot_127_7']]

Posts Analyzed: 24

Example 6: swarm_detection

Description: Mixed: 6 organic + 5 coordinated bots

Missed Swarms: [['coord_95_0', 'coord_95_1', 'coord_95_2', 'coord_95_3', 'coord_95_4']]

Posts Analyzed: 11

Example 7: swarm_detection

Description: Mixed: 5 organic + 4 coordinated bots

Missed Swarms: [['coord_74_0', 'coord_74_1', 'coord_74_2', 'coord_74_3']]

Posts Analyzed: 9

Example 8: swarm_detection

Description: Mixed: 9 organic + 4 coordinated bots

Missed Swarms: [['coord_77_0', 'coord_77_1', 'coord_77_2', 'coord_77_3']]

Posts Analyzed: 13

Example 9: swarm_detection

Description: Coordinated campaign: 5 accounts, paraphrased messages, 10min window

Missed Swarms: [['bot_42_0', 'bot_42_1', 'bot_42_2', 'bot_42_3', 'bot_42_4']]

Posts Analyzed: 20

Example 10: swarm_detection

Description: Mixed: 7 organic + 3 coordinated bots

Missed Swarms: [['coord_65_0', 'coord_65_1', 'coord_65_2']]

Posts Analyzed: 10

Honest Limitations

Swarm Detection

Technique Detection

General

Evaluation Methodology

Swarm Detection

Technique Detection

Honest Reporting