← All tasks

Does your model follow a function-call schema?

ranked by score ↓
Source

Paste as source: in your trap.yaml

git+https://github.com/trapstreet/trapstreet-tasks@00d0632172c69e6f31c9ce26799ea34865e67930#subdirectory=tasks/core_json_schema_output
Share

core-json-schema-output

An open-source evaluation task for structured output / function-call compliance. Useful as a basic sanity check when building AI agents that rely on models producing valid, schema-conforming JSON — the foundation of any tool-use, function-calling, or structured-response pipel

20 cases

Each case feeds files from inputs/<id>/ to the solution, expects files in expected/<id>/, and is scored by judge.py then aggregated by grader.py.

traptask.yaml · source on GitHub

cases (20)

▸fn_69Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
Find out the population and species of turtles in Mississippi river in 2020.

FUNCTION SPEC (JSON):
{
  "name": "ecology.get_turtle_population",
  "description": "Get the population and species of turtles in a specific location.",
  "parameters": {
    "type": "dict",
    "properties": {
      "location": {
        "type": "string",
        "description": "The name of the location."
      },
      "year": {
        "type": "integer",
        "description": "The year of the data requested. Default is 2001."
      },
      "species": {
        "type": "boolean",
        "description": "Whether to include species information. Default is false."
      }
    },
    "required": [
      "location"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_69",
  "answer": "{\"name\": \"ecology.get_turtle_population\", \"arguments\": {\"location\": \"Mississippi river\", \"year\": 2020, \"species\": true}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "ecology.get_turtle_population",
      "args_accepted": {
        "location": [
          "Mississippi river"
        ],
        "year": [
          2020
        ],
        "species": [
          true
        ]
      }
    }
  ],
  "category": "easy",
  "difficulty": "easy",
  "source_bfcl_id": "simple_python_69"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_18Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
Find the prime factors of the number 123456.

FUNCTION SPEC (JSON):
{
  "name": "number_analysis.prime_factors",
  "description": "Compute the prime factors of a number.",
  "parameters": {
    "type": "dict",
    "properties": {
      "number": {
        "type": "integer",
        "description": "The number to be factored."
      }
    },
    "required": [
      "number"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_18",
  "answer": "{\"name\": \"number_analysis.prime_factors\", \"arguments\": {\"number\": 123456}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "number_analysis.prime_factors",
      "args_accepted": {
        "number": [
          123456
        ]
      }
    }
  ],
  "category": "easy",
  "difficulty": "easy",
  "source_bfcl_id": "simple_python_18"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_153Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
Calculate the compounded interest on an initial deposit of $5000 at an annual interest rate of 3% for 5 years, compounded quarterly.

FUNCTION SPEC (JSON):
{
  "name": "calculate_compounded_interest",
  "description": "Calculate the compounded interest for a given initial deposit, interest rate, time and number of times the interest is compounded per unit time.",
  "parameters": {
    "type": "dict",
    "properties": {
      "principal": {
        "type": "integer",
        "description": "The initial amount of money that is being invested or loaned."
      },
      "rate": {
        "type": "float",
        "description": "The annual interest rate."
      },
      "time": {
        "type": "integer",
        "description": "The number of time periods the money is invested or loaned for."
      },
      "n": {
        "type": "integer",
        "description": "The number of times that interest is compounded per unit time."
      }
    },
    "required": [
      "principal",
      "rate",
      "time",
      "n"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_153",
  "answer": "{\"name\": \"calculate_compounded_interest\", \"arguments\": {\"principal\": 5000, \"rate\": 0.03, \"time\": 5, \"n\": 4}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "calculate_compounded_interest",
      "args_accepted": {
        "principal": [
          5000
        ],
        "rate": [
          0.03
        ],
        "time": [
          5
        ],
        "n": [
          4
        ]
      }
    }
  ],
  "category": "easy",
  "difficulty": "easy",
  "source_bfcl_id": "simple_python_153"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_135Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
Calculate the return on investment for a stock bought at $20, sold at $25, with a dividend of $2.

FUNCTION SPEC (JSON):
{
  "name": "calculate_return_on_investment",
  "description": "Calculate the return on investment for a given stock based on its purchase price, sale price, and any dividends received.",
  "parameters": {
    "type": "dict",
    "properties": {
      "purchase_price": {
        "type": "integer",
        "description": "The price the stock was bought at."
      },
      "sale_price": {
        "type": "integer",
        "description": "The price the stock was sold at."
      },
      "dividend": {
        "type": "integer",
        "description": "Any dividends received from the stock.",
        "default": 0
      }
    },
    "required": [
      "purchase_price",
      "sale_price"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_135",
  "answer": "{\"name\": \"calculate_return_on_investment\", \"arguments\": {\"purchase_price\": 20, \"sale_price\": 25, \"dividend\": 2}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "calculate_return_on_investment",
      "args_accepted": {
        "purchase_price": [
          20
        ],
        "sale_price": [
          25
        ],
        "dividend": [
          2
        ]
      }
    }
  ],
  "category": "easy",
  "difficulty": "easy",
  "source_bfcl_id": "simple_python_135"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_128Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
What's the quarterly dividend per share of a company with 100 million outstanding shares and total dividend payout of 50 million USD?

FUNCTION SPEC (JSON):
{
  "name": "finance.calculate_quarterly_dividend_per_share",
  "description": "Calculate quarterly dividend per share for a company given total dividend payout and outstanding shares",
  "parameters": {
    "type": "dict",
    "properties": {
      "total_payout": {
        "type": "integer",
        "description": "The total amount of dividends paid out in USD"
      },
      "outstanding_shares": {
        "type": "integer",
        "description": "Total number of outstanding shares"
      }
    },
    "required": [
      "total_payout",
      "outstanding_shares"
    ],
    "optional": []
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_128",
  "answer": "{\"name\": \"finance.calculate_quarterly_dividend_per_share\", \"arguments\": {\"total_payout\": 50000000, \"outstanding_shares\": 100000000}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "finance.calculate_quarterly_dividend_per_share",
      "args_accepted": {
        "total_payout": [
          50000000
        ],
        "outstanding_shares": [
          100000000
        ]
      }
    }
  ],
  "category": "easy",
  "difficulty": "easy",
  "source_bfcl_id": "simple_python_128"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_83Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
Calculate the distance between two GPS coordinates (33.4484 N, 112.0740 W) and (34.0522 N, 118.2437 W) in miles.

FUNCTION SPEC (JSON):
{
  "name": "calculate_distance",
  "description": "Calculate the distance between two GPS coordinates.",
  "parameters": {
    "type": "dict",
    "properties": {
      "coord1": {
        "type": "tuple",
        "description": "The first coordinate as (latitude, longitude).",
        "items": {
          "type": "float"
        }
      },
      "coord2": {
        "type": "tuple",
        "description": "The second coordinate as (latitude, longitude).",
        "items": {
          "type": "float"
        }
      },
      "unit": {
        "type": "string",
        "description": "The unit of distance. Options: 'miles', 'kilometers'."
      }
    },
    "required": [
      "coord1",
      "coord2",
      "unit"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_83",
  "answer": "{\"name\": \"calculate_distance\", \"arguments\": {\"coord1\": [33.4484, -112.074], \"coord2\": [34.0522, -118.2437], \"unit\": \"miles\"}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "calculate_distance",
      "args_accepted": {
        "coord1": [
          [
            33.4484,
            -112.074
          ]
        ],
        "coord2": [
          [
            34.0522,
            -118.2437
          ]
        ],
        "unit": [
          "miles"
        ]
      }
    }
  ],
  "category": "easy",
  "difficulty": "easy",
  "source_bfcl_id": "simple_python_83"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_67Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
Identify a small green bird in forest.

FUNCTION SPEC (JSON):
{
  "name": "identify_bird",
  "description": "Identify a bird species based on certain characteristics.",
  "parameters": {
    "type": "dict",
    "properties": {
      "color": {
        "type": "string",
        "description": "Color of the bird."
      },
      "habitat": {
        "type": "string",
        "description": "Habitat of the bird."
      },
      "size": {
        "type": "string",
        "enum": [
          "small",
          "medium",
          "large"
        ],
        "description": "Size of the bird. Default is 'small'"
      }
    },
    "required": [
      "color",
      "habitat"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_67",
  "answer": "{\"name\": \"identify_bird\", \"arguments\": {\"color\": \"green\", \"habitat\": \"forest\", \"size\": \"small\"}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "identify_bird",
      "args_accepted": {
        "color": [
          "green"
        ],
        "habitat": [
          "forest"
        ],
        "size": [
          "small"
        ]
      }
    }
  ],
  "category": "easy",
  "difficulty": "easy",
  "source_bfcl_id": "simple_python_67"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_61Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
Predict whether a person with weight 150lbs and height 5ft 10in who is lightly active will get type 2 diabetes.

FUNCTION SPEC (JSON):
{
  "name": "diabetes_prediction",
  "description": "Predict the likelihood of diabetes type 2 based on a person's weight and height.",
  "parameters": {
    "type": "dict",
    "properties": {
      "weight": {
        "type": "integer",
        "description": "Weight of the person in lbs."
      },
      "height": {
        "type": "integer",
        "description": "Height of the person in inches."
      },
      "activity_level": {
        "type": "string",
        "enum": [
          "sedentary",
          "lightly active",
          "moderately active",
          "very active",
          "extra active"
        ],
        "description": "Physical activity level of the person."
      }
    },
    "required": [
      "weight",
      "height",
      "activity_level"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_61",
  "answer": "{\"name\": \"diabetes_prediction\", \"arguments\": {\"weight\": 150, \"height\": 70, \"activity_level\": \"lightly active\"}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "diabetes_prediction",
      "args_accepted": {
        "weight": [
          150
        ],
        "height": [
          70
        ],
        "activity_level": [
          "lightly active"
        ]
      }
    }
  ],
  "category": "easy",
  "difficulty": "easy",
  "source_bfcl_id": "simple_python_61"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_282Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
Find a Yamaha flute with the specifications of open hole, C foot, and silver headjoint available for sale.

FUNCTION SPEC (JSON):
{
  "name": "find_flute",
  "description": "Locate a flute for sale based on specific requirements.",
  "parameters": {
    "type": "dict",
    "properties": {
      "brand": {
        "type": "string",
        "description": "The brand of the flute. Example, 'Yamaha'"
      },
      "specs": {
        "type": "array",
        "items": {
          "type": "string",
          "enum": [
            "open hole",
            "C foot",
            "silver headjoint"
          ]
        },
        "description": "The specifications of the flute desired."
      }
    },
    "required": [
      "brand",
      "specs"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_282",
  "answer": "{\"name\": \"find_flute\", \"arguments\": {\"brand\": \"Yamaha\", \"specs\": [\"open hole\", \"C foot\", \"silver headjoint\"]}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "find_flute",
      "args_accepted": {
        "brand": [
          "Yamaha"
        ],
        "specs": [
          [
            "open hole",
            "C foot",
            "silver headjoint"
          ]
        ]
      }
    }
  ],
  "category": "easy",
  "difficulty": "easy",
  "source_bfcl_id": "simple_python_282"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_20Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
Find the highest common factor of 36 and 24.

FUNCTION SPEC (JSON):
{
  "name": "math.hcf",
  "description": "Calculate the highest common factor of two numbers.",
  "parameters": {
    "type": "dict",
    "properties": {
      "number1": {
        "type": "integer",
        "description": "First number."
      },
      "number2": {
        "type": "integer",
        "description": "Second number."
      }
    },
    "required": [
      "number1",
      "number2"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_20",
  "answer": "{\"name\": \"math.hcf\", \"arguments\": {\"number1\": 36, \"number2\": 24}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "math.hcf",
      "args_accepted": {
        "number1": [
          36
        ],
        "number2": [
          24
        ]
      }
    }
  ],
  "category": "easy",
  "difficulty": "easy",
  "source_bfcl_id": "simple_python_20"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_30Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
What is the final velocity of a vehicle that started from rest and accelerated at 4 m/s^2 for a distance of 300 meters?

FUNCTION SPEC (JSON):
{
  "name": "kinematics.final_velocity_from_distance",
  "description": "Calculate the final velocity of an object given the acceleration and distance travelled, assuming initial velocity is 0.",
  "parameters": {
    "type": "dict",
    "properties": {
      "acceleration": {
        "type": "integer",
        "description": "Acceleration of the object, m/s^2."
      },
      "distance": {
        "type": "integer",
        "description": "Distance traveled by the object, m."
      },
      "initial_velocity": {
        "type": "float",
        "description": "Initial velocity of the object. Default is 0, m/s"
      }
    },
    "required": [
      "acceleration",
      "distance"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_30",
  "answer": "{\"name\": \"kinematics.final_velocity_from_distance\", \"arguments\": {\"acceleration\": 4, \"distance\": 300}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "kinematics.final_velocity_from_distance",
      "args_accepted": {
        "acceleration": [
          4
        ],
        "distance": [
          300
        ],
        "initial_velocity": [
          "",
          0
        ]
      }
    }
  ],
  "category": "medium",
  "difficulty": "medium",
  "source_bfcl_id": "simple_python_30"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_81Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
Find the fastest route from San Francisco to Los Angeles with toll roads avoided.

FUNCTION SPEC (JSON):
{
  "name": "map_routing.fastest_route",
  "description": "Finds the fastest route from one location to another, with an option to avoid toll roads.",
  "parameters": {
    "type": "dict",
    "properties": {
      "start_location": {
        "type": "string",
        "description": "The starting location for the journey."
      },
      "end_location": {
        "type": "string",
        "description": "The destination for the journey."
      },
      "avoid_tolls": {
        "type": "boolean",
        "description": "Option to avoid toll roads during the journey. Default is false."
      }
    },
    "required": [
      "start_location",
      "end_location"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_81",
  "answer": "{\"name\": \"map_routing.fastest_route\", \"arguments\": {\"start_location\": \"San Francisco\", \"end_location\": \"Los Angeles\", \"avoid_tolls\": true}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "map_routing.fastest_route",
      "args_accepted": {
        "start_location": [
          "San Francisco",
          "SF"
        ],
        "end_location": [
          "Los Angeles",
          "LA"
        ],
        "avoid_tolls": [
          true
        ]
      }
    }
  ],
  "category": "medium",
  "difficulty": "medium",
  "source_bfcl_id": "simple_python_81"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_192Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
Calculate the slope gradient in degree between two points on a landscape with coordinates (40.7128, -74.0060) and (34.0522, -118.2437).

FUNCTION SPEC (JSON):
{
  "name": "calculate_slope_gradient",
  "description": "Calculate the slope gradient between two geographical coordinates.",
  "parameters": {
    "type": "dict",
    "properties": {
      "point1": {
        "type": "array",
        "items": {
          "type": "float"
        },
        "description": "The geographic coordinates for the first point [Latitude, Longitude]."
      },
      "point2": {
        "type": "array",
        "items": {
          "type": "float"
        },
        "description": "The geographic coordinates for the second point [Latitude, Longitude]."
      },
      "unit": {
        "type": "string",
        "enum": [
          "degree",
          "percent",
          "ratio"
        ],
        "description": "The unit for the slope gradient. Default is 'degree'."
      }
    },
    "required": [
      "point1",
      "point2"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_192",
  "answer": "{\"name\": \"calculate_slope_gradient\", \"arguments\": {\"point1\": [40.7128, -74.006], \"point2\": [34.0522, -118.2437], \"unit\": \"degree\"}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "calculate_slope_gradient",
      "args_accepted": {
        "point1": [
          [
            40.7128,
            -74.006
          ]
        ],
        "point2": [
          [
            34.0522,
            -118.2437
          ]
        ],
        "unit": [
          "degree",
          ""
        ]
      }
    }
  ],
  "category": "medium",
  "difficulty": "medium",
  "source_bfcl_id": "simple_python_192"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_202Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
How many greenhouse gas emissions would I save if I switched to renewable energy sources for 3 months in California?

FUNCTION SPEC (JSON):
{
  "name": "calculate_emission_savings",
  "description": "Calculate potential greenhouse gas emissions saved by switching to renewable energy sources.",
  "parameters": {
    "type": "dict",
    "properties": {
      "energy_type": {
        "type": "string",
        "description": "Type of the renewable energy source."
      },
      "usage_duration": {
        "type": "integer",
        "description": "Usage duration in months."
      },
      "region": {
        "type": "string",
        "description": "The region where you use energy. Default is 'Texas'."
      }
    },
    "required": [
      "energy_type",
      "usage_duration"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_202",
  "answer": "{\"name\": \"calculate_emission_savings\", \"arguments\": {\"energy_type\": \"renewable\", \"usage_duration\": 3, \"region\": \"California\"}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "calculate_emission_savings",
      "args_accepted": {
        "energy_type": [
          "renewable"
        ],
        "usage_duration": [
          3
        ],
        "region": [
          "California",
          "CA"
        ]
      }
    }
  ],
  "category": "medium",
  "difficulty": "medium",
  "source_bfcl_id": "simple_python_202"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_395Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
Find the nearest parking lot within 2 miles of Central Park in New York.

FUNCTION SPEC (JSON):
{
  "name": "parking_lot.find_nearest",
  "description": "Locate the nearest parking lot based on a specific location and radius.",
  "parameters": {
    "type": "dict",
    "properties": {
      "location": {
        "type": "string",
        "description": "The reference location e.g. Central Park, NY"
      },
      "radius": {
        "type": "integer",
        "description": "The maximum distance from the location in miles. Default is 5 miles"
      },
      "type": {
        "type": "string",
        "description": "The type of parking lot. Default is 'public'."
      }
    },
    "required": [
      "location",
      "radius"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_395",
  "answer": "{\"name\": \"parking_lot.find_nearest\", \"arguments\": {\"location\": \"Central Park, NY\", \"radius\": 2, \"type\": \"public\"}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "parking_lot.find_nearest",
      "args_accepted": {
        "location": [
          "Central Park, NY"
        ],
        "radius": [
          2
        ],
        "type": [
          "public",
          ""
        ]
      }
    }
  ],
  "category": "medium",
  "difficulty": "medium",
  "source_bfcl_id": "simple_python_395"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_26Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
Calculate the velocity of a car that travels a distance of 50 kilometers for a duration of 2 hours?

FUNCTION SPEC (JSON):
{
  "name": "calculate_velocity",
  "description": "Calculate the velocity for a certain distance travelled within a specific duration.",
  "parameters": {
    "type": "dict",
    "properties": {
      "distance": {
        "type": "integer",
        "description": "The distance travelled by the object, typically in kilometers."
      },
      "duration": {
        "type": "integer",
        "description": "The duration of the journey, typically in hours."
      },
      "unit": {
        "type": "string",
        "description": "Optional parameter. The unit to return the velocity in. If not provided, the default is km/h."
      }
    },
    "required": [
      "distance",
      "duration"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_26",
  "answer": "{\"name\": \"calculate_velocity\", \"arguments\": {\"distance\": 50, \"duration\": 2}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "calculate_velocity",
      "args_accepted": {
        "distance": [
          50
        ],
        "duration": [
          2
        ],
        "unit": [
          "",
          "km/h"
        ]
      }
    }
  ],
  "category": "medium",
  "difficulty": "medium",
  "source_bfcl_id": "simple_python_26"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_180Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
Find the lawsuits filed against the company Google in California in the year 2020.

FUNCTION SPEC (JSON):
{
  "name": "lawsuits_search",
  "description": "Search for lawsuits against a specific company within a specific time and location.",
  "parameters": {
    "type": "dict",
    "properties": {
      "company_name": {
        "type": "string",
        "description": "The name of the company."
      },
      "location": {
        "type": "string",
        "description": "The location where the lawsuit was filed."
      },
      "year": {
        "type": "integer",
        "description": "The year when the lawsuit was filed."
      },
      "case_type": {
        "type": "string",
        "description": "The type of the case. Options include: 'civil', 'criminal', 'small_claims', etc. Default is 'all'."
      }
    },
    "required": [
      "company_name",
      "location",
      "year"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_180",
  "answer": "{\"name\": \"lawsuits_search\", \"arguments\": {\"company_name\": \"Google\", \"location\": \"California\", \"year\": 2020}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "lawsuits_search",
      "args_accepted": {
        "company_name": [
          "Google"
        ],
        "location": [
          "California",
          "CA"
        ],
        "year": [
          2020
        ],
        "case_type": [
          "",
          "all"
        ]
      }
    }
  ],
  "category": "medium",
  "difficulty": "medium",
  "source_bfcl_id": "simple_python_180"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_331Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
Find the top chess players in New York with a rating above 2300.

FUNCTION SPEC (JSON):
{
  "name": "board_game.chess.get_top_players",
  "description": "Find top chess players in a location based on rating.",
  "parameters": {
    "type": "dict",
    "properties": {
      "location": {
        "type": "string",
        "description": "The city you want to find the players from."
      },
      "minimum_rating": {
        "type": "integer",
        "description": "Minimum rating to filter the players."
      },
      "number_of_players": {
        "type": "integer",
        "default": 10,
        "description": "Number of players you want to retrieve, default value is 10"
      }
    },
    "required": [
      "location",
      "minimum_rating"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_331",
  "answer": "{\"name\": \"board_game.chess.get_top_players\", \"arguments\": {\"location\": \"New York\", \"minimum_rating\": 2300}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "board_game.chess.get_top_players",
      "args_accepted": {
        "location": [
          "New York",
          "New York City",
          "New York City, NY",
          "NYC"
        ],
        "minimum_rating": [
          2300
        ],
        "number_of_players": [
          "",
          10
        ]
      }
    }
  ],
  "category": "medium",
  "difficulty": "medium",
  "source_bfcl_id": "simple_python_331"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_194Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
What are the top three plants suitable for a hill slope in terms of erosion prevention?

FUNCTION SPEC (JSON):
{
  "name": "get_plants_for_slope",
  "description": "Retrieve the list of plants suitable for slope based on erosion control ability.",
  "parameters": {
    "type": "dict",
    "properties": {
      "slope_type": {
        "type": "string",
        "description": "The type of slope like steep, moderate etc."
      },
      "num_results": {
        "type": "integer",
        "description": "The number of top results needed. Default is 5."
      }
    },
    "required": [
      "slope_type",
      "num_results"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_194",
  "answer": "{\"name\": \"get_plants_for_slope\", \"arguments\": {\"slope_type\": \"hill\", \"num_results\": 3}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "get_plants_for_slope",
      "args_accepted": {
        "slope_type": [
          "hill",
          "steep",
          "moderate"
        ],
        "num_results": [
          3
        ]
      }
    }
  ],
  "category": "medium",
  "difficulty": "medium",
  "source_bfcl_id": "simple_python_194"
}

Scored by judge.py — see Scoring logic below for the full rule.

▸fn_358Output JSON function call matching schema

input

question.txt

You are given a function specification and a user request. Your job is to produce the JSON function call that fulfils the request, conforming exactly to the schema.

USER REQUEST:
Find a gluten-free cookie recipe that takes less than 30 minutes to prepare.

FUNCTION SPEC (JSON):
{
  "name": "recipe_search",
  "description": "Search for a cooking recipe based on specific dietary needs and time constraint.",
  "parameters": {
    "type": "dict",
    "properties": {
      "diet": {
        "type": "array",
        "items": {
          "type": "string",
          "enum": [
            "Gluten Free",
            "Dairy Free",
            "Vegan",
            "Vegetarian"
          ]
        },
        "description": "Specific dietary need."
      },
      "time_limit": {
        "type": "integer",
        "description": "The maximum time to prepare the recipe in minutes. Default is 60 minutes."
      },
      "dish": {
        "type": "string",
        "description": "The name of the dish to search for. Default is not use if not specified."
      }
    },
    "required": [
      "dish",
      "diet"
    ]
  }
}

OUTPUT FORMAT — output a single JSON object on stdout, nothing else (no explanation, no markdown fences, no preamble):

{"name": "<function_name>", "arguments": {"<arg_name>": <value>, ...}}

Required:
- Use the exact function name from the spec
- Include all REQUIRED parameters from the spec
- Match parameter types (integer / string / number / boolean / list)
- For optional params, omit them OR set them to a sensible default
- Output MUST be valid JSON, parseable by json.loads(), with no surrounding text.

expected output

answer.json

{
  "id": "fn_358",
  "answer": "{\"name\": \"recipe_search\", \"arguments\": {\"diet\": [\"Gluten Free\"], \"time_limit\": 30, \"dish\": \"cookie\"}}",
  "type": "json_function_call",
  "matchers": [
    {
      "kind": "json_call",
      "function_name": "recipe_search",
      "args_accepted": {
        "diet": [
          [
            "Gluten Free"
          ],
          [
            "GF"
          ],
          [
            "gluten free"
          ]
        ],
        "time_limit": [
          30
        ],
        "dish": [
          "cookie"
        ]
      }
    }
  ],
  "category": "medium",
  "difficulty": "medium",
  "source_bfcl_id": "simple_python_358"
}

Scored by judge.py — see Scoring logic below for the full rule.

scoring logic

judge.py runs once per case and prints a score per case. grader.py runs once at the end and folds case scores into a run-level summary. Without grader.py, the run's score is simply the average of case scores.

▸judge.py419 lines · view on GitHub
"""Per-case judge for the tenancy_agreement task — harsh by design.

Reads the agent's stdout (plain text OR JSON `{"answer": "..."}`) and applies
matchers declared in expected/{case_id}/answer.json. A case scores 1.0 only if
ALL matchers pass — partial credit is intentionally not offered. The whole
point of this task is to expose agents that hedge, miss clauses, or skip parts
of multi-part questions; lenient grading would defeat that.

Matcher kinds supported:
  - numeric          {"kind":"numeric","value":1234.5,"tolerance":0.01}
                     Passes if ANY number in the answer matches. Use for
                     show-your-working questions where the model walks
                     through arithmetic before stating the total.
  - leading_numeric  {"kind":"leading_numeric","value":1234.5,"tolerance":0.01}
                     The FIRST number in the answer must match. Use for
                     simple extraction questions where listing decoy
                     numbers should not count as a pass.
  - regex_required   {"kind":"regex_required","pattern":"...","flags":"i"}
                     Pattern must match (re.search). Default flags = i.
  - leading_word     {"kind":"leading_word","value":"yes"}
                     First alphanumeric token must equal value (case-insens),
                     after stripping common prefixes like "Answer:" or
                     markdown bold. Forces the model to commit, not hedge.
  - keywords_all     {"kind":"keywords_all","values":["a","b"]}
                     Every value must appear (case-insens substring).
  - keywords_any     {"kind":"keywords_any","values":["a","b"]}
                     At least one value must appear (case-insens substring).
  - keywords_any_word {"kind":"keywords_any_word","values":["ICE","BOE"]}
                     At least one value must appear as a whole word (\b...\b,
                     case-insens). Use for short acronyms that would
                     false-positive as substrings (ICE in "price", BOE in
                     "Boeing").
  - no_hedge         {"kind":"no_hedge"}
                     Reject answers that visibly punt the question, e.g.
                     "I cannot determine", "unclear from the document",
                     "I don't have access", "as an AI", etc.
  - min_words        {"kind":"min_words","value":5}
                     Reject one-word answers when the question asked for
                     reasoning/explanation.

Fallback (when no `matchers` provided):
  Substring match of `answer` (and any `accepted` variants) against the
  normalised agent output. Lenient but kept for cases that haven't been
  hardened yet (e.g. scenario_* cases without a curated gold).

Outputs JSON on stdout — trap stores it as CaseResult.metrics. The grader
reads `metrics.score` plus category/difficulty/reason for the report.
"""

from __future__ import annotations

import json
import os
import re
from pathlib import Path
from typing import Any

HEDGE_PHRASES = [
    "i cannot", "i can't", "i am unable", "i'm unable",
    "i don't have access", "i do not have access",
    "as an ai", "as a language model",
    "cannot determine", "unable to determine",
    "unclear from the document", "not clear from the document",
    "i don't know", "i do not know",
    "insufficient information", "not enough information",
    "i'm not sure", "i am not sure",
]

NUMBER_RE = re.compile(r"-?\d[\d,]*(?:\.\d+)?")


def normalise(s: str) -> str:
    return re.sub(r"\s+", " ", s).strip().lower()


def extract_agent_answer(stdout: str) -> str:
    """Accept JSON {"answer": "..."} or plain text. Strip surrounding whitespace."""
    stdout = stdout.strip()
    if not stdout:
        return ""
    try:
        obj = json.loads(stdout)
        if isinstance(obj, dict) and "answer" in obj:
            return str(obj["answer"])
    except json.JSONDecodeError:
        pass
    return stdout


def parse_numeric(s: str) -> float | None:
    """Extract the first plausible number from `s`. £, $, commas, spaces stripped."""
    nums = parse_all_numerics(s)
    return nums[0] if nums else None


def parse_all_numerics(s: str) -> list[float]:
    """Extract ALL plausible numbers from `s`. Used to match agents that show
    working (e.g. "£1,950 × 12 + ... = £77,400" — we want to find 77400)."""
    if not s:
        return []
    cleaned = s.replace("£", "").replace("$", "").replace(",", "")
    out: list[float] = []
    for m in NUMBER_RE.finditer(cleaned):
        try:
            out.append(float(m.group(0).replace(",", "")))
        except ValueError:
            continue
    return out


_LEADING_LABEL_RE = re.compile(
    r"^\s*(?:answer|a|response|reply)[\s*_`]*:\s*", re.IGNORECASE,
)
_LEADING_NOISE_RE = re.compile(r"^[\s*_`#>\-]+")


def leading_word(s: str) -> str:
    """First alpha token, after stripping markdown noise and labels like
    "Answer:" / "**Answer**:" / "> ". Lets models prefix their commit with
    a natural label without auto-failing the case."""
    s = _LEADING_NOISE_RE.sub("", s)
    s = _LEADING_LABEL_RE.sub("", s)
    s = _LEADING_NOISE_RE.sub("", s)
    m = re.search(r"[a-zA-Z]+", s)
    return m.group(0).lower() if m else ""


# --- Matcher implementations ----------------------------------------------

def m_numeric(answer: str, spec: dict) -> tuple[bool, str]:
    """Pass if ANY number in the answer matches the target within tolerance.
    This lets models that show working ("1950 × 12 + 2100 × 12 = 77400") pass
    as long as the right number appears somewhere — exposing the actual answer
    is what matters, not whether the model led with it. For simple extraction
    where listing decoys should NOT pass, use `leading_numeric` instead."""
    nums = parse_all_numerics(answer)
    if not nums:
        return False, "no number found in answer"
    target = float(spec["value"])
    tol = float(spec.get("tolerance", 0.01))
    for n in nums:
        if abs(n - target) <= tol:
            return True, f"numeric ok (matched {n} of {nums} against target={target} tol={tol})"
    return False, f"numeric mismatch (numbers found={nums} target={target} tol={tol})"


def m_leading_numeric(answer: str, spec: dict) -> tuple[bool, str]:
    """First number in the answer must match within tolerance. Rejects
    decoy-number dumps like "rent 1950, deposit 2250, rent yr2 2100"
    where the target appears but isn't the committed answer."""
    nums = parse_all_numerics(answer)
    if not nums:
        return False, "no number found in answer"
    target = float(spec["value"])
    tol = float(spec.get("tolerance", 0.01))
    if abs(nums[0] - target) <= tol:
        return True, f"leading number ok ({nums[0]} == target {target} tol {tol})"
    return False, f"leading number {nums[0]} ≠ target {target} (other numbers in answer: {nums[1:]})"


def m_regex_required(answer: str, spec: dict) -> tuple[bool, str]:
    flags = 0
    if "i" in spec.get("flags", "i"):
        flags |= re.IGNORECASE
    if re.search(spec["pattern"], answer, flags):
        return True, f"regex matched"
    return False, f"regex {spec['pattern']!r} did not match"


def m_leading_word(answer: str, spec: dict) -> tuple[bool, str]:
    got = leading_word(answer)
    want = str(spec["value"]).lower()
    if got == want:
        return True, f"leading word ok ({got!r})"
    return False, f"leading word {got!r} ≠ required {want!r}"


def m_keywords_all(answer: str, spec: dict) -> tuple[bool, str]:
    norm = normalise(answer)
    missing = [v for v in spec["values"] if v.lower() not in norm]
    if missing:
        return False, f"missing required keyword(s): {missing}"
    return True, "all keywords present"


def m_keywords_any(answer: str, spec: dict) -> tuple[bool, str]:
    norm = normalise(answer)
    if any(v.lower() in norm for v in spec["values"]):
        return True, "at least one keyword present"
    return False, f"none of {spec['values']} present"


def m_keywords_any_word(answer: str, spec: dict) -> tuple[bool, str]:
    """Whole-word variant of keywords_any — wraps each value in \\b...\\b so
    short acronyms (ICE, BOE) don't false-match inside "price", "Boeing", etc."""
    for v in spec["values"]:
        if re.search(rf"\b{re.escape(v)}\b", answer, re.IGNORECASE):
            return True, f"whole-word match: {v!r}"
    return False, f"none of {spec['values']} matched as whole word"


def m_no_hedge(answer: str, spec: dict) -> tuple[bool, str]:
    norm = normalise(answer)
    for phrase in HEDGE_PHRASES:
        if phrase in norm:
            return False, f"hedge phrase detected: {phrase!r}"
    return True, "no hedge phrases"


def m_min_words(answer: str, spec: dict) -> tuple[bool, str]:
    count = len(re.findall(r"\S+", answer))
    want = int(spec["value"])
    if count >= want:
        return True, f"word count ok ({count} ≥ {want})"
    return False, f"too short ({count} < {want})"


MATCHERS = {
    "numeric": m_numeric,
    "leading_numeric": m_leading_numeric,
    "regex_required": m_regex_required,
    "leading_word": m_leading_word,
    "keywords_all": m_keywords_all,
    "keywords_any": m_keywords_any,
    "keywords_any_word": m_keywords_any_word,
    "no_hedge": m_no_hedge,
    "min_words": m_min_words,
}


def run_matchers(answer: str, matchers: list[dict]) -> tuple[float, list[dict]]:
    """Run all matchers; all must pass. Returns (score, per-matcher results)."""
    results = []
    all_ok = True
    for spec in matchers:
        kind = spec.get("kind")
        fn = MATCHERS.get(kind)
        if fn is None:
            results.append({"kind": kind, "pass": False, "reason": f"unknown matcher kind: {kind!r}"})
            all_ok = False
            continue
        ok, reason = fn(answer, spec)
        results.append({"kind": kind, "pass": ok, "reason": reason})
        if not ok:
            all_ok = False
    return (1.0 if all_ok else 0.0), results


def fallback_substring(answer: str, expected: dict) -> tuple[float, str]:
    """Lenient substring match when no matchers defined. Used for scenarios
    that don't have a curated gold yet — they shouldn't fail builds outright,
    but they also shouldn't claim a passing score from nothing."""
    targets = [t for t in [expected.get("answer"), *(expected.get("accepted") or [])] if t]
    if not targets:
        return 0.0, "no gold answer set (skip-equivalent)"
    norm = normalise(answer)
    hit = next((t for t in targets if normalise(t) in norm), None)
    if hit:
        return 1.0, f"substring match ({hit!r})"
    return 0.0, f"no substring match against {targets}"


# --- Main ------------------------------------------------------------------

def main() -> None:
    # trap-cli IO contract: TRAPTASK_MANIFEST carries directory + capture paths.
    m = json.loads(os.environ["TRAPTASK_MANIFEST"])

    stdout = Path(m["run"]["stdout"]).read_text()
    exit_code = json.loads(Path(m["run"]["meta"]).read_text())["exit_code"]
    expected = json.loads((Path(m["expected_dir"]) / "answer.json").read_text())

    agent_answer = extract_agent_answer(stdout)

    # Solution crashed → hard fail.
    if exit_code != 0:
        out: dict[str, Any] = {
            "score": 0.0,
            "reason": f"solution exited {exit_code}",
            "agent_answer": agent_answer,
            "id": expected.get("id"),
            "category": expected.get("category"),
            "difficulty": expected.get("difficulty"),
        }
        print(json.dumps(out))
        return

    # Empty stdout → hard fail (silently passing the test is the worst outcome).
    if not agent_answer:
        out = {
            "score": 0.0,
            "reason": "agent produced no answer",
            "agent_answer": "",
            "id": expected.get("id"),
            "category": expected.get("category"),
            "difficulty": expected.get("difficulty"),
        }
        print(json.dumps(out))
        return

    matchers = expected.get("matchers")
    if matchers:
        score, matcher_results = run_matchers(agent_answer, matchers)
        out = {
            "score": score,
            "matcher_results": matcher_results,
            "agent_answer": agent_answer,
            "expected_answer": expected.get("answer"),
            "id": expected.get("id"),
            "type": expected.get("type"),
            "category": expected.get("category"),
            "difficulty": expected.get("difficulty"),
        }
    else:
        score, reason = fallback_substring(agent_answer, expected)
        # If there's no gold and no matchers, surface score=None so the grader
        # can flag it as "not yet curated" rather than mark the agent failed.
        if expected.get("answer") is None:
            out = {
                "score": None,
                "reason": "no curated gold yet (case not gradeable)",
                "agent_answer": agent_answer,
                "id": expected.get("id"),
                "type": expected.get("type"),
                "category": expected.get("category"),
                "difficulty": expected.get("difficulty"),
            }
        else:
            out = {
                "score": score,
                "reason": reason,
                "agent_answer": agent_answer,
                "expected_answer": expected.get("answer"),
                "id": expected.get("id"),
                "type": expected.get("type"),
                "category": expected.get("category"),
                "difficulty": expected.get("difficulty"),
            }

    print(json.dumps(out))




# ─── json_call matcher (added for core_json_schema_output) ──────────────────

def m_json_call(answer: str, spec: dict) -> tuple[bool, str]:
    """Parse answer as a JSON function call, verify name + args match spec.

    spec: {
      "kind": "json_call",
      "function_name": "calculate_triangle_area",
      "args_accepted": {
        "base": [10],
        "height": [5],
        "unit": ["units", ""]    # multiple acceptable values; "" means optional/default
      }
    }

    Rules:
      - Answer must be valid JSON (markdown fences ``` are stripped)
      - Must be a dict with "name" (or "function"/"function_name") and "arguments" (or "args"/"parameters")
      - The function name must match spec.function_name exactly
      - For each arg in args_accepted, EITHER:
          * model's value matches one of the accepted values (loose ==, with int/float coercion)
          * OR "" is in the accepted list AND the arg is omitted from model's call (optional default)
      - Extra args in the model's call are NOT penalised
    """
    import json as _json, re as _re
    cleaned = answer.strip()
    cleaned = _re.sub(r"^```(?:json)?\s*", "", cleaned)
    cleaned = _re.sub(r"```\s*$", "", cleaned)
    try:
        obj = _json.loads(cleaned)
    except _json.JSONDecodeError as e:
        return False, f"invalid JSON: {e}"
    if not isinstance(obj, dict):
        return False, f"top level is {type(obj).__name__}, expected dict"
    got_name = obj.get("name") or obj.get("function") or obj.get("function_name")
    if got_name != spec["function_name"]:
        return False, f"function name mismatch: got {got_name!r}, want {spec['function_name']!r}"
    got_args = obj.get("arguments") or obj.get("args") or obj.get("parameters") or {}
    if not isinstance(got_args, dict):
        return False, f"arguments is {type(got_args).__name__}, expected dict"

    def loose_eq(a, b):
        if a == b: return True
        # numeric coercion
        try:
            if float(a) == float(b): return True
        except (ValueError, TypeError):
            pass
        # string/list ordering
        if isinstance(a, list) and isinstance(b, list) and len(a) == len(b):
            return all(loose_eq(x, y) for x, y in zip(a, b))
        return False

    for arg_name, accepted in spec["args_accepted"].items():
        is_optional = "" in accepted
        if arg_name not in got_args:
            if is_optional:
                continue  # OK to omit
            return False, f"missing required arg {arg_name!r}"
        got_val = got_args[arg_name]
        # Strip "" placeholder when checking
        real_accepted = [v for v in accepted if v != ""]
        if not any(loose_eq(got_val, v) for v in real_accepted):
            if is_optional and got_val in (None, ""):
                continue
            return False, f"arg {arg_name}: got {got_val!r}, accepted {real_accepted}"
    return True, "json call matches spec"


MATCHERS["json_call"] = m_json_call


if __name__ == "__main__":
    main()
▸grader.py86 lines · view on GitHub
"""Overall grader for the tenancy_agreement task.

Aggregates per-case judge results into a run-level verdict. Emits JSON to stdout —
trap stores it as GraderResult.metrics. Convention: include `passed` (bool) and
`score` (float) so the reporter can render them.

Pass threshold defaults to 80% accuracy; tweak below.
"""
from __future__ import annotations

import json
import os
from collections import Counter

PASS_THRESHOLD = 0.80


def main() -> None:
    # trap-cli passes the list of per-case results directly (case_id, exit_code,
    # duration, metrics, cost).
    cases = json.loads(os.environ["TRAPTASK_MANIFEST"])

    scored = [c for c in cases if c.get("metrics") and c["metrics"].get("score") is not None]
    skipped = [c for c in cases if not c.get("metrics") or c["metrics"].get("score") is None]

    if scored:
        accuracy = sum(c["metrics"]["score"] for c in scored) / len(scored)
    else:
        accuracy = 0.0

    # Break out accuracy by category (the judge tags each case with its category).
    by_category_score: Counter[str] = Counter()
    by_category_total: Counter[str] = Counter()
    for c in scored:
        cat = c["metrics"].get("category")
        if cat:
            by_category_total[cat] += 1
            by_category_score[cat] += c["metrics"]["score"]

    by_category_pct = {
        k: round(by_category_score[k] / by_category_total[k], 3)
        for k in by_category_total
    }

    passed = bool(scored) and accuracy >= PASS_THRESHOLD

    # Latency stats — trap records `duration` (seconds) per case. The leaderboard
    # displays median latency. Round-trip to ms for the JSON contract.
    durations = [c.get("duration", 0.0) for c in cases if c.get("duration") is not None]
    if durations:
        ds = sorted(durations)
        latency_ms_median = round(ds[len(ds) // 2] * 1000, 1)
        latency_ms_p95 = round(ds[int(0.95 * len(ds))] * 1000, 1) if len(ds) > 1 else latency_ms_median
        latency_ms_total = round(sum(ds) * 1000, 1)
    else:
        latency_ms_median = latency_ms_p95 = latency_ms_total = 0.0

    # Cost — trap-cli's proxy records a per-case `cost` object {cost_usd, by_model, ...}.
    case_costs = [
        c["cost"]["cost_usd"]
        for c in cases
        if isinstance(c.get("cost"), dict) and c["cost"].get("cost_usd") is not None
    ]
    cost_usd_total = round(sum(case_costs), 4) if case_costs else None

    n_passed = sum(1 for c in scored if c["metrics"]["score"] == 1.0)

    print(json.dumps({
        "passed": passed,
        "score": round(accuracy, 3),
        "n_passed": n_passed,
        "n_total": len(cases),
        "n_scored": len(scored),
        "n_skipped_no_gold": len(skipped),
        "threshold": PASS_THRESHOLD,
        "by_category": by_category_pct,
        "latency_ms_median": latency_ms_median,
        "latency_ms_p95": latency_ms_p95,
        "latency_ms_total": latency_ms_total,
        "cost_usd_total": cost_usd_total,
    }))


if __name__ == "__main__":
    main()