Terminal benchmark task inspection

Gold Patch ⬇ diff -- git a / sphinx / util / rst . py b / sphinx / util / rst . py --- a / sphinx / util / rst . py +++ b / sphinx / util / rst . py @@ -10,22 +10,17 @@ from docutils . parsers . rst import roles from docutils . parsers . rst . languages import en as english + from docutils . parsers . rst . states import Body from docutils . statemachine import StringList from docutils . utils import Reporter - from jinja2 import Environment + from jinja2 import Environment , pass_environment from sphinx . locale import __ from sphinx . util import docutils , logging - try : - from jinja2 . utils import pass_environment - except ImportError : - from jinja2 import environmentfilter as pass_environment - - logger = logging . getLogger ( __name__ ) - docinfo_re = re . compile ( ’:\\w+:.*?’ ) + FIELD_NAME_RE = re . compile ( Body . patterns [ ’field_marker’ ]) symbols_re = re . compile ( r’([!-\-/:-@\[-‘{-~])’ ) # symbols without dot(0x2e) SECTIONING_CHARS = [ ’=’ , ’-’ , ’~’ ] @@ -80,7 +75,7 @@ def prepend_prolog ( content : StringList , prolog : str ) -> None : if prolog : pos = 0 for line in content : - if docinfo_re . match ( line ): + if FIELD_NAME_RE . match ( line ): pos += 1 else : break @@ -91,6 +86,7 @@ def prepend_prolog ( content : StringList , prolog : str ) -> None : pos += 1 # insert prolog (after docinfo if exists) + lineno = 0 for lineno , line in enumerate ( prolog . splitlines ()): content . insert ( pos + lineno , line , ’<rst_prolog>’ , lineno ) Discussion. For this task instance from the sphinx-doc/sphinx repository, a model is asked to write logic to fix a case where the title is incorrectly being rendered. Simply understanding the jargon being used and mapping such words to logic within the codebase is a significant challenge faced by the model. The model is given a command line call that can help with this, but grounding the terminology presented in the issues within the codebase is essential. From comparing the gold patch and model generated patch, it is clear that the model does not come close to solving the task. The model does generally identify that fixing the regex pattern is the correct action, as this is what the gold patch does, too. However, where the model and oracle retrieval setting collectively fall short is mainly due to the significant use of additional modules from both the codebase itself and third party libraries. This example highlights the importance and potential for training language models and designing inference procedures that allow for the automated discovery of such information.

As discussed in Section 3 , the File Viewer is fundamental to a language agent’s ability to understand file content and invoke appropriate edits. In a Terminal-only setting, there are several commands that can be used to inspect file content. However, out of the box command line tools are sub-optimal or limiting for language agents for several reasons. First, commands that print files to standard output (e.g. cat , printf ) can easily flood a language agent’s context window with too much file content, the majority of which is usually irrelevant to the issue. Enabling a language agent to filter out distractions and focus on relevant code snippets is crucial to generating effective edits. While commands like head and tail reduce length to the first/last n lines, it is not intuitive to use bash commands to perform in-file navigation. It is either impossible or requires a long list of arguments to show specific file lines. Furthermore, since such Bash commands are stateless, “scrolling” up/down relative to the current file position typically requires regenerating the same lengthy command with minor changes. Interactive tools like more and less accommodate this, but (1) representing navigation actions (multiple key up/down clicks) is intuitive for humans, but is verbose and costly for language agents, and (2) even if jumping to a specific line number is allowed, it is not possible to quickly identify what classes/methods/symbols are declared in a file and go to their definitions.

Agent Runtime provides a general environment that equips the agent with an action space comparable to that of human software developers, enabling OpenDevin agents to tackle a wide range of software development and web-based tasks, including complex software development workflows, data analysis projects, web browsing tasks, and more. It allows the agent to access a bash terminal to run code and command line tools, utilize a Jupyter notebook for writing and executing code on-the-fly, and interact with a web browser for web-based tasks ( e.g . , information seeking).

We evaluate three popular command-line agents (Claude Code, Codex CLI, and Gemini CLI) and three open-source software engineering agents (OpenHands ( Wang et al., 2025 ) , Mini-SWE-Agent ( Yang et al., 2024 ) , and Terminus 2) on Terminal-Bench 2.0.

Command line interface (CLI) agents have become a prominent setting for coding and computer use, studied in a large body of prior work [ 54 , 65 , 55 , 18 , 37 , 2 , 63 , 34 , 27 , 75 ] . CLI agents operate directly in filesystem environments through shell commands, treating executable code as their native action space rather than calling predefined tool APIs. This interface gives language agents the same operational substrate used by developers, including directory exploration, program execution, artifact editing, and result checking through terminal feedback.