Learning Board Game Rules from an Instruction Manual

Abstract

Board game rulebooks offer a convenient scenario for extracting a systematic logical structure from a passage of text since the mechanisms by which board game pieces interact must be fully specified in the rulebook and outside world knowledge is irrelevant to gameplay. A representation was proposed for representing a game's rules with a tree structure of logically-connected rules, and this problem was shown to be one of a generalized class of problems in mapping text to a hierarchical, logical structure. Then a keyword-based entity- and relation-extraction system was proposed for mapping rulebook text into the corresponding logical representation, which achieved an f-measure of 11% with a high recall but very low precision, due in part to many statements in the rulebook offering strategic advice or elaboration and causing spurious rules to be proposed based on keyword matches. The keyword-based approach was compared to a machine learning approach, and the former dominated with nearly twenty times better precision at the same level of recall. This was due to the large number of rule classes to extract and the relatively small data set given this is a new problem area and all data had to be manually annotated. This provided insufficient training data the machine learning approach, which performs better on large data sets with small numbers of extraction classes. The keyword-based approach was subsequently improved with a set of domain-specific filters using context information to remove likely false positives. This improved precision over the baseline system by 296% at an expense of an 11% drop in recall, an f-measure improvement from 16% to 49% on the rule extraction subtask which is the system's bottleneck. The overall system's f-measure improved from 11% to 30%, providing a substantial improvement though leaving plenty of opportunities for future work.