Language models increasingly support or automate information retrieval from document collections too large to index. This task becomes difficult when it requires specialist linguistic or scientific knowledge. Hickey, Knight, and Bishop used the Wikipedia Game to test language models on an analogous information retrieval task. In the game, players must reach a target article using only article links. The authors compared sentence embedding models and large language models across five Wikipedia languages and five academic-literature datasets. They constructed very large local graphs of Wikipedia pages and of paper citations using OpenAlex and provided code to build and navigate them. This approach allows study of language models' navigation capabilities without relying on third-party APIs. The authors compared models with human and TF-IDF benchmarks. The Wikipedia Game format enables robust comparisons across techniques, languages, and domains. No model significantly outperformed humans, although the best models—GPT-o3 mini, GPT-4o, QwQ:32B, and DeepSeek-R1:70B—achieved comparable success. Performance and rankings varied across languages and topics, with implications for practitioners utilising these techniques.
Wikipedia as a Graph
English Wikipedia contains over six million articles, which contain nearly 300 million links between them, making it the largest reference work in history. This encyclopedia can be conceptualised as a graph comprising connected nodes (the articles) and edges (the hyperlinks between the articles). Doing so has enabled other researchers to construct a knowledge graph or identify useful missing links. As Wikipedia is both human-written text and organised within a structured format, it has been harnessed for topic classification, improving the performance of foundation models, and providing reliable ground truth data for named entity recognition.
A directed graph is created for each language using the Wikipedia datasets, with the nodes representing articles and the edges representing links between the articles. The title and a summary containing the start of the article are stored with each node. Note that the graph is not symmetric: if Article A has a link to Article B, this does not guarantee that Article B contains a link to Article A.
Datasets: Five Languages and OpenAlex Domains
Most results are presented for English Wikipedia, however the authors tested models on a collection of different graph datasets so that performance across languages and technical subjects can be explored. Five datasets in different languages were obtained from Wikipedia, while the subject-specific datasets were compiled from a collection of abstracts in the scientific literature. Statistics comprising these datasets can be found in the appendix.
English Wikipedia contains over six million articles and four and a half billion words. Four of the five languages are considered high-resource languages (English, French, Chinese, and Russian), whereas Bengali is considered a lower-resource language. A directed graph is created for each language using the Wikipedia datasets, with the nodes representing articles and the edges representing links between the articles. The title and a summary containing the start of the article are stored with each node.
The OpenAlex dataset was categorised based on each paper's primary academic field to create multiple graphs. Using the primary topic categorisation, five subjects were chosen: Biology, Medicine, Physics, Computer Science, and Social Sciences. Unlike the Wikipedia dataset, the OpenAlex datasets are symmetric graphs because if Article A references Article B, then Article B will be cited by Article A. The connectivity of the graphs reflects the referencing patterns within each academic field, leading to significant variations in the graphs' navigability.
Agents: Sentence Embeddings, Prompting LLMs, and Reasoning Models
This work considers two types of language models: sentence embedding models and large language models. The setup for all these methods is optimised to compare how these techniques might be used in a real-world setting.
Sentence Embeddings
Sentence embedding models map text to numerical vectors within a high-dimensional space where semantically similar sentences are positioned closely together. The authors designed an agent that used a pre-trained sentence embedding model to select the link at each stage of the game with the 'closest' summary sentence or title to the target summary sentence. The 'closest' link was determined by the minimum cosine distance. Additional logic prevents them from returning to an article they have already visited to avoid becoming trapped in loops.
To aid readability, the authors focused on the performance of five embedding models studied: E5-Small, a multilingual embedding model produced by Microsoft; AllMPNetBase, recommended by the Sentence Transformers documentation and trained only on English language data; MiniLM-L6, one of the smallest models considered with only 22.7 million parameters; Roberta, which makes use of the well-known BERT architecture and is fine-tuned to produce an embedding model; and Glove, which works by averaging the individual word embedding, making it much faster in inference than similarly sized models. The other models are listed in the appendix.
Prompting LLMs
Large language models used here generate human-like text by predicting each word incrementally. By crafting detailed and specific starting text, known as the prompt, LLMs can be directed to perform precise tasks such as code synthesis, text labelling, and machine translation. Including all the necessary information in the prompt is crucial for maximising the capabilities of these models.
This work used thirteen different LLMs, including the full suite of Llama3 models, which range from 1 to 405 billion parameters. A Gemma2 model was also used, which has 27 billion parameters. OpenAI's GPT-4o was the largest LLM used in this work with its smaller version, GPT-4o mini. Due to the cost of running GPT-4o, the number of games played using this agent was limited to 100 and restricted to only a few datasets. There were also five 'reasoning (R)' LLMs evaluated, which included DeepSeek R1 and Qwen with Questions (QwQ).
Two key adjustments were implemented to address the issue of providing significant information to the LLM, which can negatively affect its performance. Firstly, a maximum limit of 500 links for GPT and 200 links for other models from the current article was established. In preliminary experiments, GPT-4o was able to cope with a longer prompt and still succeeded, but the success rate of all LLMs improved when the number of titles was limited. Secondly, the prompt included only the titles of the linked articles rather than the article summaries. These adaptations greatly improved the reliability of the prompting method.
As an LLM can, in principle, output any text, a title that does not match any in the list provided may be generated. In this case, the LLM is prompted again up to four more times with an extra paragraph emphasising the task. For experimental consistency, the same prompt was used for all non-reasoning LLMs, and the article limit was set high enough to only affect a small minority of articles.
Reasoning LLMs
This work also considered evaluating an LLM in reasoning mode where the LLM can generate a 'thinking' block of text before providing the answer. Eight of these LLMs were evaluated in this way. For GPT-o3 mini, this was done by using the same prompt as with the GPT-4o model, but allowing reasoning through the API. The open-source models were given chain-of-thought prompting and could provide an answer (contained in <answer> tags) in the output text. The DeepSeek (DS) and QwQ models are trained for this behaviour, while the Llama and Gemma models were not. To distinguish which setting the models were evaluated in, an (R) is placed after the name of the models evaluated in the reasoning format. Other than the requested format, the experiment was the same for the reasoning and non-reasoning LLMs.
TF-IDF
A non-deep-learning benchmark was created using term frequency-inverse document frequency (TF-IDF), a statistical approach that assesses the importance of a word to a document based on its frequency in the corpus. For each dataset, a corpus was created using a sample of 100,000 articles from that dataset. The articles used in the sample are not held separately and may appear in the game.
Humans
Nine human players attempted a sample of games for the English Wikipedia dataset to act as a benchmark. These games were not played under strict experimental conditions, and the players were requested to self-report their success. A time limit of 10 minutes was used instead of a step limit, with players aware of this limit. In total, 109 games were played, and an overlap of games was avoided between the players.
Model Performance on English Wikipedia
Success Rate
Figure 2A shows the success rate of a selection of the models on the English Wikipedia graph. A game was considered a success if the model reached the target article from the start article in 100 steps. GPT-o3 mini (R) had the highest success rate, comparable to that for humans. Many sentence embedding models were able to perform similarly to the largest of the open non-reasoning LLMs. The optimum formulation was used for both model types, so the embedding models used summaries of the linked articles, whereas the LLMs were only shown the title.
Unlike the embedding models, the performance of the LLMs severely deteriorated with decreasing model size, which may be related to the model's ability to stay on task reliably. This is seen in the much higher tendency for the smaller LLMs to hallucinate. Reasoning models did manage to provide more accurate answers for the same-sized LLM, but the smallest reasoning model, DS-R1:8b, did not complete the task. TF-IDF was also effective at the task. The corpus for the TF-IDF model was generated with a sample from the article summaries used to play the game.
Efficiency
Figure 3 examines the efficiency of the various agents for the games where they succeeded, as measured by the number of steps taken to reach the target article. GPT-4o can succeed in fewer steps for a greater proportion of games than E5-Small. Even though both approaches achieved a comparable success rate, humans were able to succeed with fewer steps. The LLMs of Llama:405B and GPT-4o mini can be more efficient than the sentence embedding models. Trained reasoning models were also comparable in efficiency to the non-reasoning and embedding models of similar performance. The language models' step distributions did not differ measurably from a geometric distribution, but the human games do, even when games more than 10 steps above optimal are ignored.
Languages and Topics
TF-IDF and Embedding Models
The E5-Small embedding model achieved the highest success rates (within a margin of error) across all five languages of the Wikipedia datasets. The TF-IDF approach is sometimes competitive with the embedding models, and provides a good statistical comparison across the datasets. However, this achievement does not translate to the OpenAlex datasets where the embedding model AllMPNetBase achieves a higher success rate. AllMPNetBase is a monolingual embedding model, so would not be expected to perform well on Wikipedia in other languages, although it does achieve a reasonable success rate in French. The vast majority of the OpenAlex articles are written in English, but the success rate of AllMPNetBase over E5-Small for Biology, Medicine and Physics is significant, suggesting they have different representations of scientific language. The results demonstrate that the relative performance of embedding models on one dataset will not translate to the context of another dataset.
LLMs
Figure 4B shows the success rate of LLMs on eight of the graph datasets. For the LLMs, the success rate is ordered roughly by model size but is not consistent between topics. The GPT-4o models are mostly consistent across the range of languages and topics, with the GPT-4o mini's performance not statistically significantly dropping below the performance on English Wikipedia. The success rate of Llama:405B deteriorated when operating in many languages and topics outside of English Wikipedia. When tested on the OpenAlex dataset, the performance of the 405 billion-parameter model dropped by around a third, and the 3 billion-parameter Llama model had a success rate near or at zero. This suggests that, as with the embedding models, general aptitude might not transfer between subject areas.
Discussion
This report shows that sentence embedding models and LLMs can contain sufficient semantic information to play the Wikipedia Game successfully. While the best-performing models are able to achieve a comparable success rate to humans on English Wikipedia, it took more steps on average, demonstrating that it is still not as efficient as humans at this task. The best LLMs used in the experimental work achieved a comparable success rate to humans, though they took more steps on average than a human. Performance deteriorated with decreasing size of the LLM, and hallucinations often negatively impacted their performance. By contrast, the sentence embedding models do not suffer from hallucinations and some were able to outperform many LLMs, achieving higher success rates. Reasoning models achieved much better performance for their size, but their efficiency more closely matched the non-reasoning and embedding models. It is important to note that running a reasoning LLM is vastly more computationally expensive than an embedding model, and in many practical circumstances the simpler approach would be optimal.
The results showed that the models' relative performance was not consistent between topics as their ranking changed between languages and academic fields. None of the models significantly exceeded human performance on the English Wikipedia dataset, though this result would not necessarily translate to the more technical datasets, such as Biology, Medicine, or Physics, where specialist knowledge would be required. Furthermore, given the relative time and monetary costs of human processing compared to language models, there will be practical tasks where these models are faster and cheaper than human labour, though this work shows that care must be taken when applying these techniques across domains.
Many characteristics of the OpenAlex datasets are different from English Wikipedia, with fewer links per article and longer titles with more specialist language. Despite this, the success rates for some of the LLMs on the Biology, Medicine and Physics datasets were within the margin of error of the success rate on the English Wikipedia dataset. This indicates that the LLMs are not simply recalling links that they have already 'learned' during training on Wikipedia data but have some ability to generalise the concept of the game to a different dataset. A very small sample of target Wikipedia articles published in 2024, after the models were trained, were explored. These articles are less well connected, so intrinsically more difficult to reach, but every model's success rate fell proportionally less than TF-IDF's.
There are some natural extensions to this work; for example, one could apply the large GPT-4o model and gather more benchmark data from humans playing the game on all datasets. Many native English speakers might struggle to navigate OpenAlex datasets owing to the specialised nature of the language. Cross-encoder models typically have a computational cost somewhere between the embedding and the LLM approaches used in this work. Other potential approaches include hybrid methods, such as presenting only the top choices from the embedding model in an LLM prompt or using RL to fine-tune an existing model.
Conclusions
This project has investigated whether sentence embedding models and LLMs contain sufficient semantic information about related concepts to navigate an interconnected graph of articles. 26 different language models are tested on 10 graph datasets generated from Wikipedia and OpenAlex. This experimental set-up resembled a real-world task, motivating applicability and allowing comparison with a human benchmark on the English Wikipedia dataset.
The performance of the models varied greatly on type, size, and testing domain. Notably, the best LLMs used in the experimental work achieved a comparable success rate to humans, though they took more steps on average than a human. Performance deteriorated with decreasing size of the LLM, and hallucinations often negatively impacted their performance. By contrast, the sentence embedding models do not suffer from hallucinations and some were able to outperform many LLMs, achieving higher success rates. Reasoning models achieved much better performance for their size, but their efficiency more closely matched the non-reasoning and embedding models. It is important to note that running a reasoning LLM is vastly more computationally expensive than an embedding model, and in many practical circumstances the simpler approach would be optimal.
The results showed that the models' relative performance was not consistent between topics as their ranking changed between languages and academic fields. While none of the models significantly exceeded human performance on the English Wikipedia dataset, this result would not necessarily translate to the more technical datasets, such as Biology, Medicine, or Physics, where specialist knowledge would be required. Furthermore, given the relative time and monetary costs of human processing compared to language models, there will be practical tasks where these models are faster and cheaper than human labour, though this work shows that care must be taken when applying these techniques across domains.