Description

Question answering over knowledge graphs (KG-QA) is a vital topic in IR. Questions with temporal intent are a special class of practical importance, but have not received much attention in research. We present EXAQT, the first end-to-end system for answering complex temporal questions that have multiple entities and predicates, and associated temporal conditions.

EXAQT: EXplainable Answering of Complex Questions with Temporal Intent

EXAQT answers natural language questions over KGs in two stages, one geared towards high recall, the other towards precision at top ranks. The first step computes question-relevant compact subgraphs within the KG, and judiciously enhances them with pertinent temporal facts, both using fine-tuned BERT models. The second step constructs relational graph convolutional networks (R-GCNs) from the first step’s output, and enhances the R-GCNs with time-aware entity embeddings and attention over temporal relations.

A simplified excerpt of the relevant zone in the Wikidata KG necessary for answering the question is shown in the figure.

Running Example in EXAQT
Wikidata KG excerpt for "where did obama's children study when he became president?"
with answer "Sidwell Friends School"

Complex questions of this sort must consider multi-hop constraints (Barack Obama → Malia Obama, Sasha Obama → Sidwell Friends School, University of Chicago Laboratory School), and reason on the overlap of the intersection of the start of the presidency (2009) with the study period at the school (2009 – 2016).

TimeQuestions

To evaluate EXAQT, we leverage recent community efforts in QA benchmark, and we search through eight KG-QA datasets for time-related questions. The result is a new compilation with about 16,181 questions called TimeQuestions.

We tag each question with its temporal category, link the answers to Wikidata and Wikipedia, and split the benchmark in a 60 : 20 : 20 ratio for creating the training (9708 questions), development (3236) and test (3237) sets.

Sample questions from TimeQuestions:

Category Question
Explicit Who won Oscar for best actress 1986?
Which movie did Jaco Van Dormael direct in 2009?
What currency is used in Germany 2012?
Implicit Who was king of France during the ninth crusade?
What did Thomas Jefferson do before he was president?
What club did Cristiano Ronaldo play for after Manchester United?
Ordinal What was the first film Julie Andrews starred in?
What was the second position held by Pierre De Coubertin?
Who is Elizabeth Taylor’s last husband?
Temporal answer What year did Lakers win their first championship?
When was James Cagney’s spouse born?
When was the last time the Orioles won the world series?

Please refer to our paper for further details.

Contact

For more information, please contact: Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy or Gerhard Weikum.


To know more about our group, please visit https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/question-answering/.

Demo

Please select a category or multiple categories of temporal questions, and then choose a question from the drop-down list. You can also click to randomly choose a question from the list.

Random question


The demo shows five important subgraphs constructed by EXAQT and how the answers are generated step by step.

The graphs are visualized with Echarts. You can zoom in, zoom out or drag them. When you hover over a node, the properties of the node appear. Legends are adaptive to the data series of each graph. You can uncheck (or check) a legend to remove (or add) the data series of a graph.

Paper and Code

"Complex Temporal Question Answering on Knowledge Graphs", Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum, in Proceedings of the 30th ACM International Conference on Information and Knowledge Management 2021 (CIKM' 21), Virtual Event, 1 - 5 November 2021.

[Preprint] [Code] [Slides] [Poster] [Video]

Download TimeQuestions

Training Set (9708 Questions) Dev Set (3236 Questions) Test Set (3237 Questions) TimeQuestions is licensed under a Creative Commons Attribution 4.0 International License.

TimeQuestions Leaderboard

Method P@1 MRR Hit@5
TimeR4
Qian et al. '24

TwiRGCN
Sharma et al. '23

EXAQT
Jia et al. '21

SF-TQA
Ding et al. '22

FAITH
Jia et al. '24

LGQA
Liu et al. '23

EXPLAIGNN
Christmann et al. '23

CTRN
Jiao et al. '23

GRAFT-Net
Sun et al. '18

TempoQR
Mavromatis et al. '22

TMA
Liu et al. '23

UniK-QA
Oğuz et al. '22

CRONKGQA
Saxena et al. '21

UNIQORN
Pramanik et al. '21

GPT-4
Open AI. '23

InstructGPT
Ouyang et al. '22

PullNet
Sun et al. '19

0.781


0.605


0.565


0.539


0.535


0.529


0.525


0.465


0.452


0.438


0.436


0.424


0.395


0.331


0.306


0.224


0.105
  ---   


  ---   


0.599


  ---   


0.582


  ---   


0.587


  ---   


0.485


0.465


  ---   


0.453


0.423


0.409


  ---   


  ---   


0.136
  ---   


  ---   


0.664


  ---   


0.635


  ---   


0.673


  ---   


0.554


0.488


  ---   


0.486


0.450


0.538


  ---   


  ---   


0.186

TempQuestions

The old TempQuestions benchmark with 1271 questions from our group is now superseded by the newer TimeQuestions benchmark with 16181 questions, that can downloaded from this page. If you still want the older dataset, you can get it from here.

Feedback