Description
Question answering over knowledge graphs (KG-QA) is a vital topic in IR. Questions with temporal intent are a special class of practical importance, but have not received much attention in research. We present EXAQT, the first end-to-end system for answering complex temporal questions that have multiple entities and predicates, and associated temporal conditions.
EXAQT: EXplainable Answering of Complex Questions with Temporal Intent
EXAQT answers natural language questions over KGs in two stages, one geared towards high recall, the other towards precision at top ranks. The first step computes question-relevant compact subgraphs within the KG, and judiciously enhances them with pertinent temporal facts, both using fine-tuned BERT models. The second step constructs relational graph convolutional networks (R-GCNs) from the first step’s output, and enhances the R-GCNs with time-aware entity embeddings and attention over temporal relations.
A simplified excerpt of the relevant zone in the Wikidata KG necessary for answering the question is shown in the figure.
with answer "Sidwell Friends School"
TimeQuestions
To evaluate EXAQT, we leverage recent community efforts in QA benchmark, and we search through eight KG-QA datasets for time-related questions. The result is a new compilation with about 16,181 questions called TimeQuestions.
We tag each question with its temporal category, link the answers to Wikidata and Wikipedia, and split the benchmark in a 60 : 20 : 20 ratio for creating the training (9708 questions), development (3236) and test (3237) sets.
Sample questions from TimeQuestions:
| Category | Question |
|---|---|
| Explicit | Who won Oscar for best actress 1986? |
| Which movie did Jaco Van Dormael direct in 2009? | |
| What currency is used in Germany 2012? | |
| Implicit | Who was king of France during the ninth crusade? |
| What did Thomas Jefferson do before he was president? | |
| What club did Cristiano Ronaldo play for after Manchester United? | |
| Ordinal | What was the first film Julie Andrews starred in? |
| What was the second position held by Pierre De Coubertin? | |
| Who is Elizabeth Taylor’s last husband? | |
| Temporal answer | What year did Lakers win their first championship? |
| When was James Cagney’s spouse born? | |
| When was the last time the Orioles won the world series? |
Please refer to our paper for further details.
Contact
For more information, please contact: Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy or Gerhard Weikum.
To know more about our group, please visit https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/question-answering/.
Demo
Please select a category or multiple categories of temporal questions, and then choose a question from the drop-down list. You can also click to randomly choose a question from the list.
The demo shows five important subgraphs constructed by EXAQT and how the answers are generated step by step.
The graphs are visualized with Echarts. You can zoom in, zoom out or drag them. When you hover over a node, the properties of the node appear. Legends are adaptive to the data series of each graph. You can uncheck (or check) a legend to remove (or add) the data series of a graph.
Paper and Code
"Complex Temporal Question Answering on Knowledge Graphs", Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum, in Proceedings of the 30th ACM International Conference on Information and Knowledge Management 2021 (CIKM' 21), Virtual Event, 1 - 5 November 2021.[Preprint] [Code] [Slides] [Poster] [Video]
Download TimeQuestions
Training Set (9708 Questions) Dev Set (3236 Questions) Test Set (3237 Questions) TimeQuestions is licensed under a Creative Commons Attribution 4.0 International License.TimeQuestions Leaderboard
| Method | P@1 | MRR | Hit@5 |
|---|---|---|---|
|
TwiRGCN Sharma et al. '23 EXAQT Jia et al. '21 SF-TQA Ding et al. '22 FAITH Jia et al. '24 LGQA Liu et al. '23 EXPLAIGNN Christmann et al. '23 CTRN Jiao et al. '23 GRAFT-Net Sun et al. '18 TempoQR Mavromatis et al. '22 TMA Liu et al. '23 UniK-QA Oğuz et al. '22 CRONKGQA Saxena et al. '21 UNIQORN Pramanik et al. '21 GPT-4 Open AI. '23 InstructGPT Ouyang et al. '22 PullNet Sun et al. '19 |
0.605 0.565 0.539 0.535 0.529 0.525 0.465 0.452 0.438 0.436 0.424 0.395 0.331 0.306 0.224 0.105 |
--- 0.599 --- 0.582 --- 0.587 --- 0.485 0.465 --- 0.453 0.423 0.409 --- --- 0.136 |
--- 0.664 --- 0.635 --- 0.673 --- 0.554 0.488 --- 0.486 0.450 0.538 --- --- 0.186 |
TempQuestions
The old TempQuestions benchmark with 1271 questions from our group is now superseded by the newer TimeQuestions benchmark with 16181 questions, that can downloaded from this page. If you still want the older dataset, you can get it from here.