2024-09-20: 🍻 Accepted to EMNLP 2024 Findings!
🗒️ arXiv link: https://arxiv.org/abs/2406.08101

Conversational Explanation Query Language (CoXQL): A text-to-SQL-like benchmark.

We recommand to use Python 3.8+

python -m pip install --upgrade pip
pip install -r requirements.txt

dataset folder's structure is listed below:

dataset
    |- data
    |- filters
    |- global_prediction
    |- includes
    |- local_explanation
    |- local_predicrion
    |- meta
    |- modification
    |- perturbation

In dataset folder, you can find the dataset in json format: coxql_train.json and coxql_test.json. More details about the number of pairs in each operation category can be found in dataset/README.md.

Both json files have the same structure as follows:

{
  "idx": ...,
  "text": ...,
  "sql": ...
}

Parsing accuracy results can be found in parsing/guided_decoding/results, parsing/multi_prompt/results and parsing/multi_prompt_plus/results

You can run calculate_parsing_accuracy.py to get an overview of parsing accuracy.

python calcualte_parsing_accuracy.py {guided_decoding, multi_prompt, multi_prompt_plus}

🤗We evaluate in total seven state-of-the-art LMs:

@misc{wang2024coxqldatasetparsingexplanation,
      title={CoXQL: A Dataset for Parsing Explanation Requests in Conversational XAI Systems}, 
      author={Qianli Wang and Tatiana Anikina and Nils Feldhus and Simon Ostermann and Sebastian Möller},
      year={2024},
      eprint={2406.08101},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2406.08101}, 
}