- Install required package:
pip install -r requirements.txt - Download benchmark dataset from google drive to
datadir - Download auther generated outputs from google drive [Available Soon!] to
runtimedir - Update
api.pyto your own OpenAI config
|-- data # four code generation datasets
|-- runtime # runtime files including LLM generated results and inter-consistency measurements
|-- src
|-- pipeline.py # the entry point for LLM sampling & inter-consistency measurements. All results will be saved in `runtime`.
|-- graph.py # the entry point of MPSC
|-- evaluation.py, _evaluation.py # evaluation metrics
|-- execution.py, _execution.py # execution process for inter-consistency measurements
|-- api.py # OpenAI api
|-- exemplars # ICL exemplars for test case generation- Directly apply author provided LLM generated results for MPSC
python3 graph.py - MPSC from scratch (Warning: may cause a large number of OpenAI API calls)
python3 pipeline.py python3 graph.py
We also provide a code snippet of MPSC for other tasks in MPSC dir.