Dataset Release Timeline

[iNeil77]

Hello to the authors!

I was reading the ProSec paper and was excited by the direct use of CWE descriptions to induce vulnerabilities in the LLM generations. I would like to ask about the authors' timelines for releasing the secure-vulnerable code pairs data and the synthesized instructions as mentioned in the paper.

Many Thanks

[XZ-X]

Thank you for your interest! We are currently working on an updated version of the dataset.

We plan to release the version by next week.

Please feel free to let us know if you have further questions!

[iNeil77]

Thanks for getting back. I eagerly await the data release!

[XZ-X]

Thank you again for your interest!

We released the first version of our vulnerability-inducing instruction dataset at Hugging Face🤗.

We plan to release the model-specific code pairs and the aligned models in the following one or two weeks.

Please definitely let us know if you have questions!

[iNeil77]

Thanks for getting back! I eagerly await the code pairs dataset.

[Robin-Pwner]

Hello to the authors!
Is there any update about the timelines for releasing the model-specific code pairs?
And could you provide the synthesized instructions first so that we could generate the model-specific code pairs ourselves?
Many Thanks

[XZ-X]

Hello to the authors! Is there any update about the timelines for releasing the model-specific code pairs? And could you provide the synthesized instructions first so that we could generate the model-specific code pairs ourselves? Many Thanks

Hello Robin, Thank you for your interest!

Sorry for the late response. We will release the model-specific code pairs today.

We already released the synthesized instructions at Hugging Face🤗. Please feel encouraged to give it a try!

Let us know if you have further questions!

[XZ-X]

We have updated the model-specific code pairs for phi3-mini-4k-inst and codellama-7b-inst .

Feel free to ping us if you have further questions!

[Robin-Pwner]

Thanks a lot!

[Robin-Pwner]

Hi, the authors.
I am working on reproducing Prosec with the released code pairs and I have a question about the test dataset.
In the paper, you select 38 ⟨language, CWE⟩s from PurpleLlama that are overlapped with SafeCoder as the test dataset. Could you provide a detailed list about the selected 38 ⟨language, CWE⟩s?
Many thanks.

[XZ-X]

Hello Robin,

Sorry for the late response. I put all the CWE list here

Each CWE is corresponded to a file.

Please note that after we constructed the list, we realized that SafeCoder does not targets the Rust and CSharp programing languages, so we did not include such data samples in the final dataset.

We will update the number of CWEs in the next revision.

Feel free to ping us if you have further questions!

[nmangaokar]

Hi, I was looking at the models released at https://huggingface.co/prosecalign/models?sort=downloads. Do you know which ones are the ones you used in the paper in Table 7? Also, is there code available for computing the benchmark numbers in Table 7?

[XZ-X]

Thank you for your interest. We will upload the models and publish the code within a week.

[XZ-X]

@nmangaokar Thank you for your patience. Our models are released on huggingface as follows:

Phi3-mini-Inst: prosecalign/phi3-mini-inst-step-1500
CodeLlama-7B-Inst: prosecalign/codellama-7b-inst-step-400
Qwen2.5-Coder-3B: prosecalign/qwen2.5-coder-3B-inst-step-400
Llama-3.2-1B: prosecalign/llama-3.2-1B-inst-step-400
Phi4-mini-Inst: prosecalign/phi4-mini-inst-step-1500

The security evaluation scripts are in https://github.com/PurCL/PurpleLlama
Please refer to the README.md for setups.

For the utility evaluation, we majorly reuse the scripts in MXEval and HumanEval.

Feel free to ping us if you encounter issues.

We will release the remaining data synthesis pipeline very soon.