Dual-Adversarial Safety Alignment: Cultivating Intrinsic Threat Comprehension in LRMs

This is the official repository for the paper "Dual-Adversarial Safety Alignment: Cultivating Intrinsic Threat Comprehension in LRMs" by Hongli Shen, Shaopeng Fu, Qinbo Zhang, Jian Li and Di Wang.

Pipeline

Environment Setup

We recommend using a Conda environment with Python 3.10.

conda create -n dualadv python=3.10
conda activate dualadv
pip install -r requirements.txt

If you need API-based data generation, copy the environment template and fill in your keys:

cp .env.example .env
set -a && source .env && set +a

Set API endpoint base URLs through environment variables instead of editing scripts. Data-generation scripts use DEEPSEEK_BASE_URL and QWEN_BASE_URL; evaluation scripts use TARGET_BASE_URL and GUARD_BASE_URL.

Data Preparation

1. Generate adversarial jailbreak prompts

Run the agentic attack script to generate successful and failed jailbreak attempts:

python scripts/prepare_data/attack/agentic_generation.py

This script writes attack results to:

• data/after_attack/advsafe_successful_jailbreaks.jsonl
• data/after_attack/advsafe_failed_attempts.jsonl

2. Build training data

Use the successful jailbreak set to construct the main training data:

python scripts/prepare_data/process_advsafe.py

Use the failed-attempt set to augment the dataset with additional defensive examples:

python scripts/prepare_data/process_advsafe_failed.py

The resulting training files are stored in data/.

3. Train on other datasets

If you want to build training data from other safety datasets, run the corresponding processing scripts under scripts/prepare_data, for example:

• process_star1.py
• process_safechain.py
• process_thinksafe.py
• process_directrefusal.py
• process_safepath.py

You can also use the model-specific variants such as:

• process_advsafe_final_r1_7b.py
• process_advsafe_final_qwen3_235b.py

Training

We use LlamaFactory for LoRA supervised fine-tuning.

First clone and install LlamaFactory:

git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .
cd ..

The training configuration is provided in configs/train/train.yaml.
You can launch training with:

llamafactory-cli train configs/train/train.yaml

After training, merge the LoRA adapter into the base model according to configs/train/merge.yaml:

llamafactory-cli export configs/train/merge.yaml

Evaluation

Evaluation settings are summarized in configs/eval/eval.json.

The merged model used for evaluation is:

outputs/merge/deepseek_r1_distill_qwen_7b_advsafe/lora/sft_merged

1. Start vLLM services

For evaluation, serve the merged target model with vLLM.
For safety evaluation, also serve a Llama Guard model.

Example target server:

vllm serve outputs/merge/deepseek_r1_distill_qwen_7b_advsafe/lora/sft_merged \
  --served-model-name target \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code

Example Llama Guard server:

vllm serve meta-llama/Llama-Guard-3-8B \
  --served-model-name guard \
  --host 0.0.0.0 \
  --port 8001 \
  --trust-remote-code

When reproducing our setup, use the max-length settings specified in configs/eval/eval.json:

• safety evaluation target model: 4096
• Llama Guard: 8192
• gsm8k and mmlu_pro: 4096
• other utility evaluations: 32768

2. Run evaluation scripts

All evaluation scripts are located in evaluation:

• safety:
  • evaluation/advbench_llama_guard_eval.py
  • evaluation/harmbench_llama_guard_eval.py
  • evaluation/strongreject_llama_guard_eval.py
  • evaluation/wildjailbreak_llama_guard_eval.py
• utility:
  • evaluation/gsm8k_eval.py
  • evaluation/mmlu_pro_eval.py
  • evaluation/aime24_eval.py
  • evaluation/gpqa_diamond_eval.py
  • evaluation/math500_eval.py

Run them with Python after the corresponding vLLM service is ready, for example:

python evaluation/gsm8k_eval.py
python evaluation/mmlu_pro_eval.py
python evaluation/harmbench_llama_guard_eval.py