safety

NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.

The Destructive Command Guard (dcg) is for blocking dangerous git and shell commands from being executed by agents.

A reading list for large models safety, security, and privacy (including Awesome LLM Security, Safety, etc.).

Safe RLHF: Constrained Value Alignment via Safe Reinforcement Learning from Human Feedback

Awesome-Jailbreak-on-LLMs is a collection of state-of-the-art, novel, exciting jailbreak methods on LLMs. It contains papers, codes, datasets, evaluations, and analyses.

Chinese safety prompts for evaluating and improving the safety of LLMs. 中文安全prompts，用于评估和提升大模型的安全性。

Awesome-LLM-Robustness: a curated list of Uncertainty, Reliability and Robustness in Large Language Models

AutoHarness: Automated Harness Engineering for AI Agents

A survey on harmful fine-tuning attack for large language model (ACM CSUR)

[ICLR 2025] Dissecting adversarial robustness of multimodal language model agents

This is the official code for the paper "Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation"

This is the official code for the paper "Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbation" (ICLR2025 Oral).

[WSDM 2026] LookAhead Tuning: Safer Language Models via Partial Answer Previews

This is the unofficial re-implementation of "Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Attack" (ICML2025)