Large Language Model Alignment Techniques

Abstract

Large language model alignment -- the challenge of ensuring that AI systems behave in accordance with human values, intentions, and safety requirements -- has moved from a theoretical concern to an urgent engineering problem as LLMs are deployed in consequential applications. This study systematically evaluates six alignment techniques across four alignment dimensions: instruction following, harmlessness, honesty and calibration, and robustness to adversarial prompting. Techniques evaluated include Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimisation (DPO), Constitutional AI (CAI), RLAIF (RL from AI Feedback), Supervised Fine-Tuning on curated data (SFT-Curated), and a proposed Hybrid Alignment Framework (HAF) combining DPO with constitutional self-critique. Base models evaluated: LLaMA-2-13B, Mistral-7B, and LLaMA-2-70B. Evaluation benchmarks include MT-Bench, AlpacaEval 2.0, TruthfulQA, BBQ (bias), AdvBench (adversarial robustness), and a purpose-built EU AI Act Compliance Benchmark (EAACB, n=840 scenarios covering prohibited and high-risk AI behaviours). RLHF achieves the highest MT-Bench score (8.24) on LLaMA-2-70B. DPO achieves comparable instruction following (MT-Bench 8.18) at 2.4x lower training cost. HAF achieves the best harmlessness-helpfulness balance (EAACB: 94.2% compliance, MT-Bench: 7.86). Constitutional AI reduces harmful output rates by 82.4% vs. unaligned baseline. All techniques show adversarial vulnerability: AdvBench attack success rate averages 18.4-34.2% across methods. A practitioner alignment recipe selection framework is proposed.