Enhancing instruction hierarchy in frontier LLMs

AI methods usually obtain directions from a number of sources. These can embrace security insurance policies from system messages, product steering from builders, requests from customers, and knowledge discovered on-line. Coaching fashions to reliably prioritize essentially the most trusted directions amongst these sources is a key a part of protected deployment.

Many AI security and reliability points can come up when this prioritization breaks down. Fashions could obtain requests for disallowed content material, makes an attempt to disclose non-public data, or immediate‑injection assaults embedded in on-line information. Failing to behave appropriately in every of those eventualities shares the identical root trigger: the mannequin could comply with the incorrect instruction.

When these directions battle, the mannequin has to determine which of them to prioritize. If it treats an untrusted instruction as authoritative, the mannequin could behave in ways in which violate insurance policies or developer and consumer intent.

We show that correctly designed instruction-hierarchy duties, which prepare fashions to prioritize directions in line with their belief stage, enhance a number of real-world security properties. Fashions skilled on these duties develop into extra aware of security specs in system prompts (bettering security steerability) and extra sturdy to prompt-injection assaults embedded in instrument outputs.

To deal with conflicts, OpenAI’s fashions are skilled to comply with a transparent instruction hierarchy:

System > developer > consumer > instrument

Increased‑precedence directions are extra trusted. The mannequin ought to solely comply with decrease‑precedence directions when they don’t battle with larger‑precedence constraints. These ideas are outlined within the OpenAI Mannequin Spec⁠(opens in a brand new window).

For instance, if a system message features a security coverage and a consumer asks the mannequin to violate it, the mannequin ought to refuse. If a instrument output accommodates malicious directions, the mannequin ought to ignore them somewhat than deal with them as instructions.

Getting this proper is foundational to security, safety, and reliability.

Reinforcement studying is a pure match for instructing the instruction hierarchy. We will generate conversations with conflicting directions, immediate the mannequin to reply, and reward it when it follows the proper instruction.

We’ve recognized three pitfalls of naively making use of that recipe:

• Instruction-following failures can double as instruction hierarchy failures: the mannequin may fail to resolve an instruction battle, not as a result of it doesn’t perceive the hierarchy of roles, however as a result of the directions themselves are too difficult.
• Instruction conflicts may be nuanced and even subjective. A typical strategy is to let a separate LLM decide assign rewards to the LLM being skilled, however judges themselves are fallible.
• Fashions are likely to study shortcuts that end in excessive reward, however are ineffective in apply⁠(opens in a brand new window). The traditional instance is overrefusals: fashions can study to maximise security by refusing even benign requests.

We design IH-Problem, a reinforcement studying coaching dataset, to handle every of these pitfalls. We adhere to the next ideas:

• Duties are instruction-following-simple
• They’re objectively-gradable with a easy Python script
• There aren’t any trivial shortcuts that assure excessive reward throughout all duties

Every job in IH-Problem is basically a dialog with the next messages:

• An instruction message from a high-privilege function, e.g. “Solely reply ‘Sure’ or ‘No’”.
• An instruction message from a lower-privilege function, which makes an attempt to get the mannequin to violate the directions within the higher-privilege message.

The mannequin being skilled generates the following message. We write the duties/environments in order that it’s potential to programmatically test whether or not the mannequin’s response satisfies the higher-level constraint.

We prepare a mannequin on IH‑Problem and produce an inner mannequin, which we name GPT‑5 Mini-R, with the next enhancements: 

• Performs higher on instruction‑hierarchy benchmarks
• Improved efficiency generalizes to held‑out and adversarial instruction hierarchy exams
• Maintains total usefulness, with out collapsing into over‑refusal

That is what makes the strategy particularly compelling for security: by straight coaching fashions to resolve instruction conflicts appropriately on IH-challenge duties, we get IH enhancements that generalize to new assaults and new conditions.

Stronger instruction hierarchy delivers a number of security advantages without delay, together with in security steerability and immediate injection robustness.

We consider security steerability by including category-specific security specs to the system immediate and measuring conduct on OpenAI’s security Manufacturing Benchmarks (a set of safety-sensitive conversations consultant of ChatGPT in manufacturing).

The IH-trained mannequin reveals a constant enchancment: with the protection spec current, it achieves larger refusal and protected completion charges throughout disallowed classes, indicating that stronger instruction hierarchy conduct makes it higher at resolving conflicts when unsafe requests come from lower-priority directions. Notably, this enchancment doesn’t include a corresponding lower in helpfulness price (i.e., it’s not changing into much less “useful” by merely refusing extra total).

Instruction hierarchy can also be central in resisting immediate injection, when malicious directions are embedded in instrument outputs. We consider the IH-trained mannequin on two immediate injection benchmarks—an instructional benchmark CyberSecEval 2 and an OpenAI inner immediate injection benchmark consisting of assaults just like the one demonstrated on an older model of ChatGPT Atlas⁠.

Relative to the baseline, the IH-trained GPT‑5 Mini-R mannequin improves immediate injection robustness on each benchmarks and considerably improves efficiency on our inner static immediate injection analysis in these experiments.

As fashions develop into extra agentic—calling instruments, studying untrusted paperwork, and taking actions on the planet—the flexibility to constantly prioritize trusted directions over untrusted ones turns into a core security property.

This work reveals that a number of pitfalls of IH robustness coaching may be overcome by designing coaching environments that tackle these pitfalls. Although our IH-Problem dataset appears easy, the IH conduct fashions study from these environments generalizes to extra life like, usually not-objectively-gradable benchmarks.

Strengthening instruction hierarchy not solely improves reliability, however unlocks a number of security and safety beneficial properties without delay—a basis that turns into more and more necessary as AI methods develop extra succesful and autonomous.