9 Protection and Operating-System Design

Protection and security

An operating system must do more than provide processes, memory, files, and devices: it must control how those resources can be used. Protection mechanisms restrict access to system resources. Security is the broader goal of preventing unauthorized access, misuse, or damage; protection mechanisms help enforce security policies, but do not guarantee security if they are incorrectly designed or implemented.

Subjects, objects, and access decisions

A protection system can be understood through three elements: Subjects, Objects, and Rights. Subjects are active entities that request operations, such as users, processes, or services. Objects are resources being accessed, such as files, memory, devices, or other processes. Rights specify which operations a subject may perform on an object, including reading, writing, executing, or deleting.

For example, a process might be permitted to read a configuration file but not modify it. The operating system associates a request with a subject, checks that subject’s rights, and enforces the result at a protected boundary.

Authentication, Authorization, and Access enforcement are distinct steps. Authentication establishes or verifies an identity. Authorization determines what that identity—or a process acting on its behalf—is permitted to do. Access enforcement ensures that requests cannot bypass the decision. Knowing who a user is does not, by itself, establish that the user may access a particular file.

Least privilege and secure defaults

The principle of least privilege is to grant each user, process, or system component only the permissions and resources needed for its task. If a component is compromised or malfunctions, limiting its authority can reduce the damage.

Secure defaults are also important: when a permission has not been explicitly granted, access should generally be denied. These principles guide access decisions across operating-system resources.

Isolation across operating-system resources

Isolation limits how much one activity can affect another. It protects against malicious behavior and ordinary bugs, such as a process accidentally overwriting another process’s data. Operating systems implement isolation using hardware features, kernel checks, and carefully controlled sharing.

Processes and memory

Each process typically receives its own virtual address space. Memory-management hardware and the operating system translate its addresses and prevent it from directly reading or writing another process’s memory or kernel memory. A process may still request controlled sharing, for example through shared memory, but the operating system must deliberately establish that access.

System calls provide another controlled boundary. A program requests a service, and the kernel checks and performs the privileged operation rather than allowing the program unrestricted access to hardware.

Files and devices

File permissions and other access-control mechanisms regulate operations on files. A file system can allow one user to read a file while denying writes or access to other users. Device access is also mediated: processes request operations through operating-system interfaces instead of directly controlling hardware. These protections depend on the operating system enforcing permissions consistently across relevant access paths.

CPU time and concurrency

Scheduling gives processes and threads opportunities to use the CPU, while isolation and permission checks limit what they can do during that time. Scheduling also supports availability: a system should avoid letting one activity monopolize resources indefinitely.

With concurrency, multiple threads or processes may share data or other resources. Synchronization mechanisms such as locks help prevent conflicting operations, but shared resources must also be protected from unauthorized access. Isolation does not forbid all sharing; it means that sharing is deliberate and controlled.

Integrating protection into operating-system design

Operating-system components depend on one another. The scheduler chooses which thread runs; virtual memory determines which addresses it can use; system calls let it request kernel services; and the file system stores persistent data under access rules. A failure at one boundary can undermine others. For example, unchecked input to a kernel service might corrupt kernel memory, potentially affecting every process on the machine.

A sound design combines several goals:

• Define clear boundaries. Separate user programs from the kernel and restrict access between processes.

• Mediate access. Check permissions when subjects request operations on protected objects.

• Minimize authority. Give users and components only the privileges they need.

• Control sharing. Make shared memory, files, devices, and communication channels explicit and protected.

• Limit complexity and exposure. Keep privileged interfaces and components as small and understandable as practical; fewer reachable operations can mean fewer opportunities for misuse.

• Balance protection with usefulness and cost. Stronger isolation can require additional checks or restrict convenient sharing, so design must account for performance and legitimate system needs.

No single mechanism is enough. Memory isolation cannot prevent a user from being granted inappropriate file permissions, and file permissions cannot prevent a faulty kernel component from corrupting memory. Security depends on layers working together, with clear responsibilities and consistent enforcement.