Operating Systems Security

Chapter Seven

Prepared by: Raval, Fichadia

Raval • FichadiaRaval • FichadiaJohn Wiley & Sons, Inc. 2007

2

Chapter Seven Objectives Learn the basic concepts of operating systems (OS)

and associated terminology.

Understand the risks that impact operating systems and the controls to mitigate them.

Gain the skills to assess the security posture of an operating system and make management recommendations.

Apply security principles and best practices to an operating system.

3

The Big PictureElements of an operating

system environment.

Some risks that impact the operating system.

4

Operating system primer

Operating system: the system software on a machine upon everything else depends.

Most critical software relied upon by almost every other application on a machine.

Key responsibilities of an OS include: CPU/process management

Memory management

Hardware management

Disk and file system management

Examples of popular operating systems include Windows, Unix, Linux, zOS, MacOS.

5

Operating system primer

Operating system: the system software on a machine upon everything else depends.

“Orange book” describes the criteria for security ratings on an operating system.

Division D (Minimum Security) Class D Systems that aren’t rated higher

Division C (Discretionary Protection) Class C1 Discretionary security protection (C1 is no longer used to rate OSs) Class C2 Controlled access protection (rated versions include versions of OpenVMS,

Windows, AS/400, RS/6000) Division B (Mandatory Protection)

Class B1 Labeled security protection (rated versions include DEC, HP-UX, IRIX) Class B2 Structured protection (rated versions include XENIX 4.0) Class B3 Security domains (rated versions include XTS-200)

Division A (Verified Protection) Class A1 Verified design (rated versions include Boeing MLS)

6

Management concerns

Concerns about operating system security typically include the following:

Minimizing the company’s risk exposure in the most cost-effective manner.

Defining the optimal security posture across a heterogeneous environment.

Keeping up with existing and upcoming security threats against operating systems and implementing mitigating controls.

Having an effective backup, recovery, business resumption and a disaster recovery plan.

7

Risks and controls

Authentication: Process of validating user identity Most operating systems require a user ID and a

password combination for authentication. Some allow for third-party based authentication. User IDs are used by regular end users, administrators,

and applications to claim their identity. Knowledge of the password proves the user’s identity. Operating systems typically never store passwords,

rather they store the password hash - a unique fixed length output based on the password and/or a “salt”.

8

Risks and controls

Authentication: Process of validating user identity

Wrong password generate wrong hashes and user is denied access. Right password generates right hash and allows user access.

9

Risks and controlsAuthentication risks: Poor password selected by end users. Dictionary and/or brute-force attacks against password

hashes.

Controls: Password management routines such as requiring

periodic password changes, quality passwords, longer passwords, etc.

Enabling account lockouts after a given number of wrong password attempts.

Using two-factor authentication techniques: something you have and something you know.

10

Risks and controls

Authorization: Process of determining user privileges

Most operating systems determine user ID’s privileges after user’s authentication.

Protected resources have an Access Control List (ACL) that defines who has what level of access.

User IDs are assigned privileges to protected resources either directly, or by them being part of a group that has access.

Some operating system like Unix/Linux allow users to exceed their privileges via features like setuid/setgid.

11

Risks and controlsAuthorization risks: Poorly defined ACLs. Attacks that let users again unauthorized privileges

(setuid/setgid exploits, exploiting services/applications with higher privileges).

Controls: Periodic review of ACLs and privileges granted to users

and groups. Protecting setuid/setgid related files from exploited by

implementing stronger file permissions. Disabling unnecessary privileged services / uninstalling

unnecessary privileged applications.

12

Risks and controls

Trust relationships: Relying on other OS for validating a user identity.

Many operating systems allow access to their resources without verifying a user’s credentials if they trust other OS of doing so.

Trust simplifies and adds convenience for user and application by not having to reauthenticate.

Without trust relationships credentials may have to be stored in multiple places.

Trust can be unidirectional or bidirectional.

13

Risks and controls

Trust risks:

Spoofing attacks wherein users forge their identities.

Transitive trust that let users go from AB and from BC, however AC wasn’t intended.

Controls:

Periodic review of trust relationships and their business need. Disable the ones that aren’t needed.

Implement patches and controls to prevent spoofing attacks.

14

Risks and controls

Job scheduling: Allows programs and scripts to run without user interaction at scheduled times.

Most operating systems allow definition of jobs (programs, scripts, batch files) to be run periodically at scheduled hours with different level of privileges.

Schedulers can offload backups during off-peak hours.

Jobs can be scheduled at different hours, or upon different triggers such as system startup, shutdown, results of previous job etc.

15

Risks and controls

Job scheduling risks:

Piggybacking attacks wherein malicious code is inserted into unprotected scripts (or scripts referenced by the job) that are scheduled as privileged jobs.

Program substitution wherein a malicious script is executed instead of the original due to lax permissions on directories or because of lack of full path names.

Controls:

Periodic review of all jobs and their privileges. Disable the ones that aren’t needed and/or minimize privileges.

Implement strong file and directory permissions.

16

Risks and controls

File systems: OS feature to manage files stored on the hard drive(s).

File systems allow for tracking of the location of files and their directories, their classification, their attributes, their ACLs, and their security.

File systems can be for local files (files stored on locally on the computer) or for remote files (files shared with remote computers).

OSs can support multiple types of file systems such as NTFS, FAT, EFS (local), NFS, CIFS (remote) etc.

Local and remote file systems are protected via ACLs that define who has what type of access to the file.

17

Risks and controls

File systems: OS feature to manage files stored on the hard drive(s).

Most file systems classify files in an inverted tree structure.

18

Risks and controlsFile system risks: Default ACLs or user-defined ACLs on local and shared

files could be poor. Critical files containing sensitive/confidential data could

be poor secured.

Controls: Periodic review ACLs for sensitive files and directories. Use encryption on sensitive files and directories. Baseline the file system to detect unauthorized

changes. Periodic review ACLs for shared files.

19

Risks and controls

Software updates: Updating OS with vendor provided fixes.

Operating systems have defects that are identified either by end users or by vendors.

Vendors release fixes – patches – to fix/mitigate the defect.

Hotfixes are patches that have limited regression testing done and typically address a small number of issues.

Service packs are collection of patches that are regression tested and address several issues.

20

Risks and controls

Software update risks:

Inadequate testing of patches by vendor or end-user could break existing functionality.

Zero-day exploits could hit OS before patches are applied.

Controls:

Periodic review threats and their fixes.

Maintain inventory of OS and their patch level.

Have a test bed ready to test the impact of patches on existing functionality.

Use automated tools to deploy patches at the earliest.

21

Assurance considerationsAn audit to assess operating system security should include

the following:

Evaluate the need for each service on the OS. Unnecessary services should be turned off.

Assess the patch deployment process. All necessary patches should be implemented.

Review authentication methods, user accounts and their privileges. Ensure quality user ID and password management techniques are used.

Determine the legitimacy of trust relationships.

Ensure core operating system files are baselined and protected.

22

Assurance considerations Evaluate the need and security of all file shares.

Determine if additional controls like anti-virus, personal firewalls, and intrusion prevention systems are in place.

Review all scheduled jobs and their controls.

Ensure that functional plans for backup and recovery, business resumption, disaster recovery are in place.

23

Recap