Task I - Identifying a memory DOS and responding In this task, you will start a memory denial of service against yourself and then add swap on the fly to attempt to buy more time. If you were the client from the module 4 exercise, you will have to create the memory DOS script (step 16) from the module 4 lab. 1) Open 3 seperate terminal windows. In the first terminal, start a vmstat with an interval of one second. [1] # vmstat 1 procs memory page disk faults cpu r b w swap free re mf pi po fr de sr dd f0 s5 s1 in sy cs us sy id 0 0 25 811888 326896 3 11 8 5 5 0 5 2 0 0 0 311 564 89 2 1 97 0 0 25 781816 264256 13 8 680 0 0 0 0 119 0 0 0 634 6097 360 1 21 78 0 1 25 781816 263288 20 0 896 0 0 0 0 135 0 0 0 742 3591 421 2 12 86 2) In the second terminal, invoke the "hog" script in the /export/home/guest directory. [2] # cd /export/home/guest [2] # ./hog 3) In the third terminal window, create an 128mb swap file and add it on the fly. [3] # mkfile 128m /export/swapfile [3] # swap -a /export/swapfile 4) Observer the vmstat output in terminal 1. Did the "swap" column grow? procs memory page disk faults cpu r b w swap free re mf pi po fr de sr dd f0 s5 s1 in sy cs us sy id 1 1 25 1560 8584 612 6607 624 8 8 200 0 69 0 0 0 514 7979 1086 27 72 1 1 1 25 1280 7680 534 2471 4536 3656 3744 40 439 145 0 0 0 841 3537 671 14 40 0 1 25 127608 8232 415 513 4640 5648 5688 0 202 100 0 0 0 546 615 245 1 11 88 0 3 25 123104 8464 449 558 5008 5080 5080 0 0 113 0 0 0 557 974 322 2 13 85 5) Observe the hog output in terminal 2. It appears that the script continued to run even though /tmp was full. As soon as the swap space was added, the script started writing files into the /tmp space. cat: write error: No space left on device <---script still running even though /tmp is full + let x=x+1 + [ 1805 -eq 100000 ] + cat /var/sadm/install/contents + 1>> /tmp/file.1805 cat: write error: No space left on device <---script still running even though /tmp is full + let x=x+1 + [ 1806 -eq 100000 ] + cat /var/sadm/install/contents <---script appears to resume writing to /tmp + 1>> /tmp/file.1806 + let x=x+1 6) Stop the script by issuing a ^C in terminal 2. Clean up the /tmp directory. [3] # cd /tmp [3] # rm -r /tmp/file* 7) Stop the vmstat by issuing a ^C in terminal 1. Task II - Limiting the size of /tmp in the /etc/vfstab In this exercise, you will limit the size of the /tmp filesystem and then run a memory DOS against yourself to see if the filesystem limit worked. 1) Observer the size of the /tmp file system with the df command. # df -k /tmp Filesystem kbytes used avail capacity Mounted on swap 900360 16 900344 1% /tmp 2) Edit the /etc/vfstab and configure /tmp to have a maximum size of 128m # vi /etc/vfstab swap - /tmp tmpfs - yes size=128m 3) Since /tmp cannot unmount. You will have to reboot the workstation # init 6 4) After the workstation has rebooted, check the size of /tmp again. Notice that it is much smaller than the previous size. # df -k /tmp Filesystem kbytes used avail capacity Mounted on swap 131072 344 130728 1% /tmp 5) Open 3 terminal windows. In the first window, start a vmstat at 1 second intervals. [1] # vmstat 1 procs memory page disk faults cpu r b w swap free re mf pi po fr de sr dd f0 s5 s1 in sy cs us sy id 0 0 0 856456 426696 29 122 134 0 0 0 0 19 0 0 0 363 662 157 3 8 89 0 0 0 878352 414024 0 8 0 0 0 0 0 0 0 0 0 356 171 91 0 0 100 6) In a second terminal window, start the hog script. [2] # cd /export/home/guest [2] # ./hog 7) Notice in terminal 2 that the hog script will error out more quickly with a "No space left on device" while in terminal 1, the vmstat reports plenty of virtual memory in the "swap" column. Since the /tmp file system has been limited, the workstation is now protected against a /tmp DOS. However, the /tmp fiilesystem is still full. Any applications that need to write to the /tmp space will be unable to do so. Task III - Identifying a CPU DOS and responding to it. The following task teaches how to detect a CPU DOS and prevent future CPU DOS. The task requires you to fork bomb your own system. This may cause the system to stop responding. Be sure to save all of your work. 1) Open 4 terminal windows. In the first terminal window, use the sar command to monitor the process table size. Have sar monitor every second for 1000 seconds. Notice the proc-sz value. [1] # sar -v 1 1000 19:40:20 proc-sz ov inod-sz ov file-sz ov lock-sz 19:40:21 63/7914 0 1955/33952 0 392/392 0 0/0 19:40:22 63/7914 0 1955/33952 0 392/392 0 0/0 19:40:23 63/7914 0 1955/33952 0 392/392 0 0/0 19:40:24 63/7914 0 1955/33952 0 392/392 0 0/0 2) In the second terminal window, use the vmstat command to monitor the run queue (r) field. [2] # vmstat 1 procs memory page disk faults cpu r b w swap free re mf pi po fr de sr dd f0 s5 s1 in sy cs us sy id 0 0 0 800712 352024 26 102 40 0 0 0 0 5 0 0 0 334 432 124 2 3 95 0 0 0 875544 420648 0 8 0 0 0 0 0 0 0 0 0 431 317 120 1 0 99 3) In the third terminal window, login via telnet to the localhost as the user guest. [3] # telnet localhost Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. SunOS 5.8 login: guest Password: Last login: Tue Jul 30 15:58:55 from localhost Sun Microsystems Inc. SunOS 5.8 Generic Patch October 2001 $ 4) As user guest, create a fork bomb by editing two scripts called "a" and "b". These scripts will do nothing but call each other an execute sleep processes. They will continue in an infinite loop until the process table fills to capacity. [3] $ vi a ./b & sleep 20 & [3] $ vi b ./a & sleep 20 & 5) Make the scripts executable. [3] $ chmod 555 a b 6) Execute the scripts. As soon as these scripts execute, look immediatley to terminal windows 1 and 2 and notice the drastic change. [3] $ ./a & 7) If the system is still responsive, monitor the vmstat and sar output. Also, in terminal window 4, issue a ps -ef command. [4] # ps -ef <> guest 9723 9722 0 19:51:18 ?? 0:00 -sh guest 9767 9766 0 19:51:18 ?? 0:00 -sh guest 8427 1 0 0:00 guest 9709 9708 0 19:51:18 ?? 0:00 -sh guest 9800 9774 0 19:51:18 ?? 0:00 -sh 8) As the system administrator, stop the CPU DOS by killing all of the user guest's processes. [4] # pkill -u guest 9) Search the user guest's home directory for any files created in the last day. [4] # find /export/home/guest -mtime -1 /export/home/guest /export/home/guest/a /export/home/guest/b Task IV - Preventing CPU DOS The purpose of this task is to configure the /etc/system file on the server to limit the ammount of processes a user can take. 1) As root on the server, open up the kernel using the mdb command in read mode. The adb utility is used for core dump analysis and information gathering on a live kernel. All of the features of mdb are covered in "ST-350 - System Fault Analysis". The following mdb command will display the current value for the maximun amoumt of user proceeses allowed on the server. # mdb -k Loading modules: [ unix krtld genunix ip ufs_log nfs isp ipc random ptm logindmux ] >maxuprc/D <-----Ask the kernel how many proccess a user can own. maxuprc: maxuprc: 7909 <-----The kernel reports that a user can own 7909 process table slots >max_nprocs/D <-----Ask the kernel what the total process table size can be. max_nprocs: max_nprocs: 7914 <-----The kernel reports that the maximum table size is 7914. This means that a user can reserve almost the entire process table $q <----- exit adb 2) Since a regular user can consume an entire process table. Set a kernel tuning parameter in the /etc/system file to limit the maximum user processes. # vi /etc/system <> set maxuprc=100 3) Reboot the workstation. # init 6 4) After the workstation has rebooted, verify with mdb that the kernel tuning setting worked. # mdb -k Loading modules: [ unix krtld genunix ip nfs ipc ptm logindmux ] > maxuprc/D maxuprc: maxuprc: 100 > 5) Open three terminal windows. In the first terminal window, use the sar command to monitor the process table size. [1] # sar -v 1 1000 SunOS gabriel 5.8 Generic_108528-13 sun4u 08/01/02 18:48:57 proc-sz ov inod-sz ov file-sz ov lock-sz 18:48:58 42/7914 0 1430/33952 0 264/264 0 0/0 6) In the second terminal window, use the su command to assume the identity of guest. Run the fork bomb. [2] # su - guest [2] $ id uid=1001(guest) gid=10(staff) [2] $ ./a 7) Observe the output in terminal window #1. Did the process table continue to grow or did it level off? sar -v 1 1000 SunOS gabriel 5.8 Generic_108528-13 sun4u 08/01/02 18:48:57 proc-sz ov inod-sz ov file-sz ov lock-sz 18:48:58 42/7914 0 1430/33952 0 264/264 0 0/0 18:48:59 42/7914 0 1430/33952 0 264/264 0 0/0 18:49:00 42/7914 0 1430/33952 0 264/264 0 0/0 18:49:01 141/7914 0 1430/33952 0 461/461 0 0/0 18:49:02 141/7914 0 1430/33952 0 461/461 0 0/0 18:49:03 141/7914 0 1430/33952 0 461/461 0 0/0 8) The "guest" user was limited to 100 processes by tuning the kernel.