Files Marked for Deletion under Linux

Working in the IT industry there has been times when may need to for one reason or another perform a compromise check on a server, some of these times I will notice some of the key files have been deleted. Another scenario which comes up abit is with MySQL files being deleted on accident and needed to retrieve them, below I'll go over one of the way you may be able to recover a file.

The way this particular technique works is do to a process having a lock on the file in question. The tool we will be using to track down these files is called lsof, lets go ahead and get down to it as it will begin to make better sense below.

• Alright lets go ahead and assume you believe you server may of been compromised, the first place you may want to look is your security log located at /var/log/secure.

  # tail /var/log/secure
  tail: cannot open `/var/log/secure' for reading: No such file or directory
  

  At first you may think the you made a type but further digging you see it is in fact gone.

  # ls -l /var/log/secure
  ls: /var/log/secure: No such file or directory
  

  OK, at this point you may think this file is gone but you may be wrong is the system still has as lock on the file (to be more accurate syslogd), so lets go ahead and move on.

• So now that we know the file is gone and we also know syslogd may of had a lock on the file we can go ahead and see if any files are marked for deletion.

  #  lsof | grep deleted
  syslogd   1649      root    1w      REG      253,0   200712     983176 /var/log/messages (deleted)
  syslogd   1649      root    2w      REG      253,0     5414     983177 /var/log/secure (deleted)
  

  To our surprise there are two files pending deletion, and one happens to be /var/log/secure!

• Alright so far we appear to be in good shape. The file is still actively being written to, so in theory we should be able to read and even recover it, but how?

  Well we can check the /proc file-system , this file-system is held completely in memory and handled by the kernel.

  Using our above output from lsof we see the process holding this file is syslogd at pid (Process ID) 1649, lets go ahead and have a look at /proc/1649.

  # ls -l
  total 0
  dr-xr-xr-x 2 root root 0 Mar  1 03:30 attr
  -r-------- 1 root root 0 Mar  1 03:30 auxv
  -r--r--r-- 1 root root 0 Mar  1 02:42 cmdline
  -rw-r--r-- 1 root root 0 Mar  1 03:30 coredump_filter
  -r--r--r-- 1 root root 0 Mar  1 03:30 cpuset
  lrwxrwxrwx 1 root root 0 Mar  1 02:58 cwd -> /
  -r-------- 1 root root 0 Mar  1 03:30 environ
  lrwxrwxrwx 1 root root 0 Mar  1 02:42 exe -> /sbin/syslogd
  dr-x------ 2 root root 0 Mar  1 02:58 fd
  -r--r--r-- 1 root root 0 Mar  1 03:30 io
  -r-------- 1 root root 0 Mar  1 03:30 limits
  -rw-r--r-- 1 root root 0 Mar  1 03:30 loginuid
  -r--r--r-- 1 root root 0 Mar  1 02:58 maps
  -rw------- 1 root root 0 Mar  1 03:30 mem
  -r--r--r-- 1 root root 0 Mar  1 03:30 mounts
  -r-------- 1 root root 0 Mar  1 03:30 mountstats
  -rw-r--r-- 1 root root 0 Mar  1 03:30 oom_adj
  -r--r--r-- 1 root root 0 Mar  1 03:30 oom_score
  lrwxrwxrwx 1 root root 0 Mar  1 02:58 root -> /
  -r--r--r-- 1 root root 0 Mar  1 03:30 schedstat
  -r-------- 1 root root 0 Mar  1 03:30 smaps
  -r--r--r-- 1 root root 0 Mar  1 02:42 stat
  -r--r--r-- 1 root root 0 Mar  1 03:30 statm
  -r--r--r-- 1 root root 0 Mar  1 03:30 status
  dr-xr-xr-x 3 root root 0 Mar  1 03:30 task
  -r--r--r-- 1 root root 0 Mar  1 03:30 wchan
  

  What we are really concerned about here is the fd directory, fd stands for "file descriptors", so lets pull it up!

  # ls -l
  total 0
  lrwx------ 1 root root 64 Mar  1 02:58 0 -> socket:[5799]
  l-wx------ 1 root root 64 Mar  1 02:58 1 -> /var/log/messages (deleted)
  l-wx------ 1 root root 64 Mar  1 02:58 2 -> /var/log/secure (deleted)
  l-wx------ 1 root root 64 Mar  1 02:58 3 -> /var/log/maillog
  l-wx------ 1 root root 64 Mar  1 02:58 4 -> /var/log/cron
  l-wx------ 1 root root 64 Mar  1 02:58 5 -> /var/log/spooler
  l-wx------ 1 root root 64 Mar  1 02:58 6 -> /var/log/boot.log
  

  OK, so here we see some files with numbers as names which have points (->) to what appears to be common *NIX names, so if all works as planed I should be able to read directly from this file. Below I'll do just that.

  # tail 2 -n 1
  Mar  1 02:56:32 SERVER sshd[2808]: pam_unix(sshd:session): session opened for user root by (uid=0)
  

  And what might we do with this newly found power?

  # cp 2 /var/log/secure && service syslod restart