Unix Interview Questions: purging older files

Unix for software developers

1.Top 17 Unix commands Java developers use frequently		2.History commands	3.Shell scripting	4. Reading from a file
5.Purging older files	6.Splitting and archiving files	7.Emulator, SSH client	8.Unix commands for developers	9.Unix basic interview Q&A

In many applications, feed files and log files are generated and it is imperative to archive and purge these file periodically based on the retention requirements defined in the non functional requirements.



Q. Can you write a script to purge files that a older than say x days from a given folder?
A.

Firstly, you can define a purging configuration file as shown below to define the folders that needs to be purged. The purging can be done either recursively (i.e. including sub folders) and non recursively. Here is a sa,ple file that defines the folder to purge, records older than x days, and a flag to indicate recursive or not (1 means recursive and 0 means non-recursive)

/env/uat/out/myapp/archive +30 0
/env/uat/myapp/process     +180      1
/env/uat/myapp2/archive     +180     1

#!/bin/sh
#  CFG=$1    config script

CFG=$1

echo "Reading config file..... "$CFG

while read DIRPATH WAITTIME RECURSIVE
do
     echo  `date +"%I:%M:%S %p"` " purging....." $DIRPATH  $WAITTIME  $RECURSIVE
     if [ $RECURSIVE -eq 1 ]; then
        /usr/bin/find $DIRPATH* -mtime $WAITTIME -exec rm -r {} \;
     else
        /usr/bin/find  $DIRPATH* -prune -mtime $WAITTIME -exec rm -r {} \;
     fi
done < $CFG

exit 0

The -prune option is a bit tricky and it is an action like -print command. In the above example, -prune action stops descending into the directories even though the remove command (i.e. rm) is used with the -r option for recursion. For example

$find src/test* -prune

src/test
src/test/.svn
src/test/resources
src/test/resources/.svn
src/test/resources/jms
src/test/resources/jms/.svn

Now with the -prune option

$find . src/test -prune

src/test

as you can see, the sub directories are pruned.

find [path] [conditions to prune] -prune -o [your usual conditions] [actions to perform]

For example:

$find  . -name . -o  -type d
.
./archive
./archive/20130426
./archive/20130426/20130426
./archive/zip20130521

$find  . -name . -o  -type d -prune

.
./archive

As you can see, the sub directories are not included.


Q. How will you move files that are older than 7 days to  their own archive folders to be zipped later on?
A.

#!/bin/sh -x

# DIRPATH=$1    --  path to source files
# ARCPATH=$2    --  path to move archived files 
# WAITTIME=$3   --  days to archive e.g +7 files (older than 7 days)

DIRPATH=$1  
ARCPATH=$2  
WAITTIME=$3 

if [ ! -d $ARCPATH ]; then
    mkdir $ARCPATH
fi

cd $DIRPATH


FILES=`/usr/bin/find . ! -name . -prune -type f -mtime $WAITTIME`


for file in $FILES
do
 #for each file find the creating date so that it can be archived under that directory
 #returns something like 2013Apr26
 DATE=` ls -e  $file| awk '{print $11"-"$8"-"$9}' | sed 's/-//g'` 

 if [ ! -d $ARCPATH/$DATE ]; then
    mkdir $ARCPATH/$DATE
 fi

 echo "Moving " $file " to " $ARCPATH/$DATE
 mv $file $ARCPATH/$DATE 
done;

exit 0

you run it like

sh archive.sh /etc/myapp /etc/archive +7

The files will be archived under /etc/archive/{date}