# http://www.jpsdomain.org/linux/OnStream_DI-30-RedHat_Backup_mini-HOWTO.html # v0.9 27-Feb-2001 JP Vossen (jp@jpsdomain.org) # v0.9.1 02-Jun-2001 JPV Added code to delete TapeName.log to avoid appending # Removed the "-@ ${EMAIL}" by default (see Note 3 below). # Added data on disk size to Finished backup log message, just in case. # Updated for RPMs and minor typos # v0.9.2 16-Jun-2001 JPV bugfixes in tape, corrections to OnStream docs. # v0.9.3 15-Nov-2001 JPV Change calcsum to zsh #MYVER="v0.9.4a 2002-03-23" # JPV Removed -Q stuff for testing #MYVER="v0.9.5 2002-04-01" # JPV Removed -b 32k -c 1024 -M 10m stuff for testing # SERIOUS BUGFIX: SIZEONTAPE was REALLY broken! # Could probably do the whole thing in gawk and get rid of calcsu,.zsh # MYVER="v0.9.6 2002-04-08" # JPV Add '-Z -E ${NOCOMPRESS}' back into verify # MYVER="v0.9.6 2002-06-18" # JPV Make BUMONTHLYJOB same as BUWEEKLYJOB # MYVER="v0.9.7 2003-02-10" # JPV Add ntpd restart # MYVER="v0.9.8 2003-04-26" # JPV Add /var/www/ and /opt MYVER="v0.9.9 2003-05-21" # JPV Add /usr/src/redhat # Should probably use -s {size_of_tape} but since this runs in the # background and tries to do a verify, prompting for the next tape is a # bit problematic... # Copyright 2001-2002 JP Vossen # This script is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # In no event shall the author be liable for any damages whatsoever # (including, without limitation, damages for loss of business profits, # business interruption, loss of business information, or any other # pecuniary loss) arising out of the use of or inability to use this script. # http://www.jpsdomain.org/linux/OnStream_DI-30-RedHat_Backup_mini-HOWTO.html # REQUIRES: # afio, developed and tested with afio-2.4.6, which has a minor bug that prints # "-- compressed" twice during verify operations. That bug is fixed in my RPMs # (http://www.jpsdomain.org/public/rpms). # See http://www.linux.org/apps/AppId_266.html # and http://metalab.unc.edu/pub/linux/system/backup/afio-2.4.6.tgz (unpactched) # calcsum.zsh requires zsh to be installed on the system. This is provided # by the RedHat "zsh" package under RedHat 7.1. # RDISK optionally requires mkbootdisk, fdisk and mkkickstart. Lack of these # programs will not affect backups, but then RDISK will not be able to gather # some of the information it tries to gather before a backup. fdisk is provided # by the util-linux package, the other two are provided by packages of the same # name as the tool. # NOTES [see Other Notes below]: # 1. This script assumes (actually forces) only one archive per tape. # 2. See the nobackup file to see stuff you should probably never backup. # NEVER backup /proc -- it's utterly useless, and could crash your tape # software! # 3. Remove the "-@ ${EMAIL}" lines when you get tired of getting boring # e-mail about it. # 4. Uses my RDISK script without using a floppy (see RDISK for details) # 5. Calculates size of data on disk v. size of data on tape. This way you # can see how your compression options are working, and how you are on # tape space. If you have space, you can lower compression to speed up # the backup. The totals are not 100% accurate, as the "ls -lR" method # includes the sizes of the files in which sub-directory information is # stored, and the "afio -r" method does not. Still, they are close # enough. (SizeOnDisk should ALWAYS be bigger) I do this instead of # using du because that includes slack space (allocated disk use) # rather than file size, which is a LOT less meaningful in this context. # 6. Your verify will have errors such as the following. That's because some # files CHANGED between when they got backed up and when they got verified. # This is normal! # afio: "var/log/backup/backup.log": Archive data and file cannot be aligned (disk 1) at Wed Feb 21 16:07:56 2001 # afio: "var/log/backup/backup.log": Corrupt archive data (disk 1) at Wed Feb 21 16:07:56 2001 # 7. This script complies with FSH-2.1 (http://www.pathname.com/fhs/) #################################################################### # Variables you might need to change # Unfortunately, if you install a new version of jpbackup, any changes here # will be lost. # These are the actual directories that get backed up for each job or type. # You can make them the same if you want # Remember that no matter what is listed here, directories in nobackup file # will not get backed up! BUWEEKLYJOB="/etc /home /root /var/log /var/named /var/lib/mysql /var/spool /var/www /opt /usr/src/redhat" BUMONTHLYJOB="/etc /home /root /var/log /var/named /var/lib/mysql /var/spool /var/www /opt /usr/src/redhat" #BUMONTHLYJOB="/" BUJOB="" TAPEDEVICE=/dev/tape # Tape device to use EMAIL=root # Address to send completion e-mail to GZIPCOMPRESSION=9 # GZip compression level; 1=fastest, 9=smallest; GZ default=6 # Number of tapes (weekly and monthly) [I use 8, see Other Notes below] NUMWK=3 # Weekly NUMMO=5 # Monthly # My other scripts that we need too. If you don't want to use them # comment them out here and where they are checked for and called. CALCSUM=/opt/jpbackup/calcsum.zsh # Provides total space used (disk and tape) RDISK=/opt/jpbackup/RDISK # Gathers recovery info about file system(s) #################################################################### # Variables you probably should not change # File path and names (per FSH-2.1) LOGFILEPATH=/var/log/jpbackup/ # Default log file patch LOGFILE=${LOGFILEPATH}jpbackup.log # The log file for this script EXFILEPATH=/var/opt/jpbackup/ # Default EXception and flag file path NOCOMPRESS=${EXFILEPATH}nocompress # Files not to compress NOBACKUP=${EXFILEPATH}nobackup # Directories not to backup LASTMONTHLY=${EXFILEPATH}lastmonthly # The last Monthly tape used LASTWEEKLY=${EXFILEPATH}lastweekly # The last Weekly tape used LASTTAPE=${EXFILEPATH}lasttape # The last tape used, period #################################################################### # Make sure the files we need exist, create them with useful defaults if not. if [ ! -f "${LASTMONTHLY}" ]; then # The combination of this setting and the next 2... echo "${NUMMO}" > ${LASTMONTHLY} fi if [ ! -f "${LASTWEEKLY}" ]; then # ... will start off so that the first backup... echo "${NUMWK}" > ${LASTWEEKLY} fi if [ ! -f "${LASTTAPE}" ]; then # ... is weekly 1 (Monday_1) echo "monthly" > ${LASTTAPE} fi if [ ! -f "${NOCOMPRESS}" ]; then # These are the defaults, except the last two, which I added for Ghost images echo ".Z .z .gz .bz2 .tgz .arc .zip .rar .lzh .lha .uc2 .tpz .taz .tgz .rpm .zoo .deb .gif .jpeg .jpg .tif .tiff .png .gho .GHO" > ${NOCOMPRESS} fi if [ ! -f "${NOBACKUP}" ]; then # I really doubt you want to back these up (esp. /proc) echo "/proc" > ${NOBACKUP} # Add others to real file once created... echo "/tmp" >> ${NOBACKUP} echo "/var/tmp" >> ${NOBACKUP} fi if [ ! -d "${LOGFILEPATH}" ]; then # Make the log file directory mkdir ${LOGFILEPATH} fi if [ ! -d "${EXFILEPATH}" ]; then # Make the EXception and flag file directory # Note this will make parent directories as needed mkdir -p ${EXFILEPATH} fi if [ ! -f "${CALCSUM}" ]; then # Is calcsum.ksh there? echo "${CALCSUM} not found. Aborting. Reinstall jpbackup." exit 3 fi if [ ! -f "${RDISK}" ]; then # Is rdisk there? # (Note RDISK will not complain on errors, it just won't work right. Better check it.) echo "${RDISK} not found. Aborting. Reinstall jpbackup." exit 3 fi #################################################################### # Figure out what we are doing # What did we do last time? MYLASTTAPE=`cat ${LASTTAPE}` MYLASTWEEKLY=`cat ${LASTWEEKLY}` if [ ${MYLASTTAPE} = weekly ]; then # If we did a weekly last time if [ ${MYLASTWEEKLY} -ge ${NUMWK} ]; then # And we did the LAST weekly BUTYPE=monthly # Do a monthly now BUJOB=${BUMONTHLYJOB} # Use the Monthly data set MYLASTMONTHLY=`cat ${LASTMONTHLY}` # Which monthly did we do last? if [ ${MYLASTMONTHLY} -ge ${NUMMO} ]; then # If we did the LAST monthly THISTAPE=1 # Start at the beginning again TAPENAME=Month_${THISTAPE} else # Otherwise, just do the monthly let THISTAPE=${MYLASTMONTHLY}+1 fi TAPENAME=Month_${THISTAPE} else # OK; we're in the middle of the weekly cycle: just do it BUTYPE=weekly BUJOB=${BUWEEKLYJOB} # Use the Weekly data set let THISTAPE=${MYLASTWEEKLY}+1 TAPENAME=Monday_${THISTAPE} fi elif [ ${MYLASTTAPE} = monthly ]; then # If we did a monthly last time, now we restart the weekly cycle BUTYPE=weekly BUJOB=${BUWEEKLYJOB} THISTAPE=1 TAPENAME=Monday_${THISTAPE} else # What the heck? Should never get here echo "" | tee -a ${LOGFILE} echo "`date`; ERROR: MyLastTape not set right!" | tee -a ${LOGFILE} echo "" | tee -a ${LOGFILE} exit 2 fi # Now that we know what to do -- set the names (You probably will not need # to touch these). Note DumpFiles are in TMP so they will not get backed up # (see nobackup), should also be removed, so if they are still there, # you probably had a problem. DUMPFILE=/tmp/${TAPENAME} # File find dump (easy to capture to verify it's as expected) CATFILE=${LOGFILEPATH}${TAPENAME}.cat # "Catalog" for the tape AFIOLOG=${LOGFILEPATH}${TAPENAME}.log # afio's log file if [ -f "${AFIOLOG}" ]; then # Remove old TapeName.log file rm -f ${AFIOLOG} fi #################################################################### # Provide a ton of operation feedback echo "" echo "Running $0 '${MYVER}' on `date`" echo "BUType: ${BUTYPE} TapeName: ${TAPENAME}" echo "NumMO: ${NUMMO} NumWk: ${NUMWK}" echo "ThisTape: ${THISTAPE} GZip Level: ${GZIPCOMPRESSION}" echo "BUWeekly Job: ${BUWEEKLYJOB}" echo "BUMonthly Job: ${BUMONTHLYJOB}" echo "This BU Job: ${BUJOB}" echo "" echo "LastTape: `cat ${LASTTAPE}` Path: ${LASTTAPE}" echo "LstMntly: `cat ${LASTMONTHLY}` Path: ${LASTMONTHLY}" echo "LstWkly: `cat ${LASTWEEKLY}` Path: ${LASTWEEKLY}" echo "" echo "DumpFile path: ${DUMPFILE}" echo "CatFile path: ${CATFILE}" echo "afioLog path: ${AFIOLOG}" echo "LogFile path: ${LOGFILE}" echo "NoBackup path: ${NOBACKUP}" echo "NoCmprss path: ${NOCOMPRESS}" echo "CalcSum: ${CALCSUM}" echo "rdisk: ${RDISK}" echo "" #################################################################### # Now go do it # Make sure the tape is rewound, then erase it (also initializes new tapes) STARTDATE=`date` echo "${STARTDATE}; START: ${BUTYPE} backup to ${TAPENAME}..." | tee -a ${LOGFILE} mt rewind mt erase # Run my RDISK script, but do not write anything to floppy ${RDISK} nopfy echo "Doing ${BUTYPE} backup to ${TAPENAME} -- Collecting files..." # Find the files to backup, then use grep to remove anything listed in the # NoBackup file. Save data twice, once as just names, and once with file # size too. Note the NoBackup file (grep) only gets applied to the file # size data stream here. afio applies it to the backup later using the # name only stream. find ${BUJOB} -fprintf ${DUMPFILE}.name "%p\n" -printf "%s\t%p\n" | grep -v -f ${NOBACKUP} > ${DUMPFILE}.size SIZEONDISK=`cut -f1 ${DUMPFILE}.size | ${CALCSUM}` # Get the files sizes (on disk), and sum them # Actual afio command (all the rest of this crap is just gravy!) # We take the data set we collected in the find above, send it through # grep to apply the NoBackup file, then give it to afio to backup. # To remove all compression, nuke (here and below) the -Z, -E ${NOCOMPRESS} # and -Q "-c${GZIPCOMPRESSION}" # To remove the boring e-mail, nuke (here and below) the -@ ${EMAIL} # You may still get an e-mail from cron though! # -b is critical block size for DI-30! If you have memory problems, play # with -c and -M # -c 1024 = 32 Meg of I/O buffering, -M 10m = 10 Meg of GZip buffering! # cat ${DUMPFILE}.name | grep -v -f ${NOBACKUP} | afio -ov -b 32k -c 1024 -M 10m -Z -Q "-c${GZIPCOMPRESSION}" -E ${NOCOMPRESS} -@ ${EMAIL} -L ${AFIOLOG} ${TAPEDEVICE} # 2002-04-01 cat ${DUMPFILE}.name | grep -v -f ${NOBACKUP} | afio -ov -b 32k -c 1024 -M 10m -Z -E ${NOCOMPRESS} -L ${AFIOLOG} ${TAPEDEVICE} cat ${DUMPFILE}.name | grep -v -f ${NOBACKUP} | afio -ov -Z -E ${NOCOMPRESS} -L ${AFIOLOG} ${TAPEDEVICE} # Misc. admin rm -f ${DUMPFILE}.name ${DUMPFILE}.size echo "`date`; FINISH: ${BUTYPE} backup to ${TAPENAME} (data: ${SIZEONDISK})..." | tee -a ${LOGFILE} echo "`date`; START: Verify ${BUTYPE} ${TAPENAME}..." | tee -a ${LOGFILE} cd / # Have to be in / or the verify will fail with "No such file or directory" (relative paths) # We verify and create a "catalog" by redirecting STDOUT and STDERR to the # catalog file # The egrep -v below works around a minor afio bug that prints # "-- compressed" twice. That bug is fixed in afio v2.4.7 & my RPMs # (http://www.jpsdomain.org/public/rpms). # To remove all compression, nuke (here and above) the -Z and -E ${NOCOMPRESS} # To remove the boring e-mail, nuke (here and above) the -@ ${EMAIL} # -b is critical block size for DI-30! (Not sure if -c and -M help on verify, # but...) # -c 1024 = 32 Meg of I/O buffering, -M 10m = 10 Meg of GZip buffering! # afio -rv -b 32k -c 1024 -M 10m -L ${AFIOLOG} -Z -E ${NOCOMPRESS} -@ ${EMAIL} ${TAPEDEVICE} 2>&1 | egrep -v "^ -- compressed$" | tee ${CATFILE} # 2002-04-01 afio -rv -b 32k -c 1024 -M 10m -L ${AFIOLOG} -Z -E ${NOCOMPRESS} ${TAPEDEVICE} 2>&1 | egrep -v "^ -- compressed$" | tee ${CATFILE} afio -rv -Z -E ${NOCOMPRESS} -L ${AFIOLOG} ${TAPEDEVICE} 2>&1 | tee ${CATFILE} # The egrep gets only digits, optionally surrounded by white space, since the # cut gets other crap now and then. HOWEVER, it also missed any LARGE file # which serious skewed the results!!! The gawk line is more accurate. # REALLY BROKEN: SIZEONTAPE=`cut -b 32-40 ${CATFILE} | egrep "^\W*[[:digit:]]+\W*$" | ${CALCSUM}` # Get the files sizes (on tape) SIZEONTAPE=`gawk '{ print $5 }' ${CATFILE} | egrep "^\s*[[:digit:]]+\s*$" | ${CALCSUM}` ###### If we (finally ever) get to here, update the flag files that keep track # of what we did. Since we don't update them before this, if we barfed above, # the next job will pick up where we left off. That means you'd better notice # and fix the problem and run it again before next week! echo ${BUTYPE} > ${LASTTAPE} # Keep track of the last type of backup we did if [ ${BUTYPE} = weekly ]; then # We did a weekly echo ${THISTAPE} > ${LASTWEEKLY} else # We did a monthly echo ${THISTAPE} > ${LASTMONTHLY} fi # Retension the tape (FF to end, Rewind to beginning), then offline it. # On a DI-30, offline does not eject the tape, when nopt using OSST, but # it seems to help make it pop out faster when you do manually hit the # eject button. mt retension mt offline # Set/re-set some permissions chmod 600 ${LOGFILEPATH}* chmod 600 ${EXFILEPATH}* # Print exit message (note first line to console only, others to console and log) echo "${STARTDATE}; START: ${BUTYPE} backup to ${TAPENAME}..." echo "`date`; FINISH: Verify ${BUTYPE} ${TAPENAME}..." | tee -a ${LOGFILE} echo "Data size on disk: ${SIZEONDISK}, size on tape (GZ ${GZIPCOMPRESSION}) ${SIZEONTAPE}." | tee -a ${LOGFILE} echo "`date`; Restarting NTPd..." | tee -a ${LOGFILE} /etc/init.d/ntpd restart | tee -a ${LOGFILE} echo "" | tee -a ${LOGFILE} #################################################################### #################################################################### # Other notes ### To Restore (which is NOT automated here, but see my "tape" script) # To restore everything from a compressed tape archive # To overwrite, you need to be in the / dir. (see the -n option!) # If you are someplace else, the dir. structure will be re-created. # afio -ivxZ -b 32k -M 10m -L /var/log/jpbackup/jpbackup.log -@ root /dev/tape # To restore just "/root" from a compressed tape archive # To overwrite, you need to be in the / dir. (see the -n option!) # If you are someplace else, the dir. structure will be re-created. # Can handle leading / in the path. # afio -ivxZ -b 32k -M 10m -L /var/log/jpbackup/jpbackup.log -@ root -y "/root/" /dev/tape # jpbackup uses 8 tapes # Monday_1, Monday_2, Monday_3 = 1st, 2nd, 3rd Monday of the month, # backup important, dynamic stuff # Month_1, Month_2 ... Month_5 = 4th Monday of 1st, 2nd .. 5th month, # backup just about everything # RedHat Vixi Cron entry to run every Monday at 4:20 am # 20 4 * * Mon /opt/jpbackup/jpbackup # afio options used (see 'man afio' for more information) # afio -ov -b 32k -c 1024 -M 10m -Z -Q "-c${GZIPCOMPRESSION}" \ # -E ${NOCOMPRESS} -L ${AFIOLOG} -@ ${EMAIL} ${TAPEDEVICE} # -o reads pathnames from the standard input and writes an archive # -v verbose; report pathnames as they are processed, -t gives an ls -l style # -b {size} read or write {size} archive blocks # -c {count} buffer count archive blocks between I/O operations # -M {size} specifies the maximum amount of memory to use for compression # [default -M 2m] # -Z gzip the files on the way out and in # -Q "{opt}" pass the option {opt} to the compression program # (Need quotes to correctly pass the argument.) # -E read non-compressible file extensions separated by white space, # from {filename} # -L {Log_file_path} the name of the file to log errors and the final totals to # -@ {address}send email to address when the operation is complete # /dev/tape & ntape are symlinks to the real /dev/ht0 & nht0 tape devices # Other options (do not use -Q, even if used in backup creation, do need -b, # also need -Z if -Z used in creation. # -t reads an archive and writes a table-of-contents to the standard output # -i installs the contents of an archive relative to the working directory # Use -iy to do a partial or selective restore (E.G. -iy "root/stuff*") # -n protect newer existing files (comparing file modification times) # -x retain file ownership and setuid/setgid permissions (default # for root) # -r reads archive and verifies it against the filesystem. # -W {filename} do not process files whose names match shell patterns in # {filename} # Note, we could have used -W instead of the "grep -v -f" above, but using # the find/grep is easier to debug, because you can get to see the file list # *before* afio gets it. # Other Monthly method we could have used # We used the one above because it catches more, even if non-standard # directories are added to / # find /bin /boot /dev /lib /misc /opt /sbin /usr /var/lib -print > ${DUMPFILE} # cat ${DUMPFILE} | afio -ovxZ -b 32k -c 64 -M 10m -Q "-c9" -L ${LOGFILE} -E ${NOCOMPRESS} -@ ${EMAIL} /dev/tape