Welcome to my domain… The domain name is a play on words, because I couldn’t think of a more creative one. I kept asking my wife (who is far more creative than I) what my domain name should be. I suspect she got sick of me asking, and this is the result.
This is my personal site and does not represent the opinions of my employer.
O’Reilly Books
I am the co-author on the following O’Reilly Books.
We created a
Website for All Things Bash for
the cookbook. Among other things, it indexes a lot of great material
from the bash source code that most people never see.
I finally got around to migrating this site from my own personal static site generator to
Hugo using the
Relearn theme, which I’ve tweaked a bit. That involved converting my “templated” HTML to Markdown, using
Pandoc, various Bash, and some Perl one-liners. That took about four days of solid work. It turns out web technologies have change a bit since the last time I did this. Who knew? But “static site generators” are not a recognized thing. When I wrote my own in 2003 I didn’t have a name for it, I just knew that I wanted static HTML with a navigation bar. I was about five years too soon for Jekyll, and ten for Hugo.
2003
In September 2003 I redesigned the site format, for the fourth time if I remember correctly. I hope this format will be more simple, and easier to navigate and maintain. (UPDATE: 2006-07-27: But I doubt it is. I’ll be re-designing again, using a real CMS this time, Real Soon Now.)
This site was created and is maintained using
GenSite.pl, a simple Perl script I wrote to implement a template-based site (now called a “static site generator”). This was needed to achieve my goal of getting rid of all the JavaScript, frames and other junk and am using nothing but HTML with CSS and nested tables. I use a simple text editor to edit templates files, then run my script to regenerate the entire site, including navigation menus, in seconds.
I made a conscious decision to stay as simple as possible while still being usable and useful, and while still leaving display choices up the the viewer. One of the things that bothers me quite a lot of badly designed site that force you to view them in a certain window size. If I wanted to see my web sites in 800x600, I would not have a 20 inch monitor! Thus this site allows you to view in any window and font size your browser permits.
That also raises the question of line length. One of the reason many annoying sites force a window size is to gain more control over formatting, particularly graphics layout and line length. Lines that are to long are more tiring to read. This site has a very simple solution. Resize your browser window to any line length you find comfortable. Try adjusting your font size while you are at it.
I found the following helpful at the time of the redesign in 2003:
According to
The Internet Archive this site has been at this address since at least 2001-04-01, but it’s older than that. I experimented with MS FrontPage for a while, but the HTML code that churned out was so awful I stopped using it. I’m don’t recall what other tools I may have used until the 2003 re-write.
Favicon
I created my favicon.png (which started as favicon.ico) around this time as a play on the basic DOS/Windows C:\> prompt, in the garish blue background/yellow font I used for terminal windows at the time. I had been playing with DOS batch files since 1986 or so, but was getting more into “scripting” at the time. I’ve kept it ever since, converting formats as needed.
1995
This site started in 1995 as
JP’s Boring Homepage, my “home” page on my ISP at the time,
Netaxs (Net Access – Philly’s Original
Internet Access Provider). I don’t recall how I wrote it, but it was likely in Notepad.
If you have any comments about the site, additions to my tools pages, or
if you find something broken, please email me at JPATjpsdomainDOTorg
.
JP has been working with computers since the early Eighties and has been
in the IT industry since the early Nineties, specializing in Information
Security since the late Nineties. He spent some of that time working as
a consultant and about 12 years as a Senior Security Engineer for BT MSS
(i.e., BT Counterpane). He was also
Director of Customer Support for SGP Technologies (Blackphone.ch) for a
while, but prefers hands on technical work.
Mr. Vossen has worked with DOS, Windows, UNIX, VMS and AS/400 platforms,
with duties ranging from first-level technical support to network and
security architecture and design. He currently loves to work with Linux,
Perl, Snort and other Open Source and Free Software but is otherwise not
much of a programmer. Unlike many of his technically inclined
colleagues, he also enjoys writing and documentation, which has led to
the publication of various InfoSec articles, scripts and tips,
O’Reilly’sbash
Cookbook 2nd,
and O’Reilly’sbash
Idioms.
2001-03: “Kane
Enable,”
a (~ 3,000 word) review of Kane Security Analyst v5. (Note: I
will not take the blame for the title, that goes to then
Editor-In-Chief, Andy Briney ;-)
A Whitepaper (PDF) and
slideshow
(PDF),
copyright 2002 AlphaNet
Solutions, called “Securing
(Hardening) Windows Servers.” Also download the
MoveTools batch file, then rename
from .TXT to .CMD as needed. They were written for a “TechNet
Workshop: Microsoft Security Solutions” presentation on January
22nd, 2002 at the Microsoft Greater PA Office in Malvern,
PA.
Reviewed and contributed to a draft of “The 60 Minute Network
Security Guide (First Steps Towards a Secure Network Environment)”
from the NSA Systems and Network Attack
Center (SNAC). This was referenced at
http://www.sans.org/newlook/resources/NSA_guide.htm, but that page
is no longer there. I’m guessing the document is undergoing revision
again. You might check the NSA Security Recommendation
Guides site.
See the Windows port of Logcheck
page for my Windows port of
logcheck,
the famous UNIX log processing tool. You can also download the
PowerPoint presentation I gave
to the Philadelphia Area Network Technologies User Group
(PANTUG) on September 12th 2001.
I have written a couple of management scripts for CheckPoint
Firewall-1 (LogSwap & CPFWBack). See my Firewall-1 tools
section for more
information.
I have also packaged an NTP service that will run on a really
hardened NT server (unlike the non-free MS TimeServ service). See my
Firewall-1 tools
section for more information.
My Job:
Here is my resume in
PDF or
HTML, last updated 2021-07-25.
Anyway, my certifications include:
CISSP
(Certified Information
Systems Security Professional)
#11049,
granted March 4 2000, to present.
Microsoft Certified Systems Engineer (MCSE); late 1990’s
CheckPoint Firewall-1 v3.x Certified
Systems Engineer (CCSE); mid-late 1990’s
a Novell 3.x
CNA
(Certified Novell Administrator); mid 1990’s
See my Bio above.
I have worked with many small and mid-size companies in eastern
Pennsylvania and New Jersey. I have experience with companies in a broad
range of markets, including Telecommunications, Pharmaceutical,
Financial, Healthcare, and Manufacturing.
I have worked with networks from Netware Lite and Netware 3.x on up to
30+ site TCP/IP WANS. I have written programs in COBOL, Pascal, C,
BASIC, Perl, Bourne Shell and many application macro languages including
VBScript. I have also been a beta tester for Microsoft (Win95 & NT4),
Symantec and others.
What I like to do:
I am kind of a generalist rather than a specialist, one reason for my
interest in Information Security–it is a very broad and far-reaching
topic. I find I enjoy the tactical (in-the-weeds) side of things much
more than the strategic. I really get into the nuts and bolts of how
best to accomplish the goal and am very methodical about the process and
documentation. I do best with solid, uninterrupted blocks of time during
which I can fully focus on a task (see Maker’s Schedule, Manager’s
Schedule).
I really enjoy:
Scripting, building system & tool automation/integration
Ansible, shell scripts, Perl, Regular Expressions, Unix
TextUtils and pipelines
DevOps concepts and implementation
Technical writing and documentation (ideally in wiki markup)
Getting as much as possible under revision control (git, bzr, hg,
svn)
Linux (mostly Debian and Ubuntu/Mint, and CentOS) and other Free and
Open Source software
The likelihood of a repair or modification working is directly dependent on whether you’ve reassembled everything without testing it first and inversely proportional to the difficulty of disassembling everything in the first place.
Corollary #1
If it is REALLY difficult to remove and replace, you will break something else just as you get it back together after fixing the first thing.
This really became clear to me working on servers (especially rack-mounted), cars (especially Hondas) and small electronics. But it seems like it should apply to just about anything that is a pain to get at or disassemble.
This content is old! It’s still useful, but it’s old, and there may be bit rot, newer/better tools or ways to do things. Sanity check and do your research.
Bash (readline) Hints
If you use bash, are as bad a typist as I am, and especially if you came
up in the DOS/Windows world, there is one thing that probably drives you
nuts—the way bash’s file/path completion works. I hate that bash
either complains or just gives you a list of the possibilities on an
ambiguous completion. I’d much rather have it cycle through the
possibilities each time you hit TAB the way (shudder) cmd.exe/4NT.exe
does. (See Windows Shell Scripting
to enable this in cmd.exe.)
So how can you get bash to cycle though files or directory completion
the way 4NT does? It turns out
this is possible in bash since 2.02, but it’s amazingly obscure. The
feature is called “menu-complete”, and you can enable it in the
/etc/inputrc file by binding it to TAB.
Here are the tweaks I add to the top of my /etc/inputrc file in Linux
(note second to last):
set completion-ignore-case on # Ignore case when doing completionset mark-directories on # Completed dir names have a slash appendedset visible-stats on # List ls -F for completion"\C-i": menu-complete # Cycle through ambiguous completions instead of list#set show-all-if-ambiguous on # List possible completions instead of ringing bell
You can edit the file and test it using bind -f /etc/inputrc to
activate your changes immediately.
Why Use Open Source Software?
The two best papers I’ve seen on the subject are the following:
OSS/FS has significant market share, is often the most reliable
software, and in many cases has the best performance. OSS/FS scales,
both in problem size and project size. OSS/FS software generally has
far better security, particularly when compared to Windows. Total
cost of ownership for OSS/FS is often far less than proprietary
software, particularly as the number of platforms increases. These
statements are not merely opinions; these effects can be shown
quantitatively, using a wide variety of measures. This doesn’t even
consider other issues that are hard to measure, such as freedom from
control by a single source, freedom from licensing management (with
its accompanying litigation), and increased flexibility. I believe
OSS/FS options should be carefully considered any time software or
computer hardware is needed.
Linux RPM Packages
The RedHat Package Manager (RPM) is an Open
Source “package manager” for Linux.
Developed by RedHat, it is the defacto
standard used by a majority of Linux developers and distributions. It
offers far better modularity, manageability and ease-of-use than the
more traditional “tarball” distribution method. It’s slightly easier to
use than Solaris’ package manager, in that there is only one program to
deal with. And it is vastly superior to any of the Microsoft
installers because a) it wasn’t written by Microsoft, b) it was written
for a decent OS, c) merely installing a simple application (such as a
web browser) will not 1) crash the OS completely or 2) make fundamental
changes to underlying OS services and/or functionality, d) you can
actually completely and cleanly uninstall applications, e) you can
easily get a definitive list of what packages are installed on the
system (rpm -qa).
If you do not understand item c above, go install IE 5.5 on an NT server
and see what happens. Hint, check the “AT” service.
I never liked Red Hat’s “Up to date” service. I never really got it to
work, and I just don’t like the idea of how it works. It’s also a pain
to have a local repository.
I used to use a Perl based RPM updated called
autoupdate.
It worked very well for me. It is highly configurable and supports a
distributed architecture where I can have one server download updates,
then all my other machines get the updates from that local server. I
found autoupdate needed an hour or two’s time investment to get set up
and working, but it was well worth it. It’s not too complicated, it’s
just that there are quite a few options and it may take some reading and
experimentation before you find a setup that works the way you want.
But now I’m using cAos Linux, which uses
Yum (Yellow dog Updater,
Modified), which rules! A simple yum update updates every single
installed application from my local repository, which I rsync daily. If
I need to install an application, such as NTP, yum -y install ntp does
the trick, resolving any dependencies along the way. Yum uses a simple
web site for repositories making it drop-dead simple.
Fedora is even using Yum. Yum Just Works,
and is awesome!
This content is old! It’s still useful, but it’s old, and there may be bit rot, newer/better tools or ways to do things. Sanity check and do your research.
$Revision: 1.5 $, $Date: 2026-02-15 15:31:17 -0500 (Sun, 15 Feb 2026) $
UTC
Introduction
See “Update (2003-11-29)–OnStream Bankrupt again” for
important bankruptcy information about OnStream.
This document describes how to use an OnStream DI30 tape drive with Red
Hat Linux and several free backup utilities. It is intended for a
anyone planning to use an OnStream DI-30 tape drive, or anyone trying to
backup Linux, especially Red Hat. Most especially, it’s intended for
anyone trying to do both!
It assumes some basic hardware and Linux/UNIX knowledge but little to no
knowledge of tape drives and tape backup software on Linux. It provides
the tools (all of which are free) and information need to implement a
relatively simple rotating weekly backup scheme that is suitable for
home or small business use.
If you have not already purchased an OnStream drive to use with Linux,
read Part 1 to make sure this is an acceptable solution for you. You
might also want to skim the rest to make sure you are comfortable with
everything. Then check the OnStream site at
http://www.onstreamdata.com/
especially the DI30 product page at
http://www.onstreamdata.com/desktop/di30_d.html.
If you have already bought the drive–read on!
I wrote this because I could not find anything already out there that
answered my need. Since I had to do the research anyway, why not
document it properly? I am going to be pretty specific with this
mini-HOWTO, because I do not have a lot of resources (time or equipment)
to spend on this. If you have different experiences, or can add
information, please let me know. Contact information is included with
the history section at the end.
This mini-HOWTO covers both of the situations where you must reboot
Linux. That is, you should only ever have to reboot Linux when adding or
replacing non-hot-swappable hardware, and when you need to switch to a
different kernel version. Other than that, you should never need to
reboot!
This documents IDE devices only. It does not cover any OnStream SCSI
devices. It may still be helpful–Your Mileage May Vary.
Also, I am not affiliated in any way with Red Hat, OnStream or anyone
else mentioned herein.
Update (2003-11-29)–OnStream Bankrupt again
It seems that all the “official” OnStream sites listed in this
document are off the air! That is a Bad Thing. It you know why and can
point me to new sites, please let me know. It
looks
like they have gone bankrupt. Again…
You can find software, firmware, drivers, manuals and support at
Hastec who seems
to be a reseller of some kind. There are also 3 (as of 2003-11-30) files
at http://www.driverguide.com/. This site requires a free membership
just to search, which is highly annoying.
Update (2001-11-24)
I have switched to the OSST drivers, as they work much better for
me. I’ve also updated this document to include OSST information.
Update (2001-10-29)
I just got a message from Jack, an OnStream Software Development Manager
in the Netherlands, with some excellent up-to-date Linux information.
Here are the high points. Note that this site used to be something like
http://linux1.onstream.nl/.
“We […] have a server that is dedicated to Linux issues and which
also hosts a mailing list for driver development. Have a look at
http://www.linux1onstream.nl/ where you’ll find a description
of the list and driver sources for download (see
http://www.linux1onstream.nl/test).
“Another interesting tidbit is that you can find firmware updaters
here that run on Linux as opposed to the […] bootable DOS floppy
tool that you refer to [in your HOWTO] (look at
http://www.linux1onstream.nl/Firmware/).
“Finally, we plan to put some information up on tapetype definitions
such as are required by Arkeia and Amanda.
“[…] The DI-30 solution we most often advise our customers to
use is ide-scsi emulation combined with the osst driver (
http://www.linux1onstream.nl/test/ide-tape.html). This
solution will offer you more features (such as 512 byte block size and
“mt eject” e.g.) but also better performance (since it uses a filemark
list on tape for rapid seek operations).”
This howto and the associated documentation and scripts are distributed
in the hope that they will be useful, but WITHOUT ANY WARRANTY; without
even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
PURPOSE. See the GNU General Public License for more details.
In no event shall the author be liable for any damages whatsoever
(including, without limitation, damages for loss of business profits,
business interruption, loss of business information, or any other
pecuniary loss) arising out of the use of or inability to use this
documentation or scripts.
If you have questions or comments, please contact me at JPATjpsdomainDOTorg
.
Obsolete Content
This content is obsolete, but I am leaving it here as a historical reference.
Can be made to work with Linux (with a little effort)
Installing the Drive & Configuring the System
I had a lot of trouble accomplishing this, which is largely this reason
I wrote this document–I couldn’t find anything to help me. Two issues
particularly stick out in my mind. 1) I was not very comfortable
re-compiling or installing kernels. 2) I had no idea how it should
work, and what it would look like then it was, in fact, working!
What I’ve learned is that the Linux kernel is pretty darn
resilient–it’s hard to screw it up (but I managed!). When you do
screw it up, it’s very good about telling you, “No, dumb-ass, you have
to turn on packet filtering to allow DNS to run” or whatever. I’ve also
learned how to make the drive work, and what it should look like when it
does. I hope you find that the information below answers your
questions!
After finally getting everything to work, and writing the backup scripts
included below, I am very happy with this solution. It is very
inexpensive by any metric you want to use; cost of data loss, cost of
alternative high-capacity solutions, cost in media and tape-swapping
time for other low-capacity solutions, etc. It works quite well for me,
but of course your mileage may vary.
First, install the hardware as covered in the documentation that came
with your drive. In my case, I had to move the master/slave jumper to
master, but that was the only change I made. Otherwise, I took it out
of the box and plugged it in. Note that the red stripe on the IDE cable
(for pin 1) goes AWAY from the power connector in the drive, which is
the opposite of hard drives and CD-ROMs!
Per OnStream Tech Support, do not make your drive a slave to an IDE hard
drive. Either make it a master on the second IDE interface, or make it
a slave to IDE CD-ROM. Do not make an IDE hard drive a slave to an
OnStream tape either.
First, figure out with driver (next three sections) you are going to
use. Read the sections and get a feel for everything. Then, follow the
instructions for the section. Since you have to reboot anyway, don’t
bother installing the drive until after setting up the driver.
I originally tested this under Red Hat Linux 7.1, then upgraded to 7.2,
7.3 and 8.0.
Red Hat 8.0 Just Works with OSST drivers for a DI-30!!! On a
clean install I did not need to add an append to the boot loader, or
create the device files. I did make sure the modprobe lines where in
/etc/rc.local though. Presumably the same is true for Red Hat 9 and
Fedora, but I have not tested that. So you can skip below and install
the drive.
The good news is that the modules you need are already built in to Red
Hat 7.1 and more recent distributions, so this is pretty easy, and it
seems to work a lot better for me than the IDE interface.
First, you need to edit your /etc/lilo.conf file. You need to know
which IDE interface your OnStream is connected to. If you don’t know,
cat /proc/ide/hd**x**/model where x is a,
b, c, or d. Mine is hdc. You should see something like “OnStream
DI-30” when you get the right one.
Edit /etc/lilo.conf and add ‘append=”hdx=scsi”’ where hdx is the
correct IDE interface. This allows the OSST driver to grab that IDE
drive and emulate OnStream SCSI on it (more or less). For example, my
lilo.conf looks like this (remember, my DI-30 is on hdc):
Now you need to load the correct modules. You should not need
“IDE-Tape” when using OSST. You will need “ide-scsi” and “osst.” Since
you need to reboot so the hdx=scsi will take effect, you have two
options here. You can do nothing, reboot, and load the modules manually
to see what happens. Or you can add the modules to be loaded now, and
then reboot. We’ll do the former.
But first, you need to create the device files.
Verify that you do not have the device files–you should get nothing
using this command:
ll/dev/osst\*/dev/nosst\*
Issue this command to extract Makedevs.sh tar xvzf onstream 20011101.tar.gz onstream/driver-24/Makedevs.sh
Run ./onstream/driver-24/Makedevs.sh
Verify that you have the device files
ll/dev/osst\*/dev/nosst\*
Optionally, remove the source and directory you just created
rm ifonstream\* and you may also want to
remove the temp directory, if you used one
Power down and installed the drive. Power back up. After some services
start, and if you are using Kudzu (Red Hat’s hardware recognition
program), you may be asked if you want to configure your new OnStream
DI-30, TAPE drive. Say yes. After logging in as root, type
modprobe ide-scsi and
modprobe osst. If you see something like the
capture below, everything is almost working (commands you type are in
bold).
You should be able to access the drive at /dev/osst0 or /dev/nosst0
(assuming this is your first tape drive). You can test this with this
command:
mt -f /dev/nosst0 status
Assuming all of that worked, you now need to add the modprobe lines so
they are called when the system in next rebooted. Edit /etc/rc.local and
add something like this:
# 2001-11-15 JPV Added modprobes for OSST stuff
# 2002-05-05 JPV Upgraded to RH 7.2
# 2002-10-06 JPV Upgraded to RH 8.0
modprobe ide-scsi
modprobe osst
That’s it!
Skip down to the “Tape Device Files” section.
Using IDE Drivers with Kernel 2.4.x (Not recommended)
I have only tested this under Red Hat Linux 7.1.
After you finish installing the drive and power back up, you should see
something like the following. If you miss it, try
dmesg | grep -i hd once you have logged in):
hda: WDC WD450AA-00BAA0, ATA DISK drive
hdb: IOMEGA ZIP 100, ATA DISK drive
hdc: OnStream DI-30, ATAPI TAPE drive
hdd: ATAPI CDROM 48X, ATAPI CDROM drive
After some services start, and if you are using Kudzu (Red Hat’s
hardware recognition program), you will be asked if you want to
configure your new OnStream DI-30, ATAPI TAPE drive. Say yes. This
will create the device files you need.
That’s it! Once the system is completely up and you log in, you should
be able to access the drive at /dev/ht0 or /dev/nht0 (assuming this is
your first tape drive). You can test this with this command:
mt -f /dev/nht0 status
Skip down to the “Tape Device Files” section.
Using IDE IDE Drivers with Kernel 2.2.x (REALLY Not recommended)
I have only tested this under Red Hat Linux 6.2.
While not trivial, this is not as bad as you think it is. Sooner or
later, if you continue to use Linux, you’re going to have to learn how
to compile stuff, especially the kernel. Why not start now?
“The new OnStream Drive (30gig drive in IDE, SCSI, and parallel
flavors) does NOT work under 6.2. OnStream Inc. is currently working
to develop a driver however. To reiterate, not even the SCSI version
works yet.”
Obviously, they are wrong, but they are right with one thing–OnStream
tape drives will not work with Red Hat 2.2.x kernels–you do have to
roll your own (does not apply to 2.4.x kernels. If you are using a
2.4.x kernel, you are a reading the wrong section!).
Get the latest pristine kernel source (as I write this on
2001-11-24, it’s v2.2.20). Do not use 2.2.16 as it has security
issues. When I wrote the rest of this, I was using 2.2.18.
However, I can only find the IDE patch for 2.2.19, so:
ftp://ftp.us.kernel.org/pub/linux/kernel/v2.2/linux-2.2.19.tar.gz
or linux-2.2.19.tar.bz2
The trick here is to enable all the stuff you need, without enabling
stuff you don’t need. I recommend using make menuconfig or if running X make xconfig
as they are far easier to use and more tolerant of changing your mind
than just make config. I had some trouble
getting make menuconfig to run. It kept
whining about curses so I eventually had to
install all the ncurses RPMs on the Red Hat 6.2 CD. Something did the
trick (I suspect the ncurses-devel package), because it worked after
that.
After you finish recompiling and installing the 2.2.x kernel and reboot,
you should see something like the following. If you miss it, try
dmesg | grep -i hd once you have logged in):
hda: WDC WD450AA-00BAA0, ATA DISK drive
hdb: IOMEGA ZIP 100, ATA DISK drive
hdc: OnStream DI-30, ATAPI TAPE drive
hdd: ATAPI CDROM 48X, ATAPI CDROM drive
After some services start, and if you are using Kudzu (Red Hat’s
hardware recognition program), you will be asked if you want to
configure your new OnStream DI-30, ATAPI TAPE drive. Say yes. This
will create the device files you need.
That’s it! Once the system is completely up and you log in, you should
be able to access the drive at /dev/ht0 or /dev/nht0 (assuming this is
your first tape drive). You can test this with this command:
mt -f /dev/nht0 status
Tape Device Files
There are two types of tape device files, the rewinding device and the
non-rewinding device. As you can probably guess, the rewinding device
rewinds the tape after each operation, the non-rewinding device
doesn’t. Be careful with this! If you use the rewinding device, then
make two consecutive backups, the second will overwrite the first, which
is probably not what you wanted to do! They are specified by the
device name for the rewinding device, and the device name prefixed with
“n” (for non) for the non-rewinding device. For example, a correctly
installed DI-30’s IDE devices are: /dev/ht0 and /dev/nht0 or OSST device
are /dev/osst0 and /dev/nosst0. Well, technically the 0 indicates that
this is the first device of its type. Osst1 would be the second device,
etc.
Some tape software looks for an environment variable, imaginatively
called TAPE, to see what tape device to use if nothing is specified.
TAPE is often set to /dev/tape, which may or may not actually exist on
your system. If it does exist, it’s quite likely to be a symbolic link
to the real device. Also, note that /dev/tape may be linked to the
rewinding device! Type echo $TAPE to see if
it’s set on your system. You can edit your /etc/profile and add
TAPE=/dev/tape (or ntape) if necessary. Don’t
forget to add TAPE to an export line somewhere in there too.
You also want to create or verify the following symbolic links:
# DI-30 using the OSST driver:
ln -s /dev/osst0 /dev/tape # Rewinding device
ln -s /dev/nosst0 /dev/ntape # Non-rewinding device
# DI-30 using the IDE driver:
ln -s /dev/ht0 /dev/tape # Rewinding device
ln -s /dev/nht0 /dev/ntape # Non-rewinding device
General Tape Drive Operation
Make sure you have a reasonable recent version of “mt” installed.
Anything news than mt-st-0.6-1.i386.rpm should be OK. Then see the man
page for the “mt” command–you’re going to need it. Some highlights to
get you started are the following. Note mt’s default device is
“/dev/tape” so you should set up the symbolic links above to whatever
device you are actually using (OSST: /dev/osst0, IDE: /dev/ht0). The
device may be specified or overridden using the -f switch, such as “-f
/dev/ht0” or “-f /dev/osst0”.
Function
Command
Comments
IDE
OSST
Rewind
mt rewind
Rewind the tape to the beginning (remember about the rewinding and non-rewinding devices!)
Yes
Yes
Erase
mt erase
Erase from the current position to the end of the tape (some versions only). Thus, to clear the whole tape, rewind first. This also initializes a new tape.
Yes
Yes
Status
mt status
Get tape status (more below).
Yes
Sort-of
Eject
mt offline
Does not actually eject the tape on DI-30 tape drives, but may help if the tape will not come out when you manually press the eject button.
No
Yes
retension
mt retension
Rewind, fast forward to the end of the tape, then rewind. This increases the life of the tape.
Yes
Yes
fast forward
mt fsf #
Fast forward to the beginning of a next archive, where # is the number of archives to skip over.
Not tested
Not tested
end (of data)
mt eod
Fast forward to the end of the last archive.
Not tested
Not tested
Variable block size
Depends on backup software
Allows you to adjust the block size used on the tape for maximum efficiency.
No.
Yes, but not tested.
Try the tape status command from above. It looks like this if it works
and there is an initialized tape in the drive:
mt-st-0.7-6 and OSST Driver (nice), on Red Hat 8:
/root/# mt status
OnStream SC-, DI-, DP-, or USB tape drive:
File number=0, block number=0.
Tape block size 512 bytes. Density code 0x40 (DLT1 40 GB, or Ultrium).
Soft error count since last status=131
General status bits on (41010000):
BOT ONLINE IM_REP_EN
mt-st-0.6-1 and OSST Driver (nice):
/root/# mt status
OnStream SC-, DI-, DP-, or USB tape drive:
File number=0, block number=0.
Tape block size 512 bytes. Density code 0x40 (no translation).
Soft error count since last status=0
General status bits on (41010000):
BOT ONLINE IM_REP_EN
mt-st-0.5b-10 and OSST Driver (mixed results here):
/root/# mt status
Unknown tape drive type (type code 97)
File number=0, block number=0.
mt_resid: 0, mt_erreg: 0x24
mt_dsreg: 0x40000200, mt_gstat: 0x41010000
General status bits on (41010000):
BOT ONLINE IM_REP_EN
IDE Driver:
/root# mt status
SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 32768 bytes. Density code 0x0 (default).
Soft error count since last status=0
General status bits on (41000000):
BOT ONLINE
And the tape is not initialized, it’ll take about 10 minutes for the
command to come back and fail like this:
/root# mt status
SCSI 2 tape drive:
File number=0, block number=-1, partition=0.
Tape block size 32768 bytes. Density code 0x0 (default).
Soft error count since last status=0
General status bits on (1000000):
ONLINE
If there is no tape in the drive you’ll get this:
/root# mt status
/dev/tape: Device or resource busy
If you got the first message, congratulations. You’re all set! If you
got the second one, you forgot to erase the tape, which you need to do
before using it for the first time. This initializes the ADRL
header, about which you probably just got a bunch of console messages.
OK, we now have something to backup to. Now, what data do we want
to backup, and how do we want to do it? Before we get to that, however,
we have one more quick tape operation issue.
Finding Out What Is On An “Unknown” Tape
With a DI-30, there is no easy way to find out what is on a tape. You
have to just know. Thus, having a simple system, labeling and
cataloging are important. If you don’t know, the best you can do is try
various table of contents (ToC) commands from various tape backup
programs to see what you get. You can also try using mt to fsf and try
ToC commands again. Using my tape script, you could look for tar and
afio (cpio) archives like this:
When you get afio: "/dev/tape": No input you are past the data (i.e.
archives) on the tape. If you got nothing, then you are using the wrong
program, which means the archives are not tar or afio (cpio) or there is
nothing on the tape. If you get tar: This does not look like a tar archive–well, you can figure that one out. Likewise, afio will say
afio: "/dev/tape": Unrecognizable archive.
Part 2: Your Backup Strategy
First, as I was recently told by a friend and UNIX guru specializing in
very high-end high-availability and clustering, you have to be able to
RECOVER–you do not necessarily have to backup. Think about that for a
moment before you continue. What do you need to have to be able to
RECOVER?
Rather than rewrite what has already been well written, I now refer you
to the following two chapters:
The two articles above are very good, and I strongly recommend reading
them. However, those authors were trying to be Linux generic, and I am
being OnStream and Red Hat specific, so on with the show.
Recovery Strategy
You need to have a well thought out strategy if you are to recover data
successfully. There is no “one size fits all” strategy, because every
environment is too different. There are, however, some guidelines:
Do not backup unnecessary data. Sometimes is it difficult to
determine what is and is not unnecessary. For Linux, unnecessary
data includes the /proc pseudo-file system, possibly /tmp and
/var/tmp, possibly all of /var. However, /var contains log files,
among other things, and there may be audit trail and other reasons
to maintain log files. It also contains /var/spool/lpd, which has
printer configuration information in it.
Do backup data that changes frequently, is difficult to recreate, or
is very important. Important dynamic data includes /home and /etc.
Some people do not backup system binaries, since you have to
reinstall the system before you can restore the backup anyway.
Other people do backup binaries, as there may be multiple patches
installed that will be time consuming to reapply. It all depends on
your needs and environment. See also the mkkickstart and RDISK
sections.
Consider the amount of data you must backup, the length of time it
takes, and the time when the system may be unavailable or at reduced
performance. The speed of your tape device figures greatly into
this (e.g. the DI-30 has “up to 3.6GB/hr (1MB/s) native transfer
rate” according to the web site.
Decide whether to encrypt or compress your backup tapes, and
understand the implications. Either encryption or compression can
substantially reduce the portability of your tapes. Sometimes tapes
that are encrypted or compressed may only be restored by the exact
same software using the exact same make and model tape drive. Lack
of this single piece of software or hardware can undo your entire
strategy. Also, some backup utilities (e.g. tar) compress the
entire backup, not just the individual files. Thus, any media
errors render the entire backup useless. Encryption suffers from
similar and even worse problems, as a password is added to all of
the above, and encrypted tapes are even more picky about specific
hardware, software and media errors.
Notwithstanding the previous issue, backup tapes must be kept
secure, or all of your other security measures are useless. Why
bother to penetrate your network or server security when a simple
backup tape offers not only the entire system on a plate, but
virtually no chance of being detector or caught, and the leisure to
take any amount of time to examine the data?
Do keep a recent backup in a secure, off-site location. If the
location of all of your backup tapes is inaccessible or destroyed,
they are not of much use. Note that it is not recommended that a
staff member take the tape home. Many difficult issues will arise
should that staff member leave the company. The best option is a
bonded security company with secure facilities that handles such
things. If that is not an option, you will have to come up with
something yourself. Carefully consider a worst case scenario and if
at all possible, have someone other than the system administrator be
responsible for backups. No single person should have total control
over all your company data!
Consider the number of tapes you are will to search or restore to
recover data. The more differential or incremental backups you
take, the more tapes you must sift through and/or restore to get
your data back to where you want it. Conversely, if it’s important
to have multiple versions of a file, this extra overhead may be
worth it.
TEST! TEST! TEST! TEST! This cannot be stressed enough. You
can’t recover a backup that was never done, or that never worked
right. Periodic testing will also discover tapes that are starting
to go bad. And periodically running a tape though (also called
retensioning) is good for the tape. Ideally, restore a large
portion of the tape to a different location, and do a file compare
between the existing and restored file structures. The least you
can safely do is a file compare using your tape software.
Types Of Backups
There are many different types of backups and thinking about them all
gives me a headache. However, you have to understand at least a little
about some of the types in order to decide which ones you need to
implement.
A full backup is just that–you backup the full system– everything.
But even that’s not true, as there are always things you never want
to backup. As I mentioned above, the /proc filesystem and temp
directories at the best example. /proc doesn’t really exist. it’s a
made-up filesystem containing all the details about the system. it’s
only a filesystem because everything in UNIX is treated as a file.
Backup it up is not only useless, some of the system is recursive (it
points to itself, more or less) so it can really confuse and even crash
your backup. Likewise, backing up the temporary directories is pretty
silly. There’s a reason they are called temporary!
Differential and Incremental backups are just different ways of backing
up data that has been changes since the last full backup. Likewise, the
“levels” used by some program (such as dump/restore) are just ways of
representing data that’s changed since the last higher level backup.
And there are all kinds of minor variations on all of the above,
especially the fact that you can use one type of backup on one day, and
another type the next day.
Finally, each type presents different problems to backups and more
importantly, restores. As you will see below, it could easily take
restores of data from 5 or mare tapes to get back you where you left
off. And try to find just one or two specific versions of specific
files? OK, I’ll pause while you go take some aspirin. Come to think of
it, I’ll take two while you’re at it.
This figure illustrates the difference between a differential and an
incremental backup. Note that in a standard Differential Backup each
backup uses the same tape, while in a Modified Differential Backup
each backup uses a different tape.
[ F U L L ]
[ B A C K U P ]
{ DIFFERENTIAL }
{ DIFFERENTIAL }
{ DIFFERENTIAL }
{ DIFFERENTIAL }
[------------------------------------------------------------------]
[ CHANGES TO YOUR DATA OVER TIME ]
[------------------------------------------------------------------]
[ F U L L ]
[ B A C K U P ] {INCREMENTAL}{INCREMENTAL}{INCREMENTAL}{INCREMENTAL}
[------------------------------------------------------------------]
[ CHANGES TO YOUR DATA OVER TIME ]
[------------------------------------------------------------------]
Then of course, there’s the data to consider. I think that’s best
explained by example.
My File System (More Or Less Typical)
The following table summarizes most of the important information about
my environment:
Directory
File system
Recovery Criteria
/a
Symlink to /mnt/floppy_dos
SKIP
/bin
/
Static
/boot
/boot
Static
/cd
Symlink to /mnt/cdrom
SKPI
/dev
/
Static, easily recreated on system reinstall
/etc
/
Dynamic, important
/fpy
Symlink to /mnt/floppy_ext2
SKIP
/home
/home
Dynamic, important
/lib
/
Static
/lost+found
/
SKIP
/misc
/
???
/mnt
/
SKIP, easily recreated
/opt
/
Static, not easily recreated on system reinstall
/proc
/proc (pseudo)
SKIP!
/root
/
Dynamic, important
/sbin
/
Static
/tmp
/
SKIP
/usr
/
Static
/var
/var
Varies widely. Much data in /var is useless and should not be backed up, while other data such as log files, mail spools and printer configuration is important and should be saved.
/var/tmp
/var
SKIP
/var/lock
/var
SKIP
/var/log
/var/log
Dynamic, important
/var/spool/mail
/var
Dynamic, important
/var/spool/lpd
/var
Dynamic, important (printer configuration data as well as actually spool file - bad design!)
/zip
Symlink to /mnt/zip
SKIP
See http://www.pathname.com/fhs/ for the Filesystem Hierarchy
Standard, (v2.1 as of this writing). This details what things should be
located where, and is an excellent reference.
My Requirements
Backup the dynamic data at least once a week.
Backup dynamic and static data (but skip the useless data) at least
once a month.
Be able to recover versions of data at least 4 months old.
Be as simple as possible to backup, find files in a catalog, and to
restore.
“Set it and forget it” except to change tapes, and allow a wide
window to actually remember to do it.
I tend to do a lot of work over the weekend, so backups should
probably be very early Monday mornings.
Did I mention it has to be simple?
Some Possible Solutions
The easiest thing to do is a full backup once a day, week or month,
depending on your environment and then just call it a day. Depending on
how much data you have, how big your tapes are and how fast your tape
drive is, this may work for you. Most of the time, not all of your data
will fit on one tape (less of a problem with 30 Gig DI-30 tapes), or
it’ll take too long to do a full backup, or something. Also, it can
take a lot of tapes, which do not grow on trees.
Seven (7) tapes labeled: Week1, Week2, Week3, Month1, Month2, Month3,
Month4. The Week tapes are used every week, either Monday to Friday, or
just Friday (note these tapes will need to be replaced most often, as
they will get the most use). Month1 is used at the end of the first
month, and so on. Either the previous week or the previous month tape
is moved off-site. Depending on space requirements, the Monday to
Friday backups could be incremental, differential or full, and could be
appended to each backup set. The Month tapes are complete system
backups. This strategy also gives you a 4 month window to recover data,
but you may lose the weekly/daily backups of different versions of
highly dynamic data, depending on exactly how you set it up.
Another possible option is eleven (11) tapes labeled: Monday, Tuesday,
Wednesday, Thursday, Friday1, Friday2, Friday3, Month1, Month2, Month3,
Month4. The Monday to Thursday tape are used every Monday to Thursday
(note these tapes will need to be replaced most often, as they will get
the most use). Friday1 is used on the Friday of the first week, Friday2
at the end of the second week, and so on. Month1 is used at the end of
the first month, and so on. Each week, the preceding Friday tape (which
will sometimes be a Month tape) is taken to a secure off-site location,
while the old off-site tape is brought back. Either the Friday or the
Month tapes are complete system backups of EVERYTHING, while the Monday
to Thursday tapes are full backups of “dynamic and important” data.
This strategy gives you a 4 month window to recover data, plus 5 days of
different versions of highly dynamic data. The most you would have to
restore is two tapes, the most recent full system, and the most recent
full data tapes. However, this requires a good number of tapes, and may
take a while to do a full backup. The Monday to Thursday backups could
also be differential or incremental, that that will substantially
increase restore complexity, while substantially lowering backup time.
Additional monthly tapes may be added to give any number of “archival”
copies.
Finally, a cheaper way to do it is four (4) tapes labeled: Tape1, Tape2,
Tape3, Tape4. These are used either everyday or at the end of the week
as above, with the previous tape being taken off-site.
Needless to say, the above barely even scratches the surface of the
possible options. If the number of tapes is not an issue, all sorts of
other plans will work well. I have found the above plans to work well
for me, in my environment over the last several years–your mileage may
vary.
My Solution
My solution in this case is to use eight (8) tapes labeled: Week1,
Week2, Week3, Month1, Month2, Month3, Month4, Month5. The weekly tapes
are used once a week, early Monday morning (note these tapes will need
to be replaced most often, as they will get the most use). Month1 is
used at the end of the first month, and so on. Either the previous week
or the previous month tape is moved off-site. The Monday backups are
“full” backups of the dynamic data, the monthly tapes are complete
system backups (with excepts for junk). This strategy also gives you at
least a 4 month window to recover different versions.
Now a problem crops up because it works out that some months have five
Mondays in them. There are a couple of ways to solve that problem, but
I took the easiest–I ignored it. So periodically your “Monthly” tapes
will get out of sync with the last Monday of the month. Too bad.
My weekly backup set is the set of: /etc /home /root /var/log /var/named
/var/spool
My monthly backup set is the set of: /, minus the set of: /a /cd /fpy
/mnt /proc /tmp /var/tmp /zip
Interestingly, it turns turn that the space and time different between
my two sets is not very much. I could just use the slightly larger full
or monthly set for all tapes, but keep the rotation and other part of
the strategy the same. That came as a surprise to me. I expected there
to be much more difference. So you’ll just have to try it, and see how
you make out. I’m leaving it alone as I see no compelling reason to
change it.
I also use the Red Hat KickStart and mkbootdisk tools with my own RDISK
script. I have written a shell script that automates everything except
changing the tape, and I have a 7 day windows to remember to do that.
Part 3: Putting It All Together
OK, given everything above, let’s actually get into the details.
KickStart
KickStart is a Red Hat automated installer program. If you install and
then use the “mkkickstart” program, you can create an “answer file” that
allows you to automatically install everything exactly the same as you
just did. That, combined with your CD-ROM, allows for a pretty cool
recovery tool in case of disaster. Just replace the failed hardware (or
in most cases get close enough) and you’re set. See the RDISK script
below and
http://www.redhat.com/support/manuals/RHL-6.2-Manual/ref-guide/ch-
kickstart2.html for more details.
UPDATE: There is not a command line “mkkickstart” program in later
versions of Red Hat Linux.
mkbootdisk
You should also install and use the “mkbootdisk” command to make a
recovery disk that may be able to boot your system if something goes
wrong. it’s not a bad idea to keep two of these, and alternate using
them when you make changes.
RDISK
RDISK (the name comes from the similar facility on NT) uses mkkickstart
and mkbootdisk, fdisk, rpm and du to capture a lot of critical
information about your system, mostly your file system configuration.
You can write the data to a floppy, or not (in which case mkbootdisk
does not run).
configbackup
This simple script just copies important files someplace else. You can
copy them to a floppy, if they fit, or to another drive or server or
whatever. You can keep multiple versions if you want. How you
implement it is up to you. It is never used in any of the other scripts
here, and is included only as a convenience.
/root/updates
Another useful strategy is to create a /root/updates directory (or
whatever) and keep all the installed patches and updates in it. You do
updates your system as necessary, don’t you? If you need to use the
KickStart file, then restore, it’s amazingly easier to bring the system
back up to speed when you can go into /root/updates and basically do an
rpm -Fvh *.rpm. OK, it’s a little more complicated than that for some
updates such as the kernel, but that works 90% of the time. Also, this
directory doubles as a record of how your system differs from a “stock”
installation.
jpbackup
NOTE: the “tape” program below may actually be a lot easier to use, even
though (or maybe because) it has less options and automation. I use it
for ad hoc backups of file systems that change infrequently. It works
much better than I would have guessed, even though I wrote it. Thanks to
Robert Squire for the tip!
ALSO NOTE: this script is pretty buggy. It works for me, the way I use
it, but i do not recommend its use in a production environment. If
you do use it, test it thoroughly and make sure you understand exactly
what it’s doing.
jpbackup is the heart of the system. It pulls the other scripts
together and actually runs the show. It implements and automates the 3
weekly and 5 monthly tape scheme above, and under normal circumstances
(i.e. you do not have to do a restore) all you have to do is change the
tape between every Monday. You should really look at the logs as well.
In particular, I’ve added code that shows how long the backup took, and
how big the backup set is on disk, then how big it is on tape. Given
that information, you can tailor your compression settings to speed up
your backups if your tapes are big enough. It uses the rewinding
device, and pretty much forces you to have only one archive per tape.
Conceivably, this wastes tape, but is far more simple in many ways. It
keeps a catalog of what is on each tape, named “Monday_1.cat,”
“Monday_2.cat,” etc. The afio log file is also kept, with the same
name except .log. Finally, a backup.log is kept with start times, and
the data sizes.
Here is an outline of operation:
Set a bunch of variables used in the script.
Make sure data and flag files exist, and create them if needed.
Read the flag files and find out if we are doing a Weekly or a
Monthly job, and which one (e.g. #1-3 or #1-5).
Output a screen-full of operational information, just in case anyone
is watching and cares. (Oh yeah, it’s useful for troubleshooting
too.)
Start the log, then rewind (just in case) and erase the tape that’s
in there!
Find the data to backup, and write it two ways, one with files sizes
(to sum up amount of data on disk) and one without (used by afio).
Sum up file sizes of data on disk being backed up.
Use afio to actually run the backup, printing filenames, backup
status and compression ratio (if applicable) to the screen, just to
keep things interesting. Use the NoBackup file to identify date not
to backup and the NoCompress file identify data not to try to
compress.
Add some backup.log entries then cd / so the verify (using relative
paths) will work.
Run the verify, piping into grep to remove a minor output formatting
bug.
Sum up file sizes of data on that was backed up to tape.
Update the flag files, so if we barfed above, we don’t pretend it
actually worked.
Write the last log entries, including the file sizes.
Go back to sleep for a week.
Sample backup.log
Thu Feb 22 02:58:41 EST 2001; START: weekly backup to Monday_1...
Thu Feb 22 06:38:06 EST 2001; FINISH: weekly backup to Monday_1...
Thu Feb 22 06:38:06 EST 2001; START: Verify weekly Monday_1...
Data size on disk: 12069397386, size on tape (GZ 4) 4269799725.
Thu Feb 22 09:25:02 EST 2001; FINISH: Verify weekly Monday_1...
From this you can see that the backup took under 4 hours, that 12 Gig
was backed up, but using only Gzip compression level 4 (9 is best
compression/slowest, 6 is default) it took up only 4.2 Gig on tape. We
also see that the verify took just under three hours.
If you look at Monday_1.log, you’ll also see some verify errors such as
the following. That’s because some files CHANGED between when they got
backed up and when they got verified. This is normal! For example,
the backup.log file was updated by the backup itself, after it got
backed up. Thus it fails the verify since the disk version is different
than the tape version.
afio: "var/log/backup/backup.log": Archive data and file cannot be aligned
(disk 1) at Wed Feb 21 16:07:56 2001
afio: "var/log/backup/backup.log": Corrupt archive data (disk 1) at
Wed Feb 21 16:07:56 2001
afio
jpbackup requires afio, which is not a part of Red Hat’s default
install. There is a version in the Red Hat 7.1 Power Tools, but it’s
ancient. Just use
http://www.rpmfind.net/ and grab if. I’ve been
using afio-2.4.7-1mdk for forever.
See the end of the backup script for the options I used. See the afio
man page for all the options, of which there are a plethora.
tape
Tape is just a simple front end to keep all the tape related commands in
one place. Especially since afio has so many options, it’s a real pain
to remember and type them all. So you can edit tape to work on your
system, then pretty much just run it ad hoc if needed.
Restoring
The ability to restore or recover is the entire point of this exercise,
yet there is not all that much I can say about it. There are many
variables, but by now you should be getting a feel for them. The
following questions might help. Note that I am dealing only with
restoring from tape. Rebuilding the system, using KickStart, recovering
from hardware failures, etc. are all beyond the scope of this document.
Which tape (or tapes) do I need and where are they (on-site,
off-site)?
If there is more than one tape archive on the tape, which one do I
need? (If you used jpbackup and did not modify it, there is only
one.)
Is the tape archive compressed or not? (If you used jpbackup and
did not modify it, they are compressed.)
Do I want to restore everything, or just some files? Running a
table of contents (-t) might be useful if you do not have the
catalog file.
Do I want to restore to the same location, or to a different
location and then compare files? Do I have enough free space to do
that (df -h)?
To restore everything from a compressed tape archive, and to overwrite,
you need to be in the root ( / ) directory. To restore everything from
a compressed tape archive to a different directory, you need to be in
that directory. Then:
To restore just the “/root” directory from a compressed tape archive,
and to overwrite, you need to be in the root ( / ) directory. To
restore everything from a compressed tape archive to a different
directory, you need to be in that directory. It can even handle the
leading / in the path (even with the use of relative paths)! Then:
Backup is the script that actually backs up my system. it’s called from
cron every Monday, and it figures out what kind of backup (weekly or
monthly) to do by itself.
A generic front-end, so you don’t have to remember the block size and
other options.
NOTE: this may actually be a lot easier to use than jpbackup, even
though (or maybe because) it has less options and automation. I use it
for ad hoc backups of file systems that change infrequently. It works
much better than I would have guessed, even though I wrote it. Thanks to
Robert Squire for the tip!
This is never used in any of the other scripts here, and is included
only as a convenience. It copies various important files to some
specified backup location
Obsolete Content
This content is obsolete, but I am leaving it here as a historical reference.
Appendixes
Some Notes About Common Backup Programs
See the following URLs for lists of Linux backup tools. Some of these
tools are free, some are commercial, and some are in-between. I’m only
going to talk about free ones.
The list below is of tools I’ve found to out there and interesting
looking. I use afio, but I’m not making any recommendations that you
should too. Just look at the list. I’ve quoted
http://www.linux.org/apps/all/Administration/Backup.html and the
home pages of some of the tools quite a bit (read, I stole their
descriptions). That text remains the property of its respective owner.
One interesting issue is that of relative paths. Older UNIX tar
commands stored absolute paths (e.g. / home/user/mystuff by default.
This is bad, because you can only restore to the exact same directory
you backed up from. You may not always want to do that. Most GNU tools
use relative paths (home/user/mystuff) so you can restore wherever you
want. The downside is that unless you are in the root of the filesystem
when you do a verify, it will fail, because it will use a relative path
to find the files to compare with the tape and it won’t find them. For
example, if you are in /home/user, trying to verify a backup of your
home directory, the tape software will be looking for
/home/user/home/user, which is probably not there. The moral of the
story is, cd / before doing a verify.
The same goes for restores, except you might actually want to do
this. There are often time when I need to restore /home/user, but I do
not want to actually mess with /home/user, I just want a part of
it. One solution is to do a partial restore. The other is to restore
to the relative path, get what you need, then nuke the rest. Remember,
this is only with the newer, usually GNU versions. The “traditional”
tar does not work this way.
All of the examples below assume only one archive (file) per tape.
While this can be construed as wasting tape, it’s a heck of a lot
simpler to manage! (See the discussion in
http://www2.linuxjournal.com/lj-issues/issue22/1216.html.)
I have only included commands for some tools. If you have the
commands for others, send them to me and I’ll include them and give
you credit (and/or blame :-).
Tar is the most widely known UNIX backup tool. It stands for Tape
ARchive and does not have to actually use tape. You have almost
certainly seen a .tar, .tar.Z or .tgz file. These all use tar. It has
some problems though. Most notably, IMHO, it compresses the entire
archive, so if tape is damaged, entire archive lost. That’s a bit of a
problem. So I don’t like tar, but you pretty much have to know about it
anyway.
Operation
Command
Comments
Full Backup
tar cvb 64 -f /dev/tape
Full Restore
tar xvb 64 -f /dev/tape
Partial Backup
tar cvb 64 -f /dev/tape {directories}
See the man page about how tar deals with directory selection (note I did not say file selection).
Partial Restore
tar xvb 64 -f /dev/tape {directories}
Ditto.
Verify
tar dvb 64 -f /dev/tape
This will fail unless you are in the root directory - relative paths.
Table of Contents
tar tvb 64 -f /dev/tape
You must use the correct block size (-b 64) or you get all kind of
bazaar errors such as:
ide-tape: ht0: I/O error, pc = 2b, key = 2, asc = 4, ascq = 1
ide-tape: Reached idetape_chrdev_open
ide-tape: ht0: chrdev_write: use 32768 bytes as block size (10240
used) ide-tape: Reached idetape_chrdev_open
ide-tape: ht0: skipping frame 21, frame type 8
ide-tape: ht0: skipping frame 21, frame type 8
Next to tar, dump is another of the most widely known tools. As far as
I know, it does not do compression at all. It uses “levels” from 0 to 9
to determine what to backup. You can create very complex and convoluted
schemes to backup different things at different times. As I said above,
thinking about this stuff gives me a headache.
“The dump package contains both dump and restore. Dump examines files
in a filesystem, determines which ones need to be backed up, and
copies those files to a specified disk, tape or other storage medium.
The restore command performs the inverse function of dump; it can
restore a full backup of a filesystem. Subsequent incremental backups
can then be layered on top of the full backup. Single files and
directory subtrees may also be restored from full or partial backups.”
cpio is that last of the big three most widely known UNIX backup tools.
it’s interface is a bit different than tar or dump, in that it must be
used as a filter (e.g. find / -print | cpio -ov --block-size=64 -C 32768 \>/dev/ht0). It also suffers from the
same compression issues as tar.
“Afio makes cpio-format archives. It deals somewhat gracefully with
input data corruption, supports multi-volume archives during
interactive operation, and can make compressed archives that are much
safer than compressed tar or cpio archives. Afio is best used as an
`archive engine’ in a backup script.”
I like afio a lot. It works well with the DI-30, and I can script it to
just exactly what I want. It is used as a filter, the same as cpio, and
in fact uses the cpio format (as do RPMs). See my scripts in the
appendix.
The following examples are all very simple, and use gzip compression.
Unlike tar or cpio, afio compresses each file, rather than the entire
archive. That means if you have a media error, only the data where the
error is are lost, instead of the entire archive.
“Star is able to make backups with more than 12MB/s if the disk and
tape drive support such a speed. This is more than double the speed
that ufsdump will get. Star performs 13.5 MB/s with a recent DLT tape
drive while ufsdump gets a maximum speed of about 6MB/s with the same
hardware. Star development started 1982, development is still in
progress although it is stable to use.”
“Taper is a tape backup and restore program that provides a friendly
user interface to allow backing/restoring files to a tape drive.
Alternatively, files can be backed up to hard disk files. Selecting
files for backup and restore is very similar to the Midnight Commander
interface and allows easy traversal of directories. Recursively
selected directories are supported. Incremental backup and automatic
most recent restore are defaults settings. SCSI, ftape, zftape, and
removable drives are supported.”
Note the last line. Taper was developed for ftape (floppy tapes, like
the QIC series drives). It is not recommended for use with OnStream
drives.
I have feedback from two different people that taper works fine:
Date: Mon, 29 Apr 2002 16:03:30 +0200
From: Siegfried Heim
Subject: DI-30 mini-howto
Dear JP,
In your mini-howto for DI-30 tape drive backup you liked to know, whether
taper works with this streamer.
I tested the DI-30 drive using taper 6.9a for my backup. So far it seems
to work well with the following settings:
rewinding device: /dev/osst0 (non-rewinding: /dev/nosst0)
block size: 32k
tapesize (in MB): 15000
I'm using the 2.4-18 Kernel that came with SuSE 8.0 Professional
Distribution. It has built-in support for OnStream tape drives (uses
ide-scsi emulation).
Greetings from Germany
-Siegfried Heim-
Date: Mon, 26 Nov 2001 19:08:04 +0100
From: Freerk J.
Subject: About Taper
I updated Linux to 2.4.2-2. [Which] contains complete installation for
Onstream DI-30. It is clearly visible during startup and also can it be
found in /proc/ide. I also discovered that taper 6.9b was automatically
installed. Just startup with taper -T ide, [but] you have to change the
block size in the menu: Change Preference, tape drive Preferences, Block
size is default on 28k. With arrow keys to change to 032K and it works!
Momentarily testing a restore procedure........ That is OK too.
I also have feedback that taper does not support backups larger than 4
Gig:
Date: Mon, 7 Apr 2003 12:41:50 -0400 (EDT)
From: JP Vossen
To: Cor van den Berghe
Subject: Re: Can you explain someting to me?
On Mon, 7 Apr 2003, Cor van den Berghe wrote:
> After reading the OnStream DL30 Backup mini-HOWTO on you're website I was
> wondering if you could help me out with someting. I've been using an
> OnStream DI30 (osst drivers) and Taper on a RedHat 8 system with no
> problems, at least thats what I thought. A couple of weeks ago I tried to
> restore something and Taper told me that the tape was corrupted. When I
> looked on the Taper Homepage I found out that Taper does'nt support
> backups > 4 Gb [...]
“KBackup is a backup program for UNIX machines. It supports any OS
supported tape drive. It can use tar or afio to create the archives.
It can even compress using gzip. It supports include lists, exclude
lists, and even backing up to a file.
“KBackup is an easy-to-use backup package for Unix. It was originally
written by Karsten Balluder. Currently, its development has stagnated,
and several fixes are needed. The main mailing-list for KBackup is in
egroups (
www.egroups.com).”
OK, I lied. I said I would only talk about free programs, and BRU is
not free. But it’s one of the most popular backup system for small
Linux systems, so…
“BRU Backup & Restore Utility features data-verified backups,
scalability, configurability, and ease of use, for functionality with
Linux and UNIX.”
Please note that you MUST use a 32k block size when writing to the DI30
drive. Also note that the tar statement uses “-b 64” due to its 512
block size e.g. bru -cvvf /dev/ht0 -b 32k /home. Get a complete BRU configuration file for this drive from
http://www.estinc.com/downloads/brutabs/adr.bt
Hints from OnStream Tech Support
The DI-30 cannot programmatically eject tape (i.e.
mt offline doesn’t work) when using the IDE
interface. It does work when using OSST/SCSI.
Using tar, you may get a message at end of full backup from “/” –
too many errors. You may ignore it.
ALWAYS use the 32k block size, even for ToC, etc. (This is not
strictly necessary with the OSST interface, but it does not seem to
hurt. –JP)
Don’t slave to IDE hard drive, make master or slave to IDE CD-ROM.
A DI-30 tape is about 12,000 feet long.
Web References
Obsolete Content
This content is obsolete, but I am leaving it here as a historical reference.
The following used to be references to useful material on the Web, but most
have probably rotted away. Just in case, all the
various links I’ve used above are here again, along with a bunch of
other neat material. The manual (man) pages are provided in case you do
not have access to a Linux machine to get the details, and because they
are easier to read and print out.
Also, see the links in the “Update (2001-11-29)” section in the introduction.
Date: Wed, 12 Dec 2001 06:51:22 -0500
From: Willem Riede
To: JP Vossen
Subject: Re: FW: mt status question
On 2001.12.12 00:45 JP Vossen wrote:
>
[snip]
> BTW, I still have the old 32K block size hard coded in my program. Does it
> matter? Could that have any effect on all the errors ("soft" errors?) I get?
>
No. block size is your choice to make. The driver (osst) handles all
(un)packing of frame content in memory. Only entire frames go to the
tape. Some frames have the ill fortune of meeting questional media,
but that's totally independent of their content or how that content
was constructed. The great thing about the ADR format is that most
tape errors can be handled transparently and your data survives.
Regards. Willem Riede
IDE Configuration Jumpers
Date: Sun, 1 Sep 2002 13:19:48 +0200
From: Denis Faivre
Subject: DI-30 Howto
Hi,
I just bought a DI30 and noticed that the indication engraved on the
metallic case regarding Master/Slave/Cable jumpers is wrong. The right
indication is that of the paper documentation.
[CSM] [: : : : : : : : : : : : : : : : : : : :][o o o o]
Maybe would it be useful to include this information into your HOWTO, or
at least warn the reader about a possible confusion...
Media Errors
Date: Wed, 7 Nov 2001 11:35:23 +0100
From: Bombeeck, Jack
Subject: RE: Beta test for ADR2.60ide?
To get to your suspected media problems: one issue that repeatedly comes
up is temperature related problems. They obviously show up as media
problems, but not because of bad media, just because of working outside
the operating range. To make sure that this is not bugging you, remove the
drive's door and if need be make sure that at least one fan blows onto the
back of the drive to produce an air flow. The latter is sometimes simply
achieved by choosing the drive position in the cabinet carefully;
otherwise you might add a fan. When the cartridge has been in the drive
for a while (and been used), it should still feel cool to the touch when
removed. If not, you run the risk of the above-mentioned problem, which
results in write errors (not usually a problem since blocks are relocated
until successfully written) and erratic unrecovered read errors (bad news,
data's irretrievable!). Let me know how you fare.
Minor updates, and changed document revision and date to the CVS tags.
v1.3.0
2003-30-11
Converted to simple HTML as opposed to the insane drivel that MS Word generates. Minor corrections and additions. Major updates to links since OnStream is bankrupt and gone again.
v1.2.0
2002-05-03
Added user feedback, made correction, etc.
v1.0.0
2001-11-24
First general public release.
v0.9.3
2001-10-29
Updated some links and added comments/information from Jack, an OnStream Software Development Manager in the Netherlands. Also added a link to the new ADR2 drive.
v0.9.2
2001-06-16
Corrected a bug with all tar examples. Was “tar -tvbf 64 /dev…” but should have been “tar tvb 64 -f /dev.” Also, changed “
www.onstream.com” to
www.onstreamdata.com. Thanks to David Burleigh for pointing those out. Also other minor corrects to docs.
v0.9.1
2001-96-02
Minor corrections for typos, etc. The script itself needs work, and I need to do more testing with Red Hat 7.1 before the “public” release.
v0.9
2001-02-27
DRAFT: First public release, so various technical reviewers can access it.
Obsolete Content
This content is obsolete, but I am leaving it here as a historical reference.
This is a list of free entertaining yet educational software for your
kids and computer, and is intended for a general (read
non-computer-geek) audience I start in 2008. Most of these programs are
available for the three most common operating systems, Windows, Max OS X
and GNU/Linux (e.g.,
Ubuntu,
Debian). Everything listed here is at least
available for
Ubuntu, and much of it is
“built-in.” For Ubuntu/Debian, install these via “add/remove programs”
or via “sudo apt-get install <package name(s) here>” from the
command line (e.g.: sudo apt-get install stellarium stellarium-data).
This is just a tiny sample of the free educational and kid-related
software available. Google is your friend.
A Word About “Free”
“Free” can mean many things, especially in the context of software. The
argument is usually simplified as, “free as in beer or free as in
speech.” That is, some software is free of cost, but does not allow
modification. Other software may not only allow but encourage you to
take it, modify it, give it away, or whatever. In-depth discussion of
this issue, or why people choose to “give away” their work is out of the
scope of this document. Google for “free and open source” to learn far
more than you want to know about it. (In particular you can see
this
long discussion.)
In scope, all of the software listed here is completely free (without
cost) to use on your computer, and almost all of it allows the freedom
to do just about anything you want with it. Check the individual web
sites for licensing details if you are not sure. And note that however
you define “free” does not preclude a license to which you must agree,
though most times that license is simply to guarantee the aforementioned
freedoms. This is sometimes called a “copyleft” (as opposed to a
copyright), see
http://www.gnu.org/copyleft/gpl.html for details.
A Word about Operating Systems and Ubuntu (& Linux Mint)
Update: Use
Linux Mint! It’s built on top of
Ubuntu, but it’s better. From a user interface perspective it looks, feels,
and work just like Windows. Some may argue that’s a bug, but I think it’s
a big feature, because pretty much everyone (in the US at least) has Windows
inflicted on them at some point, so it’s familiar. A single place to (wait
for it) “start” is a good thing, and a logical order in such a menu,
especially when it’s searchable is discoverable, easy to explain, and easy
to use for beginners and experts alike.
Ubuntu is a
“
distribution” of the
GNU/Linux “
operating
system” and is an
alternative to paying Microsoft (and/or your computer dealer) lots of
money to run Windows, then paying lots of other folks for all the
anti-virus, anti-malware, etc. software required to protect Windows from
itself. This not only wastes a lot of time and money, but the overhead
of these programs make your brand-new computer run like a 486. And we’re
not even going to talk about the Vista, Windows 8, Windows 11 disasters,
and the arbitrary and unnecessary hardware requirements that “prevent”
“upgrading” (using the term very loosely) Windows 10 to Windows 11.
If
Ubuntu
looks too different for you, you can run a different spin like the
vaguely Mac-like
Xubuntu or more
the Windows-like
Lubuntu or
Linux Mint (which is based on
Ubuntu anyway, see above).
The problem with Ubuntu is that Canonical is starting down the road to
enshitification and doing
too many things that smell like Microsoft. Just use Linux Mint and be happy.
As for Apple, they make nice-looking (but expensive) hardware that works
well if you choose to do things exactly the way they want you to, and if
you accept the associated loss of privacy, control over your own device
and your own contents and their censorship. (See
details.)
So take an old PC that is either too old or too
malware infested to run Windows
anymore, download Ubuntu or Mint (for free), and try it. It isn’t
perfect, but it is constantly improving. It is not susceptible to the
vast amounts of Windows malware out there, so it’s great for kids. But
on the other hand, it doesn’t run programs written only for Windows
(well, actually it does, using
Wine, but that’s
getting out-of-scope here), so custom programs for school may not work.
As you’ll see if the Windows/Mac-only program won’t run on Ubuntu, there
is almost certainly an alternative, which is almost always free and
often (but not always) better than the Windows/Mac program it replaces.
In particular, LibreOffice (sort-of used to be OpenOffice.org, but you
don’t actually care about the details) is a free replacement for MS
Office that is improving all the time. It can trivially “File > Export
as PDF” which is very handy and can read and write all versions of MS
Office documents, though it’s not always perfect (though MS Office isn’t
always that great between versions of itself either). And importantly,
it looks like the “old” versions of MS Office, not like the totally new
Office interface that will require a lot of re-learning things you used
to know how to do.
Gramps is a free software project and community. We strive to produce a genealogy program that is both intuitive for hobbyists and feature-complete for professional genealogists. It is a community project, created, developed and governed by genealogists.
This content is old! It’s still useful, but it’s old, and there may be bit rot, newer/better tools or ways to do things. Sanity check and do your research.
This content is old! It’s still useful, but it’s old, and there may be bit rot, newer/better tools or ways to do things. Sanity check and do your research.
An IP (Internet Protocol) Address is a 32-bit number broken up into
“quads” of 1 byte each, separated by dots. 1 byte is 8 bits which in
decimal is a number in the range 0 to 255. For example, 10.234.56.71 is
an IP Address. There are only so many “real” IP addresses, and they are
(and have been) perpetually very close to being used up and thus are
very difficult to get.
One of the solutions to this problem is so-called “private” IP
Addresses. These are ranges of IP Addresses set aside expressly for use
by a company or other entity internally. Private IP Addresses cannot be
used to connect directly to the Internet–that is they are non-routable.
These are also often called
RFC1918 addresses.
Use
You use a Private IP Address when you wish to use TCP/IP on your LAN,
but do not wish to try and register enough legal or legitimate addresses
for all your devices. Even if you do wish to get than many, you will
not. Essentially all valid IP addresses are already owned, either by
very large corporations (like AT&T) or by ISPs. When you contract for
service from an ISP, you are allocated some number of legitimate IP
Addresses out of that ISP’s pool of addresses.
Advantages
Increased security (since private IP addresses are not routable
across the Internet).
You conserve the world-wide pool of IP Addresses.
You do not have to register or pay for these IP Addresses in any way
(internal independence from ISP IP addresses).
When you connect to the Internet via a Firewall and NAT (Network
Address Translation, AKA IP Masquerading) you will not block any
address ranges from yourself.
Little or no performance degradation (depending on your Firewall).
If you merge with a company that has chosen the same Private IP
Address, one or both of you will have to re-number. This can be
difficult and expensive.
Some applications don’t work with NAT.
Anything using NBT (UDP 138), i.e. NT Networking cannot communicate
behind a Firewall with NAT. See below for the reason.
Some applications needing encryption and key exchange (specifically
any application that embeds IPs in the datastream) may not work with
NAT.
Network Address Translation (NAT) AKA IP Masquerading
NAT, AKA IP Masquerading, is the process by which a “private,”
“illegal,” and non-routable IP Address is translated into a “legal,”
routable address. There are two kinds of NAT, often called static NAT
and Hide NAT. Static NAT provides a one to one correlation between the
illegal private address and the legal routable one. For example, the Web
Server on 192.168.1.10 may be statically mapped to 39.136.195.47. Hide
NAT is a many to one arrangement where the many illegal addresses behind
some device appear to the Internet as one single address (often the
legal address of the device itself). For example, the entire 172.25.1.0
network may hide behind the single valid IP address of the device at
38.111.56.96.
NAT Devices
There are three devices that typically perform NAT. They are routers,
firewalls and proxy servers.
Hide Mode NAT
In hide mode, the external address of the NAT device “hides” most or all
outgoing connections. To the Internet, it seems that all traffic
originates from this single address, when it really comes from all
different machines on the internal network. The traffic is
differentiated at the NAT device by a table of port numbers. For
example, the port used for Web Surfing is port 80 (http). If a client
computer at 192.168.1.37 surfs to
www.dell.com, the NAT device may assign that
to port 20,134. When the response comes back, the firewall knows that
anything directed to port 20,134 really goes to the client at
192.168.1.37. That way, more than one person can surf at the same time,
using the same external IP address, but everything goes to the correct
person.
Static Mode NAT
In static mode, there is a one to one correlation between internal
(illegal, non-routable) and external (legal, routable) addresses. The
must be the case if you wish top have an E-Mail server, Web server
or any other service that is accessible from the Internet. DNS (Domain
Name Service) published the IP Addresses of server (or services) that
are accessible. These published addresses must be legal, and routable.
The IP network of addresses available for this use is termed the “moat”
network, below. A typical “moat” network looks like this:
IP Address
Description
209.146.2.40
Network Name
209.146.2.41
Available IP Address (usually assigned to the internal router interface)
209.146.2.42
Available IP Address (usually assigned to the external firewall interface)
209.146.2.43
Available IP Address (may be Web server?)
209.146.2.44
Available IP Address (may be E-Mail server?)
209.146.2.45
Available IP Address
209.146.2.46
Available IP Address
209.146.2.47
Broadcast Address
A very interesting thing happens with static NAT, however. Since the
router is at IP address 209.146.2.41, when it sees a packet destined for
209.146.2.43, it “arps” for the Web server. Since the router knows that
it is on network 209.146.2.40/29 and the Web server address is
209.146.2.43 they should be on the same network. But they really aren’t.
So when the router “arps” (uses the Address Resolution Protocol to find
the Web server), the Web server will not answer, since it is really
on network 192.168.1.0/24. To solve this problem, devices that perform
static NAT also perform “proxy arp”.
Any device configured to do static NAT has a list of servers it will
“answer for” when it hears an arp request. IT will essentially lie and
say, “yes, I am that server, please send me the packet.” When it get the
packet, it forwards it to the real server.
A Typical Internet Connection Scenario
A very common small business-class (as opposed to home use) Internet
connection looks like this following:
Figure 1: Common Firewalled Network Diagram–With Router
Figure 2: Common Firewalled Network Diagram–With Bridge
Description
Network
IP Range
Company LAN
192.168.1.0/24
192.168.1.1 to 192.168.1.254
Service Network (DMZ)
192.168.200.1/24
192.168.200.1 to 192.168.200.254
Moat
10.146.2.40/29
10.146.2.41 to 10.146.2.46
Link Network
10.146.37.28/30
10.146.37.29 to 10.146.37.30
Network or Device
Default Gateway
Company LAN
192.168.1.1
Service Network (DMZ)
192.200.1.1
Firewall
10.146.2.41
ISP Router
10.146.37.29
The Company LAN uses the private (RFC1918) address of
192.168.1.0/24.
There is a “Service Network” (AKA DMZ) for hosting Web Servers, FTP
Servers, extranet (partner) connections, etc.
The Firewall is performing both hide NAT and Static NAT.
Hide NAT is that all outgoing connections from the 192.168.1.0
network are hidden behind the firewall’s address of 10.146.2.42.
Static NAT is that the E-Mail server on the company LAN has a
“routable,” external IP address of 10.146.2.44, but an internal
IP Address of 192.168.1.15.
The “Moat” network is the network between the external interface of
the firewall and the internal interface of the router.
There is confusion about the term DMZ. Originally, the term DMZ was
used to denote the “moat” network. Recently, however, the common
usage has been that the DMZ is the “Service Network”. I have used
“Service Network” and “Moat Network” to avoid confusion. The term
“Moat Network” is not in common usage, however.
Service
Internal Address
External Address
NAT Mode
Hide NAT
192.168.1.0/24
209.146.2.42
Hide
E-Mail Server
192.168.1.15
209.146.2.44
Static
Web Server
192.168.1.10
209.146.2.43
Static
Appendixes
Subnet Masks: Decimal and CIDR
CIDR
Decimal Mask
Old
A Subnets
B Subnets
C Subnets
# Useable
# Hosts
8
255.0.0.0
A
1
16,777,214
16,777,216
9
255.128.0.0
A
2
8,388,606
8,388,608
10
255.192.0.0
A
4
4,194,302
4,194,304
11
255.224.0.0
A
8
2,097,150
2,097,152
12
255.240.0.0
A
16
1,048,574
1,048,576
13
255.248.0.0
A
32
524,286
524,288
14
255.252.0.0
A
64
262,142
262,144
15
255.254.0.0
A
128
131,070
131,072
16
255.255.0.0
B
256
1
65,534
65,536
17
255.255.128.0
B
512
2
32,766
32,768
18
255.255.192.0
B
1,024
4
16,382
16,384
19
255.255.224.0
B
2,048
8
8,190
8,192
20
255.255.240.0
B
4,096
16
4,094
4,096
21
255.255.248.0
B
8,192
32
2,046
2,048
22
255.255.252.0
B
16,384
64
1,022
1,024
23
255.255.254.0
B
32,768
128
510
512
24
255.255.255.0
C
65,536
256
1
254
256
25
255.255.255.128
C
131,072
512
2
126
128
26
255.255.255.192
C
262,144
1,024
4
62
64
27
255.255.255.224
C
524,288
2,048
8
30
32
28
255.255.255.240
C
1,048,576
4,096
16
14
16
29
255.255.255.248
C
2,097,152
8,192
32
6
8
30
255.255.255.252
C
4,194,304
16,384
64
2
4
31
255.255.255.254
C
N/A
N/A
N/A
N/A
N/A
32
255.255.255.255
C
BC
BC
BC
Broadcast
Broadcast
Notes:
The “# Useable” series can be derived by “previous # Useable x 2 +
2”.
The “# Useable” series can be derived by “# Hosts - 2”.
The “# Hosts” series can be derived by “previous # Hosts * 2”.
The “# Hosts” series can be derived by “# Useable + 2”.
The number of subnets is only correct under CIDR. Using the old
classfull numbers it is “# CIDR Subnets - 2”.
3. Private Address Space
The Internet Assigned Numbers Authority (IANA) has reserved the
following three blocks of the IP address space for private internets:
10.0.0.0 - 10.255.255.255 (10/8 prefix)
172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
192.168.0.0 - 192.168.255.255 (192.168/16 prefix)
We will refer to the first block as "24-bit block", the second as
"20-bit block", and to the third as "16-bit" block. Note that (in
pre-CIDR notation) the first block is nothing but a single class A
network number, while the second block is a set of 16 contiguous
class B network numbers, and third block is a set of 256 contiguous
class C network numbers.
An enterprise that decides to use IP addresses out of the address
space defined in this document can do so without any coordination
with IANA or an Internet registry. The address space can thus be used
by many enterprises. Addresses within this private address space will
only be unique within the enterprise, or the set of enterprises which
choose to cooperate over this space so they may communicate with each
other in their own private internet.
This content is old! It’s still useful, but it’s old, and there may be bit rot, newer/better tools or ways to do things. Sanity check and do your research.
Daylight Saving Time Switch
In 2007, the US and Canada, as well as others, changed the rules for when
Daylight Saving Time begins and ends. In my opinion this is sheer idiocy,
but then, they didn’t ask me.
Anyway, it really screwed up quite a lot of things, many of
which are not in our power to easily fix (GPS, car GPS, VCRs, embedded
systems like cheap routers, and much more). Here are some resources:
ISO/IEC 17799:2000(E) (AKA BS7799),
clause 9.7.3 specifies “Clock synchronization:”
“[…]Where a computer or communications device has the capability
to operate a real-time clock, it should be set to an agreed
standard, e.g. Universal Coordinated Time (UCT) or local standard
time. As some clocks are known to drift with time, there should be a
procedure that checks for and corrects any significant variation.”
Event Logging, Auditing or Intrusion Detection across different
systems becomes very difficult.
Many cryptographic functions, especially those involving key
creation, exchange and expiration, as well as “ticketing” functions
such as used by Kerberos require precise time synchronization.
Event or program scheduling may not work as expected.
Client/Server transactions may not work as expected (transaction
precedence is incorrect).
There may be legal issues when submitting logs or other material as
evidence if the time is not known to be
correct. 1
Security certificates, WWW Cookies, DHCP and WINS leases may not
work as expected.
High Availability or clustering solutions may depend on members
clocks being exactly synchronized.
File creation and access times will be wrong across different
computers, thus:
Differential, Incremental or other backups may not work as
expected.
Revision control systems (such as CVS) may not work properly.
E-Mail Message time stamps may be wrong, leading to unexpected
transmission issues.
NetWare NDS will not work right unless all NDS servers have the same
time.
Neither will Active Directory, even though it says it will. If you
have an object collision (two objects are modified at the same time
by different people on different masters) the time stamp is used to
help resolve the conflict. If time is not synchronized, the results
will not be as expected.
For more uses or requirements for time synchronization, search the RFCs
for other RFCs that specify the use of NTP.
Why is UTC used as the acronym for Coordinated Universal Time
instead of
CUT?
In 1970 the Coordinated Universal Time system was devised by an
international advisory group of technical experts within the
International Telecommunication Union (ITU). The ITU felt it was
best to designate a single abbreviation for use in all languages in
order to minimize confusion. Since unanimous agreement could not be
achieved on using either the English word order, CUT, or the French
word order, TUC, the acronym UTC was chosen as a compromise.
Which is correct, UTC or GMT? Does GMT have summer time?
From http://wwp.greenwichmeantime.com/home.htm
During the Summer the UK is on British Summer Time which is 1 hour
ahead of GMT (GMT+1).
[…]
GMT is fixed all year and does not switch to daylight savings
time.
[…]
Although GMT has been replaced by atomic time (UTC) it is still
widely regarded as the correct time for every international time
zone. Greenwich Mean Time is international time, the basis of the
world time clock. Marks precision time and military time (sometimes
called Zulu Time).
[…]
The following articles from Sun are an excellent introduction to NTP
concepts and implementation, while from a Solaris perspective, they
are probably useful to anyone interested in time synchronization.
A list of Windows Time Sync Tools (in Excel format,
originally researched by me in 1999 and 2000, last updated May 2003
by Jason Mathews <mathews@mitre.org>)
By far the best NTP client and server that I found is
Tardis. It runs as an NTP client
and NTP server. There is one version that is an NT4/Win2K service,
and another that runs as a normal application under Win9x/ME. It has
only one minor problem – it’s not free. It runs from $20 to $2,000
USD, see the web site
for details.
The next best is the “official”
NTP package from David Mills.
Only the source is available from the Time Server site, but compiled
binaries for Windows are available.
There is also a Windows NT port
Maintained by Terje Mathisen.
NTP for Windows NT/2000/XP/2003/Vista is an NTP for
Windows binary port and
installer, along with a cool monitoring
GUI.
If you own the NT Resource
Kit
(or a newer ResKit), you should look into the NTRK TimeServ utility.
While you can only get the TimeServ tool from the NTRK, you can find
more information about it, and time in general at Doug Hogarth’s
Niceties site, specifically
the TimeServ page. Also
see other built-in Windows options below.
Tom Horsley’s NTP
Time for Windows is
a nice NTP client program. It is free, but is a client only, and can
be configured to talk to only one NTP server at a time. NTP works
much better when referencing a pool of servers.
I have also used the very cool
AboutTime
program, which is a Daytime/TCP, Time/TCP, Time/UDP and SNTP client
and server. It runs under any Windows version, but does not run as a
service. It is free!
I’ve used
Dimension4, which
is a free client for Time and NTP.
There are also Tardis/K9, which
are excellent shareware tools.
And for a minimalistic web site but neat sounding tool, see Graham
Mainwaring’s NetTime, at
Sourceforge.
The Microsoft Networking “Net Time” Command
Displays the time on or synchronizes your computer’s clock with the
shared clock on a Microsoft Windows for Workgroups, Windows NT, Windows
95, or NetWare time server.
NET TIME [\\computer | /WORKGROUP:wgname] [/SET] [/YES]
computer Specifies the name of the computer (time
server) whose time you want to check or
synchronize your computer's clock with.
/WORKGROUP Specifies that you want to use the clock on a
computer (time server) in another workgroup.
wgname Specifies the name of the workgroup containing
a computer whose clock you want to check or
synchronize your computer's clock with. If
there are multiple time servers in that
workgroup, NET TIME uses the first one it
finds.
/SET Synchronizes your computer's clock with the
clock on the computer or workgroup you
specify.
/YES Carries out the NET TIME command without
first prompting you to provide information or
confirm actions.
For example, if your PCD is named MYPDC, the following command in a
shortcut in your Startup Group, or in a logon script will synchronize a
client PC’s time at logon. Note, if your clients never log off, this
will not work. Of course, that’s very bad for other reasons. This works
for any system that runs Microsoft Networking. You can even sync again a
Linux server running Samba with this command,
if you’d like!
“net time \\MYPDC /set/ yes”
Win2K NTP Time Service
Win2K has a very simple SNTP facility built in: “net time /setsntp[:NTP
server list]”. See the following for more information:
Here is the batch file I use, since I find the documentation lacking and
the usage statement obscure:
@echo off
REM Win2k-SNTP.bat -- Set Win2K SNTP serviceREM v1.0 25-May-2001 JP Vossen JPATjpsdomainDOTorgREM v1.1 22-Jun-2001 JPV Changed to use home NTP time source onlyrem NOTE: The Win2K "Windows Time" service is manual by default, so you have torem set it to automatic and start it. Also, it will attempt to use all specifiedrem time sources and get an "average" so only specify servers that will berem available at all times. Do not use the list as a set of sequentialrem "failover" servers (as I did in v1.0 of this).rem NET TIME /SETSNTP:"192.168.1.11 172.16.1.1 10.1.1. 10.1.1.2"NET TIME /SETSNTP:"192.168.1.11"
You can see how it’s currently set by using this command: “net time
/querysntp” which will return something like:
The current SNTP value is: 192.168.1.11
The command completed successfully.
NTP Tools for Netware
For a small network with one or more NetWare servers, but no WAN links or remote sites:
Set one NetWare server as the REFERENCE server. This server will
sync itself to the NTP time.
Load MONITOR.NLM | Server Parameters | Time, change the
following parameters
Default Time Server Type = REFERENCE
Time Server Type = REFERENCE
TIMESYNC Configured Sources = ON
TIMESYNC Time Sources = <At least one PRIMARY
server>;<2-4 NTP time sources>:123;
Set one other NetWare server at the main site, and one NetWare
server at each remote site as a PRIMARY server.
Load MONITOR.NLM | Server Parameters | Time, change the
following parameters
Default Time Server Type = PRIMARY
Time Server Type = PRIMARY
TIMESYNC Configured Sources = ON
TIMESYNC Time Sources = <Your REFERENCE server from step
2>;
Set all other NetWare servers as SECONDARY.
Load MONITOR.NLM | Server Parameters | Time, change the
following parameters
Default Time Server Type = SECONDARY
Time Server Type = SECONDARY
TIMESYNC Configured Sources = ON
TIMESYNC Time Sources = <The closest PRIMARY server from
step 3>;
Note: usually, the REFERENCE server does not ever change it’s own time,
it just serves the time. However, when using NTP, the REFERENCE server
will adjust its local clock to synchronize with the NTP time source. See
the middle of TID
10050215.
Periodically set calendar from an NTP server. Supported by 7000, 7200, 4500.
ntp server {insert your favorite NTP server here}
NTP server from which to get the time
NTP for Nokia IPSO
IPSO comes with xNTP, but you can’t change the default polling, which is
something like every minute and a half. Until Voyager is enhanced to
provide this capability, there is no easy way to do it. You could edit
the ntp.conf file, but Voyager will overwrite it at bootup.
A list of Windows Time Sync Tools (in Excel format,
originally researched by me in 1999 and 2000, last updated May 2003
by Jason Mathews <mathews@mitre.org>)
Returns the number of seconds since 00:00 (midnight) 1 January 1900 GMT, such that the time 1 is 2:00:01 am on 1 January 1900 GMT. No accounting for different time zones, daylight savings, etc. Very inaccurate.
Protocols marked with
(RFC1700) are listed, but
not defined in the RFC.
Protocols marked with GnatBox Admin Tool
are listed in the services list in the executable.
Old Content
This content is old! It’s still useful, but it’s old, and there may be bit rot, newer/better tools or ways to do things. Sanity check and do your research.
I’ve always wanted to swim with dolphins, so we took a short vacation to
Florida and went to Discovery Cove.
It was pretty neat, though as homogenized and orchestrated as you would expect
in the States. They have a fake coral reef you can snorkel around that has some
really big spotted rays, and a stingray pool containing about 50 (when
we were there) smaller rays that you can touch if you’re patient and
fast enough. There is also a very boring “tropical river” that runs
through an aviary. However, if you are alert you may find an underwater
window from the “river” into the “reef” that is neat.
The main point is the dolphins though. I think they had about 30 or so.
They were very protective of the dolphins, which is good of course, but
limits how much free-form fun you can have to, oh, about none. Still, it
was a very interesting experience to see them up close, to feel their
skin (like a wet, slick, latexy, smooth velvet) and be taken for a short
ride. In that 30 feet or so, I was almost washed off by rushing water!
We met Diego, who was about 5 years old. He was born in captivity in
California (the San Diego zoo, I believe, hence his name).
Interestingly, he likes ice cubes quite a lot, of all things. Maybe it
was because they were fish flavored, since there were small food fish
mixed in to the can of ice the trainer had. I wonder what he thought of
all of this? Does he get bored, meeting humans and doing the same thing
every day? Is he just humoring the trainer when he feels like it; does
he think he is clever for conning fish and ice cubes all day long for
doing just about nothing? I’d love to know.
Each swim group has two trainers and a couple of photographers. They
have a pretty nice operation. The photographers have digital cameras and
after the half hour swim, they upload the memory sticks into a LAN. They
have a bunch of Mac’s sitting around in fake rocks and after a while you
tag all the pictures in your group with you in them, then go through a
slide show and pick out ones you want, if any. It’s insanely expensive,
of course, but what the heck…
But I couldn’t help wondering. Sand, water, Florida weather, clueless
tourists–NOT a good combination. I wonder how many mice and Macs
Tech Support goes through in a year… And the downtime figures on the
LAN must be really interesting.
I like the idea of recycling, and using recycled materials. I don’t like
the idea of maintenance. So a plastic deck is right up my alley. After
almost 2 years, it’s finally finished. The first year was spent fighting
with the town to get a variance, since we live on a corner. The second
year was spent building it. It sure looked a lot smaller on paper than
it does in real life!
I drew up the plans using an old copy of Sierra Home’s 3D Deck v3.0.
Since this is obsolete and not sold anymore, I got it for about $12 on
some liquidation Internet site. I also used an evaluation version of a
professional grade program to double-check everything, but I can’t
remember the name of that one.
Then I went to Home Depot and had them use their deck program to come up
with the materials list for the sub structure, which is pressure treated
(PT) wood. Unfortunately, their program can’t do 12" OC joists. We put
in everything, but specified no decking, railing, etc., then I took the
quote home and reworked it. I added the materials for the 12" OC joists
from their default 16". I also optimized the lengths and materials so
that my final order, with more material than the original, was $500
less, and I should have had almost no waste. They picked everything, and
delivered it on a truck a few days later. Nice! Of course, they were
out of the 2"x10"-10’ boards I ordered. So they gave me 2x10-12 for the
same price. Now have a pile of 2x10-2 waste… :-( I used a lot of that
for braces and parts in the frames for the stairs, but I still had a
bunch left over.
Getting the plastic was a bit more of a challenge. This is partly my
fault, since as a first time deck builder (and this is the largest
project like this I’ve done as well), I made a bunch of changes to the
order. If I’d had all my ducks in a row, I think everything would have
gone more smoothly. As it was, I ended up driving to the manufacture’s
site and helping to load the truck myself, because I was tired of
waiting for the material. This also seriously delayed the project into
summer, which resulted in even more delay, since I did not work when it
was too hot, or in direct sunlight.
My other complaint is with the stainless steel screws I used. I
stripped, torqued the heads off or bent at least 10% of them! They
had square drive heads which seemed to be very soft. They were also
very sharp, which means they bit into the plastic and drove well, but
you had to be very careful in reaching for a handful of them.
Will look the same in 20 years with virtually no maintenance.
Will never rot, crack, splinter, or chip, and never needs paint,
stain or sealer.
Will not leach chemicals into the ground or water.
Is not slippery when wet. (It is quite slippery when
snowy/slushy though!)
Is recycled and is 100% recyclable.
None of defects often found in conventional lumber, such as knots,
decay, splits and milling imperfections (see below about extrusion
defects though).
Requires NO special tools – ALL regular wood-working tools work
well.
Comes in different colors and custom, exact lengths (e.g. 37’ 6" –
no seams).
Can get tongue and groove decking (no visible screws).
Screws with small heads sink into the material, then the plastic
closes up a bit to minimize the appearance of screw holes.
Holds screws and nails much better than wood (in theory, not sure
about practice, especially when really hot).
No allergic reactions sawdust – much safer than PT sawdust.
Dog claws don’t seem to have any effect (not sure about cat claws)
and dogs can’t get traction when trying to move fast, which can be
amusing.
Cons
Much more expensive than PT, around the same cost as high-quality
cedar.
May have extrusion defects, such as creases, bubbles (visible when
cut), warping, thickness variation, etc.
Much more flexible than wood, thus you need joists 12"OC, the joists
must be perfectly level (or the decking will follow the contours),
and long pieces can be quite difficult to maneuver because they flex
so much (picture a 40 foot long wet lasagna noodle).
Even my light gray color gets hotter than wood in strong direct
sunlight. In SE PA my light gray deck is too hot to walk on in bare
feet in the summer in full sunlight. With any kind of shade, not in
summer, or with oblique light it’s fine. It’s actually pretty much
the same as my black asphalt driveway, come to think of it.
Presumably, darker colrs would be worse.
May generate a lot of static electricity when walking or sliding
on it! Dogs can find this very shocking (sorry) when they sniff you
and get zapped.
It is quite slippery when snowy/slushy. (Not slippery when
merely wet.)
You have to clean up the sawdust – you can’t just let it rot into
the ground, because it won’t.
While is is recyclable, my recycling people won’t take it because
it’s not a plastic jug or bottle…
Seems to get dirtier than wood. We’re in a high air traffic
location, which may have something to do with it. And I’ve never
spent a lot of time with a wood deck, but it seems like the plastic
really holds the dirt. If you walk on it in socks they get filthy
quickly.
You can scratch it up draging heavy furniture around. Be very
careful of anything with metal feet.
Conclusion
The final approval for the finished deck was granted 2001-09-18. I’m
very happy with the way it turned out. I added a grounding wire to try
to reduce the static problem, but needless to say that did not help at
all.
If I had it to do over again, knowing what I know now, I’d do pretty
much the same thing. I might try to fasten it differently, but I’m not
sure how. I’d have to figure out something that would slide a bit to
allow for more expansion/contraction. Maybe there is something on the
market now that wasn’t then?
Update 2003-11-06
So far so good. It is weathering nicely, which is
to say not much at all. Expansion and contraction is a problem as
expected. Joints (in the railings and so forth) that used to be tight
are loosening up, and I have a bit of moss on the north side. But other
than hosing it down once or twice I have not touched it!
Update 2006-10-09
Still weathering nicely, which is to say not much
at all. Expansion and contraction still a problem as expected. The
railing joints have loosened a lot, even with some additional screws.
One deck plank literally ripped across because I didn’t leave quite
enough room around a notch for a railing post. Dirt and moss about the
same, a quick pressure wash easily fixes that. A regular hose does not
seem to help much.
Update 2026-02-17
The expansion and contraction issues have ripped more boards, and have
broken out many (nearly all?) of the screws holding the boards down to the
joists. The tongue and groove is still holding together, so even if the
surface is mostly “floating” on the joists…it’s working. I probably did
it wrong back then, and/or there are probably clever new fasteners now.
I’m not worrying about it, it’s still solid after 25 years with zero
maintenance except for some half-assed pressure washing every couple of years.
We took all of the following pictures (click on the image to see the
full size version) with a cheap, disposable underwater camera (no
flash), then I scanned the pictures in and built this page.
This content is old! It’s still useful, but it’s old, and there may be bit rot, newer/better tools or ways to do things. Sanity check and do your research.
Network Computing – Not
strictly about Information Security, but they usually have a few
good security articles, and the rest of the magazine is good too.
MCP Magazine – Even less about
security, since this is about Microsoft products and certification,
but Roberta
Bragg’s
columns are always interesting, though I don’t always agree with her
100%.
There are an awful lot of security books out there. This list covers
only books that I own and have read and found useful. Some may have
newer editions than are listed here, so look for those too. I highly
recommend all of them, but if you only read a few, read the first three.
Also, see the links above for various trade magazines and web sites.
Also, Information Security Magazine
(for which I am a Technical Editor) has an excellent piece on starting a
career in Information Security called " Breaking into
InfoSec."
It has many more references than below, including degree programs in
InfoSec, and books (some of which are on my list too).
Introduction
Secrets and Lies, by Bruce Schneier, from Wiley [ISBN
0-471-25311-1]. Excellent read – accessible and very interesting.
Mostly non-technical, from a business perspective. A must read for
any executive or risk manager from a company that uses the Internet
(and who doesn’t). Also very valuable for technical people, to get
more of a sense of the business side of things. Quite entertaining.
Computer Security Basics, Deborah Russell and G.T. Gangemi Sr,
from O’Reilly [ISBN 0-937175-71-4]. One of the seminal
introductory works on the subject, but there is a lot of material
for the experienced InfoSec person as well.
Hacking Exposed, N’th Edition, by Joel Scambray, Stuart McClure
and George Kurtz, from Osborne McGraw-Hill. A very interesting and
scary read, this details innumerable exploits or hacks, and how to
protect against them. A must for any system or network
administrator. (Note I have the 1st and 2nd editions, but who knows
what it’s up to now.)
Building Internet Firewalls, Second Edition, by Elizabeth D.
Zwicky, Simon Cooper and D. Brent Chapman, from O’Reilly [ISBN
1-56592-871-7]. The updated version of the classic and seminal
work, and a must for any firewall administrator.
The NCSA Guide to Enterprise Security: Protecting Information
Assets, by Michel E. Kabay, Ph.D. from McGraw-Hill [ISBN
0-07-033147-2]. This one reads more like a text-book that the
others above. It has a lot to offer, especially references to other
literature and products, though they are getting quite dated.
White Hat Security Arsenal: Tackling the Threats, by Aviel D.
Rubin, from Addison-Wesley [ISBN 0201711141]. This is different
than most security books in that it tries to be more practical,
presenting “case studies” and solutions to every day needs. It’s a
good read.
Know your Enemy, by The HoneyNet
Project [ISBN 0-201-74613-1] is a
really cool book that talks about how the HoneyNet Project is
researching hacking tools and techniques. See also the “Know Your
Enemy” white papers from
Lance Spitzner and the Honeypots: Tracking
Hackers site.
Intermediate
Handbook of Information Security Management 1999, edited by Micki
Krause and Harold F. Tipton, from Auerbach [0-8493-9974-2]. This
is a typical “handbook” with ten chapters very roughly following the
ISC² ten CBK (Common Body of Knowledge) domains.
Each chapter is written by a recognized expert in
the field, so they all have a different style and perspective.
Computer Security Handbook: Third Edition, edited by Arthur E.
Hutt, Seymour Bosworth and Douglas B. Hoyt, from Wiley [ISBN
0-471-11854-0]. There is a 1997 supplement to my edition of this as
well. This is a very dense and difficult read. I use it more for
lookups and reference than cover-to-cover. There is a lot of
material to covered!
Essential Check Point Firewall-1(TM): An Installation,
Configuration, and Troubleshooting Guide, by Dameon D.
Welch-Abernathy (AKA Phoneboy), from Addison-Wesley [ISBN
0201699508]. There is also Essential Check Point FireWall-1 NG in
the works, probably available in early 2004.
Intrusion Detection, by Rebecca (Becky) Gurley Bace from MacMillan
Technical Press [ISBN 1-57870-185-6]. This book should be
required reading for anyone who even thinks about Intrusion
Detection Systems (IDS). I thought I knew quite a bit about IDS
until I read this book.
Advanced
Securing Windows NT/2000 Servers for the Internet, by Stefan
Norberg, from O’Reilly [ISBN 1-56592-768-0]. Excellent book on
hardening NT/2000. Does not cover details of IIS that much, but
really focuses on the OS. Under 200 pages, very readable, and it
assumes you already know quite a lot about InfoSec and Windows. Has
the best description of the totally counter-intuitive way
Windows “TCP/IP Security” works (and I use the last term loosely).
Also has excellent info on why IIS is such an amazing security risk.
Network Intrusion Detection: An Analyst’s Handbook, N’th Edition,
by Stephen Northcutt and Judy Novak, from New Riders. A very dense
and technical book, with really great material about decoding
various network traces (a lot of focus on
tcpdump and similar tools).
Finally, Sabernet has a large collection
of links for security books,
papers,
links and
tools, but I take no responsibility
for their quality.
Information Security Training
I have only attended CSI and ISC² classes. I hope to attend
some SANS and MISTI classes soon.
CSI – The Computer Security Institute.
Holds a yearly seminar and exposition, with various classes that
“travel” around the country. Usually focused more on concepts, and
less on specific products and/or technology.
SANS – System Administration, Networking,
and Security Institute. Holds a yearly seminar and exposition, with
various classes that “travel” around the country. Focused more on
specific products and/or technologies than CSI.
MISTI – MIS Training Institute. A little
of everything.
Also see below information about ISC² and the CISSP
certification.
What is a CISSP
A brochure I received from the International Information Systems
Security Certifications Consortium or
ISC² defined
the CISSP (Certified Information Systems Security Professional)
designation as follows :
“The CISSP certification is an independent and objective measurement
of professional expertise and knowledge within the information
security profession.”
I would further add that it denotes an individual who has the following
qualifications:
Three or more years of direct professional experience in one or more
areas of Information Security.
Demonstrated a comprehensive understanding of the common body of
knowledge of the Information Security field. This body of knowledge
is divided into ten
domains or areas,
and understanding of the material is demonstrated by a rigorous
test administered
once a quarter all over the world.
Was one of a group of only 4,000 individuals world-wide by end of
2000. (See below for details, but the number of CISSPs has
skyrocketed since I wrote this.)
According to an e-mail message I received from James E. Duffy, CISSP
(ISC² VP) on 9/12/2000, “there are approximately 3000 CISSPs.
The number is up from just under 2000 at the end of 1999. Based on the
number of exams scheduled for the rest of the year, on 12/31/00 we will
be very close to the 4000 number. This will mark the 3rd consecutive
year that we have doubled our base.” And according to SECURITY WIRE
DIGEST, VOL. 4,
NO.74,
OCTOBER 3, 2002,
“The ISC² Monday honored its 10,000th Certified Information Systems
Security Professional (CISSP)… According to ISC², the number of
CISSPs, one of the security industry’s most coveted certifications, has
grown from 2,000 in 1999 and is expected to hit 15,000 by the end of the
year [2002].”
Formed in mid-1989, the International Information Systems Security
Certification Consortium or ISC² was established as a nonprofit
corporation to develop a certification program for information systems
security practitioners. There is a 10 day review class that helps you
understand what material will be covered on the exam. Note this is
simply an outline of the material to be covered – it does not teach
the material! It is well worth it, just for the discussions with the
other students and instructors. The class materials are also helpful.
“Demonstrates knowledge of network and systems security
principles, safeguards and practices. Of primary interest to
full-time IT security professionals who work in internal
security positions, or who consult with third parties on
security matters. CISSPs are capable of analyzing security
requirements, auditing security practices and procedures,
designing and implementing security policies and managing and
maintaining an ongoing and effective security infrastructure.”
This content is old! It’s still useful, but it’s old, and there may be bit rot, newer/better tools or ways to do things. Sanity check and do your research.
This content is old! It’s still useful, but it’s old, and there may be bit rot, newer/better tools or ways to do things. Sanity check and do your research.
This is the companion page for my Firewall Rule Base Best Practices
document. I have listed all the resources I would otherwise have put at
the bottom of the document. In this way, I hope to keep them current,
and to add new material when I find it without having to revise the
original document. If I have written it correctly, it should need
little revision as time passes and technology changes. We’ll see.
Update 2003-01-27
When I started this document in the late 1990s, I was an InfoSec
consultant working with firewalls on a day-to-day basis. That is not my
day job anymore, and I have not found a great deal of time to devote to it. In
addition I have since moved on, and I do not work with firewalls much in
my current role.
I have been surprised at the number of requests that I get for this
draft, and I apologize to all those who I’ve kept waiting though my lack
of time. Thus, I am making this draft directly available on the Internet
in the hope that it will be useful. I disclaim any and all liability-use
it at your own risk.
If you would like to take over the maintenance of this document, let me
know at JPATjpsdomainDOTorg
.
This content is old! It’s still useful, but it’s old, and there may be bit rot, newer/better tools or ways to do things. Sanity check and do your research.
This content is obsolete, but I am leaving it here as a historical reference.
Introduction
This is a quick reference guide for installing the free GNATBox Light
firewall. GNATBox Light is a complete hardened,
stateful, BSD-based
firewall that fits on a single floppy disk (how cool is that?). See
below for references. You can download a Word document with some sample
Avery 5196 diskette labels at http://www.jpsdomain.org/public//GNATBox_Diskette_Labels.doc.
Also check out my Home Networking diagram and explanation at
http://www.jpsdomain.org/infosec/home_networks.html.
If you are interested in firewalls, you should also check out
http://m0n0.ch/wall/ a completely free and Open Source firewall
platform. It is arguably better than the GNATBox in many ways, such as
having a more standard (in firewall terminology) and intuitive
interface, many more features, no arbitrary limits on the number of
interfaces or the number of connections, etc. However, it requires more
resources to run (Pentium or better, 64 MB RAM or better, and a hard
drive, CD-ROM or CF-Card. Both M0n0wall and GNATBox are very cool, and
both have their place, so check them both out.
What’s Needed
486 or better with 32 MB RAM [I’m only using 20 MB] and a floppy
drive (no hard drive)
2 NICs (3Com 3c509b recommended for 486/ISA)
You will need a keyboard and monitor for the install only
Work Sheet
* External IP Address:
+ External MAC Address:
* External subnet mask:
* Default Gateway
ISP DNS 1:
ISP DNS 2:
Internal (PROtected) IP Address:
+ Internal MAC Address:
Internal subnet mask:
* If you have a cable modem, PPPoE or other link that uses DHCP, you
will not need these.
+ It is very helpful, but not required to know the MAC addresses of the
network cards. It’s often written somewhere on the card, especially 3Com
cards.
Basic Instructions (circa October 2002, updated 2006-03-06)
Install the software on the machine from which you will do management.
At the end of the first part of the install, you may want to unselect items
you don’t need, e.g., “Make GNATBox Light PPP floppy.”
Then there will be a few more simple install wizards and you’re finished.
Format and write a GNATBox floppy disk.
(Using GBAdmin or gbMakeFloppy you can “merge” an existing configuration
into the new image when you need to upgrade to a new version. See below.
Set the BIOS to boot without a keyboard if possible on the firewall box and
boot the install floppy.
On the firewall box itself, follow the GNATBox setup wizard to configure
the firewall.
Set the host name.
Enter the external and internal IP addresses and subnet masks as needed.
If you have a cable modem, use DHCP on the external address.
Hit the space bar to select a different interface for the PROtected
interface (it defaults to the one you probably already used for the
external interface).
Hit the space bar to skip setting up a private service network (PSN/DMZ).
This is not available in the free version (neither is VPN).
Set the default route (next hop) if necessary.
On older versions, do not set it if using DHCP externally.
On newer versions, set it to the Interface object of the connection (e.g. <EXTERNAL>).
Set the password for the administration account.
Save the configuration when finished.
When the firewall finishes loading, try ALT-F1, ALT-F2, and ALT-F3:
Screen 1: log messages
Screen 2: console admin tool
Screen 3: network stats
Next, connect to the firewall from the management machine.
Launch GBAdmin → File → Open → Network → enter the firewall’s IP.
Default admin user: gnatbox. Password: whatever you set earlier.
While there is a web GUI, the fat client is usually easier.
You can also disable the web GUI entirely to reduce attack surface.
Register your GNATBox Light (recommended; free).
This lifts some restrictions. I’ve never received spam traceable to GTA.
It is essential that you enter the correct MAC address of the PROtected interface.
Locate it in Network Information under Physical Interfaces.
Example format: 08:00:2b:9a:94:3a.
You can also find it under “Reports → Configuration” and safely copy/paste it.
Authorization → Remote Admin/Authentication:
A web server on port 80 is enabled by default.
I usually disable this, as it is not SSL (free version) and only accessible internally.
Authorization → VPNs: not available in the free version.
Content Filtering: proxies and filtering (e.g., CyberPatrol), HTTP proxy (traditional or transparent).
Routing: configure RIP/static routes if needed.
Objects → Addresses: define objects for use in rules.
Objects → VPN: usually disable (VPN not available in free edition).
Filters → Outbound: verify outgoing rules.
Default allow‑all may be too permissive.
Filters → Preferences → Email Server:
Configure SMTP alerts; otherwise disable the alarm on rule 17.
Filters → Protocols: defines protocols; typically unchanged.
Filters → Remote Access (incoming rules):
Carefully verify what is allowed inbound.
Disable rule 4 (unrestricted in/out DNS).
Rule 7 blocks junk (e.g., UPnP discovery).
Disable rule 13 or set to deny (Ident/auth protocol—obsolete).
Reports → Hardware: details about the firewall hardware.
Reports → Configuration: copy/paste a readable config backup.
System Activity: operational metrics.
Links: various helpful URLs.
Save the configuration to the floppy.
You now have a basic firewall set up.
Hints
You can save and open GNATBox configurations from the network (to
the firewall itself), any number of floppies, and files on the local
hard drive. Since the entire firewall system resides on a single
floppy, this makes the back-out plan when upgrading absurdly
simple–put the old floppy back in and reboot. Likewise, in a test
lab, you can have any canned firewall config you want just by using
a different floppy.
Backup the system by creating a backup floppy. This is also great
for testing! Open the existing configuration from the local drive,
then switch floppies and save both “Configuration” and “Runtime.”
Or, you can open a firewall over the network, save the config as a
file, then merge it to a new floppy as below.
Merge</span an old config into a new GNATBox runtime with GBAdmin:
Run the GUI admin tool.
Open the firewall over the network, or the firewall floppy.
Choose the File, Merge menu.
Load the old config file or floppy.
Verify the configuration, then save the merged config.
“Merge” an old config into a new GNATBox runtime with gbMakeFloppy.
Run “Make GB Lite Floppy”
Click the control menu (icon in the title bar, in the upper
left, directly left of the text “GNATBox Make Floppy”) and chose
the appropriate option.
OK, one very important thing we have not talked about is logging. Since
the GNATBox uses a single floppy disk, it has no room for local logging.
It can log to memory, but that usually runs out pretty fast too. So a
remote loghost is great. If you already have a syslog sever (all UNIXs
have one) you can use that (see the resources section for
syslog server configuration). If not, GNATBox Lite used to come with one
for Windows, but that seem not to be the case any more. See Windows
Syslog Servers below for solutions.
In Services, Remote Logging: Enable logging.
Enter the IP address and port (514) of your syslog server. The
defaults are not bad, so I’d start with them.
If you are using a UNIX syslog and understand facilities, you can
configure those as needed. See the RedHat example below.
If you are using a Windows syslog, you are probably not logging
anything but the GNATBox, so it’s not worth changing facilities.
Advanced Resources
RedHat Syslog & Sendmail configuration
This was tested using RedHat 7.1 and 7.2 but should be similar for most
distributions.
Edit /etc/sysconfig/syslog and add -r to enable listening to the
network like so: SYSLOGD\_OPTIONS="-m 0 -r"
Restart syslog.
Logrotate
Create /etc/logrotate.d/gnatbox with the following contents:
# gnatbox - Logrotation config file
# v1.0 23-Jul-2000 JPV
# v1.1 09-Aug-2000 JPV Bugfix - corrected killall path
# v1.2 2002-04-07 JPV Changed from 15 weeks to various
# v1.3 2002-05-27 JPV Updated to correct e-mail address, then commented, as
# 'errors' is deprecated
# Global Options
compress
notifempty
olddir /var/log/gnatbox/archive
/var/log/gnatbox/filter.log {
rotate 52
weekly
}
/var/log/gnatbox/???.log {
rotate 6
weekly
postrotate
/usr/bin/killall -HUP syslogd
endscript
}
sendmail
NOTE, this will open up your mail server to listen to all addresses that
can reach it. Only do this on an internal mail server, and if you really
understand what it does!
You will need to have the sendmail-cf rpm installed.
Edit the following line in /etc/mail/sendmail.mcChange:DAEMON\_OPTIONS(\Port=smtp,Addr=127.0.0.1, Name=MTA’)**To:**dnl # DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1,
Name=MTA’)`
Run this command to regenerate sendmail.cf, then restart sendmail.
m4 /etc/mail/sendmail.mc \> /etc/sendmail.cf
Edit /etc/hosts.allow and add or change to the following:
{replace nnn.nnn.nnn. with your network} sendmail: nnn.nnn.nnn. : ALLOWOR:sendmail: ALL : ALLOW
Windows Syslog Servers
GNATBox Light comes with a free Windows Syslog server, but here are some
others too.
“Just got your cable installed? Itching to have a personal site on your
DSL? Want to control your own e-mail? Don’t want to have to tell friends
about that annoying changing IP address or ISP- assigned hostname? We
can help!
“Our Dynamic DNS and Static DNS services give you a new name -
yourname.dyndns.org, for example, or you can choose from several other
domains. Sign up, pick a hostname, download one of our selection of
third-party update clients, and you’re on your way! Best of all, these
services are totally free for up to 5 hostnames each. Up to 20 hostnames
in each service are available to donators.”
This content is obsolete, but I am leaving it here as a historical reference.
With the advent of more widespread broadband (cable modem, xDSL)
Internet access and the greater proliferation of SOHO (Small Office/Home
Office) and Virtual Offices, Information Security is becoming more
important at home as well as at work.
Home Network Designs
Recently the question about how to design a relatively secure home
network has been coming up a lot. So rather than trying to draw the same
thing on whatever napkin happens to be handy, I diagrammed the four most
common home network designs, and wrote some text that fleshes out the
details. See
home_networks.html. Zone
Labs, now part of Check Point
Software has a similar sort of PDF
document.
Do these sound familiar:
“There is nothing on my computer I care about.”
“Why would anyone want to hack me?”
“I’m using dial-up so I’m safe.”
“Who cares?”
I hope not, but if you do not have a firewall and you believe any of the
above, you are wrong! Here’s why.
It is possibly true that there is nothing worth stealing on your PC.
But… Do you use Quicken or MS Money? Turbo Tax? The encryption in
those programs is a joke, and if you fill in all the forms them your
entire financial status is a wide open book to anyone who wants to
look. Is your name, address, phone number, credit card information
or Social Security number on your PC? Anywhere? Hum, not so
worthless any more, huh?
Do you have any kind of perr-to-peer or other file sharing software
installed? That would include things like Kazza (AKA KaZaA),
Morpheus, or even distributed computing programs like SETI@home?
Even if you did not install anything like that, did your kids? If
so, your entire hard drive may be open to the Internet. It may not
too. The point it, DO YOU KNOW?
Why would anyone want to hack you? Good question. No reason–they
wouldn’t. It’s purely a numbers game. IP Addresses to be precise. If
your IP Address (kind of like your computer’s “phone number”) is in
the range that some random attacker is scanning, and you are running
a PC that is vulnerable to whatever exploit he’s running, and you
are not otherwise protected (like by a firewall), then you are
hacked. Period, end of story. And you probably don’t even know it.
But so what, right? Wrong. If your machine is hacked in the right
(or perhaps wrong) way, the attacker can do anything he wants.
Including launch denial of service attacks against the Whitehouse,
bounce (redirect) web surfing to terrorist sites though your
computer, use your computer hard drive space for storage of illegal
software–or worse, use your computer and bandwidth (Internet
connection) to send spam, and the list goes on.
Don’t believe the problem is that bad? I used to have a page that tracked
how often my home internet connection was attacked. I stopped a long time
ago because the scanning is relentless.
Hackers steal from pirates, to no good
end. The people who
design rogue programs that take over computers from afar are now
applying the tactic that made music pirating programs so
effective–and the Internet may never be the same.
A third of spam spread by RAT-infested
PCs. Nearly
one-third of all spam circulating the Web is relayed through PCs
that have been compromised by malicious programs known as Remote
Access Trojans, according to Sophos, an antispam and antivirus
company.
US Government’s OnGuardOnline.gov
site to “help you be safe, secure and responsible online.”
*
Home Network Security*ABSTRACT:
Home computers that are connected to the Internet are under attack
and need to be secured. That process is relatively well understood,
even though we do not have perfect solutions today and probably
never will. Meanwhile, however, the home computing environment is
evolving into a home network of multiple devices, which will also
need to be secured. We have little experience with these new home
networks and much research needs to be done in this area. This paper
gives a view of the requirements and some of the techniques
available for securing home networks.
Protecting the Home Office,
7 “musts” will help extend proection to home users and road
warriers. Aimed at corporate InfoSec people, but good advice for
anyone.
James Madison University’s R.U.N.S.A.F.E. program
(End User tips and awareness).
Microsoft Personal Security Advisor
(MPSA) “is an
easy to use web application that will help you secure your Windows
NT 4.0 and Windows 2000 computer system. Simply navigate to the MPSA
site and press the Scan Now button to receive a detailed report of
your computer’s security settings and recommendations for
improvement.” More of a SOHO than corporate focus. (Curiously, this
does not seem to work too well using Netscape. I wonder why???)
CERT Advisory CA-2001-20: Continuing Threats to Home
Users and
Home Network Security
(unmaintained).
The CERT Coordination Center (CERT/CC) is
a major reporting center for Internet security problems. Staff
members provide technical assistance and coordinate responses to
security compromises, identify trends in intruder activity, work
with other security experts to identify solutions to security
problems, and disseminate information to the broad community. The
CERT/CC also analyzes product vulnerabilities, publishes technical
documents, and presents training courses.
Gibson Research Corporation, home of
“Shields Up,” SpinRite and other great tools. Interesting, well
organized information about SOHO security and privacy. Check out the
Leak Test page for interesting
personal firewall and privacy information. This site can be a little
“over the top” and sometimes gets into hysterical,
media-feeding-frenzy language, but if you take it with a grain of
salt and Don’t Panic…
A small
write-up
about the IIS 4 and IIS 5 Lockdown Tool.
Run from a single floppy disk – no hard drive needed
Simple to manage
Remote syslog logging support
I’d considered using OpenBSD with
IPFilter as well, but it does not
quite meet all of my needs. I am also running a kind of “virtual” VPN
[sic] using ssh from
OpenSSH. I’m in the
process of writing up some documentation about this. I’ll put a pointer
here when it’s finished. In the meantime, see O’Reilly’s
SSH, The
Secure Shell: The Definitive Guide.
Obsolete Content
This content is obsolete, but I am leaving it here as a historical reference.
This content is obsolete, but I am leaving it here as a historical reference.
The Risks…
Are real.
There is no security through obscurity. While is it true that it’s very
unlikely that someone will specifically try to hack you, that
doesn’t matter! There are a large number of hacking tools that simply
scan a range of IP Addresses (similar to telephone numbers) for a
vulnerability. If you happen to have an IP Address in the target range,
and if you happen to have that vulnerability–you are hacked–simple as
that. :-(
Here are some statistics from the firewall at my house. I do not have a
web server, or anything “tempting,” these are just the random scans or
“doorknob twists” I just described above. When I wrote this in mid
2002, on an average week, 88 different people tried to attack 41
different services 252 times. At my house!. It’s worse now.
So who cares if they break into my machine? Well, here are some things
to think about:
Do you use Quicken? One Russian hacking ring targeted Windows
machines expressly to steal Quicken files. Are your Quicken files
password protected? It doesn’t matter–it’s trivial to crack that
“protection.”
Do you have information from work on a home PC, or is your work
laptop connected? It would be a lot easier to hack your house than
to hack you at work.
You could become a “zombie,” that is, one of hundreds or thousands
of computers used to launch distributed denial of services attacks
such as the one that brought down Yahoo and Amazon last year.
You could become a bounce point used to conceal an attacker hacking
someone else. Wouldn’t it be interesting to have the Secret Service
show up one day because your PC was attacking Whitehouse.gov?
Adding a wireless connection only increases your security risks. See
below for more information. Wireless can be done securely–or securely
enough anyway–but that entails more work. Security is not “plug and
play.”
Having said all that, dial-up connections are somewhat less of a risk,
as are some types of cable modem. The difference is that with a
dial-up connection or a cable modem that uses PPPoE
(Point-to-Point-Protocol over Ethernet) and/or DHCP (Dynamic Host
Configuration Protocol) your IP Address is different each time you
connect to the Internet. Thus, even if you got hacked, the hacker may
have more trouble finding you again. Note that same cable modems that
use DHCP still get the same address each time, so this is not a
help. Also, depending on how you were hacked and for what purpose,
different IP Addresses may not matter. For example, some kind of
program may b installed to actively tell the hacker what your new
address is every time you connect.
The bottom line is that no matter how you connect to the Internet there
is a risk, and you should do everything you can to minimize that risk.
If you are just a bit more difficult to get into than the next guy, the
hackers will go after him instead of you. And if the various scanning
tools can’t find the vulnerability, they will pass you by. The Internet
is far too valuable not to connect to–just understand the risks and
try to mitigate them.
Typical Home Network Designs
See the diagram below.
Note: this architecture is not suitable for hosting services, such
as a web site or e-mail server, on your home LAN. For that you need to
implement a DMZ, which is out of the scope of this document. Hosting
services may also be against the terms of service of your contract with
your ISP.
Links
Link
Speed
Cost (estimates)
Availability
Dial-up
Slow
$15-25/mo
Almost always
ISDN
Medium
Expensive, and charged per minutes
Usually
xDSL
Varies
Depends on type of DSL and distance from phone company Central Office (CO)-($40-350/mo)
Depends on distance and if the service is available in the area
Cable Modem
Varies-Fast
$35-60
Depends on cable company
Satellite
???
???
???
1. Simple/Home
This is the most common situation. Whatever link is used is just
connected to 1 PC and that’s it. Unless a “personal” firewall is used,
there is very little security, especially on Windows 95/98/ME. Windows
NT, 2000 or XP can be made somewhat more secure, but the default
installation is not secure. In other words, unless you have taken
additional (and sometimes complicated) steps to secure it, it’s not
secure at all.
2. NAT/Firewall Appliance
There are three types of NAT or Firewall appliances in the SOHO market.
These are listed below in order from least to most secure. The price
tends to follow that from cheapest to most expensive, but there are
exceptions.
NAT Device
This is a step better than option 1 and it allows you to connect more
than 1 PC to the link. However, NAT (Network Address Translation) does
not provide that much protection. NAT provides translation between the
public, routable IP Address you get from your ISP when you connect to
the Internet and a private, non-routable address that you can use on
your internal network. Any PC can make a connection out, and the reply
to that request is allowed back in. This is not nearly as secure as it
sounds, but it’s better than nothing.
Firewall
Using a firewall builds on the NAT device. It will virtually always use
NAT as well, but it adds rules that allow you to define what types of
traffic are allowed in and out. A simple packet filter firewall is
better than NAT, but it also has some security problems. To vastly
oversimplify the problem, packet filters only look at what the packet of
data says it is. It’s very easy to make a packet lie, and a packet
filter will usually not catch it.
Stateful Firewall
A stateful firewall builds on the packet filter and keeps a “state
table” of what connections are in progress. This way, if a packet tries
to lie and say that it is part of an established (and thus presumably
allowed) connection, but that connection is not listed in the table of
allowed connections, it is denied. This is about as secure as you are
going to get in the SOHO environment.
The next level of security involves using application level proxies,
which you will not find in typical SOHO devices and which are outside
the scope of this paper.
3. Wireless Appliance
Using a wireless appliance is very similar to option 2, except it adds
wireless capability. The same three levels of security from above may
usually be found in wireless devices as well. But wireless adds
complexity and vulnerability due to the fact that it is wireless.
Now I can site in the street and use your Internet connection to surf,
or to hack someone. If I live next door to you, and my parents have
restricted my Internet connection, I may be able to use yours to get
around those restrictions.
4. Complex/SOHO
This is a complex network, connecting multiple PCs in different areas,
and optionally supporting servers, segmented wireless access and more.
Anything this complex is getting out of the scope of this paper.
Terms
DHCP
Dynamic Host Configuration Protocol, a protocol used to automatically assign IP Addresses to devices when they ask for one.
Firewall
Software or hardware intended to provide a separation between trusted and un-trusted networks. Firewalls often allow you to create rules to define what kinds of traffic are allowed to pass between the different networks.
ISP
Internet Service Provider. The service that connects you to the Internet, e.g. AOL, MSN, etc.
NAT
Network Address Translation, also called masquerading. The process of segregating any number of illegal, non-routable or private IP addresses behind a single or small number of legal, routable or public IP Addresses.
Personal Firewall
A program you install on an individual PC that acts as a firewall. These are often relatively simple and may not allow arbitrary rules to be created.
PPPoE
Point-to-Point-Protocol over Ethernet, a method some cable modems use that establish what looks like a dial-up connection over the cable modem. In other words, you “log in” with a user name and password to get the cable modem to work.
SOHO
Small Office/Home Office, the environment for which this paper is intended.
Stateful Firewall
A firewall that keeps track of existing and allowed connections in a “state table.” More secure than a non-stateful packet filter type firewall.
I firmly believe in the following Security Principles:
100% security is impossible.
99% security may be possible, but is too expensive in terms of
effort, money, time and productivity.
The goal is reasonable and adequate security with reasonable and
sustainable effort. How you define “reasonable” depends on the value
of the information you are protecting. It is not reasonable to spend
$10,000 to protect $5,000 worth of information. You need to
understand what you are protecting, and the realistic threats you
are facing.
Security through obscurity is no security at all.
The best Security is provided by a defense in depth:
Prevention
Hardening
Least Privilege
Separation of duties
Strong, published, security policies, with End User
awareness
Strong authentication methods (especially for Remote Access)
Detection (and Assessment)
Monitoring (logs/network/everything), IDS, etc.
Security/vulnerability assessments
Compliance audits
Response (and Correction)
CIRT (Computer Incident Response Team)
Correct environment based on incidents, assessments, audits
and changed circumstances
Update policies, procedures and guidelines based on
incidents, assessments, audits and changed circumstances
Security is a never-ending circular process, there are no silver
bullets, and it is fundamentally not a technical problem that may be
“solved” with point products.
Some frequently misused or misunderstood terms:
Policy, et al.
Policy
A high-level statement of enterprise beliefs, goals, and objectives
and the general means for their attainment for a specified subject
area. They should not be technology specific, and they should change
rarely.
Standard
Mandatory activities, actions, rules and regulations designed to
provide policies with the support structure and specific directions
they require to be meaningful and effective. They are often
expensive to administer and should be used judiciously. Standards
may or may not be technology specific and may or may not change
frequently.
Standard
Standards are documented agreements containing technical
specifications or other precise criteria to be used consistently as
rules, guidelines, or definitions of characteristics, to ensure that
materials, products, processes and services are fit for their
purpose. (Source: ISO;
http://www.iso.ch/iso/en/aboutiso/introduction/index.html)
Guideline
More general statements designed to achieve the policy’s objectives
by providing a framework within which to implement procedures. Where
standards are mandatory, guidelines are recommendations. Guidelines
may change more often than policy’s, but less often than procedures.
Procedure
Spell out the specifics of how the policy and the supporting
standards and guidelines will actually be implemented in an
operating environment. These are often step-by-step instructions,
and are usually technology (e.g. OS) specific. They may change
often, as new technologies are introduced.
Penetration Test
A covert evaluation of or attack on the environment, specifically
looking for security vulnerabilities to exploit, and often stopping
at the first successful penetration. In my view, penetration tests
are not worth the time or money, with very limited exceptions. If
the attackers are skilled enough, and take long enough, a P-Test
will always succeed. So what does that prove? That you hired someone
smart enough to break into your network-or perhaps you failed to
hire someone smart enough. Either way, of what value is that? None.
Assessment
An overt evaluation of the environment to determine “where you are”
and “what you have.” In this context, the focus is generally on
security, and network architecture, but you can (and in fact should)
assess your environment for other reasons and with other focuses. In
order to plan for the future, you must know where you are. You can
then determine where you need/want to be, and finally plan how to
get there.
Audit
An evaluation to determine if and how well you are in compliance
with an existing set of documented
policies/procedures/guidelines/standards/best practices.
DMZ
DMZ
Demilitarized Zone, as in the military usage. This was originally
the (sub) network outside your firewall, but inside your ISP router.
However, the term has been misunderstood and misapplied to the point
where it is now meaningless. Depending on the background of the
user, it can mean the network as described above, the network in the
middle of a “firewall sandwich,” or the network(s) on a three (or
more) legged firewall. Thus, I prefer the term “service network” for
the network on which Internet accessible services are hosted (which
hopefully is the third leg or between two firewalls). And I prefer
the term “moat network” for the network outside the firewall, but
inside the ISP router, which in itself may provide a layer of
protection via access control lists, etc.
CIA
Confidentiality
Information is only accessible by those people or processes
authorized to use it.
Integrity
Information is changed only in authorized ways, by people or
processes authorized to make the changes.
Availability
Information is available to those people or processes authorized to
use it, when it is needed.
This content is old! It’s still useful, but it’s old, and there may be bit rot, newer/better tools or ways to do things. Sanity check and do your research.
Tip
Everything listed on this page is free, unless otherwise noted (or unless I goofed).
You need to use a password database because humans are bad a remembering
good passwords, you can’t share passwords among sites, and so you need
to have a lot of passwords. Some useful thoughts on this include:
The Real Life Risks of Re Using The Same Passwords
There are a great many password databases out there these days. I
personally don’t trust any of the cloud or browser-based ones, because
anything automated is that much easier to crack in to. It’s a few extra
steps to manually copy & paste the password from the manager into the
correct fields, but it’s a lot more secure.
Password Safe is a free utility originally
from Bruce Schneier and Counterpane
Labs which allows you to keep your passwords securely encrypted on
your computer. A single Safe Combination–just one thing to
remember–unlocks them all. Check Password Safe’s
releases to find the
newest version.
KeePass seems to be another good one, and it has many cross-platform
variations to chose from.
See my random password/pin generator
(written in Perl). It also creates unpronounceable names for aliens,
for when you’re writing SciFi and get stuck for a name… ;-)
Mognet, a free Java-based
packet sniffer and analyzer which comes complete with source code.
It runs on handheld devices or on desktops and is available under
the GNU General Public License (GPL).
Scanners/Tools
NetCat, the “swiss army knife”, for
Win32
or
UNIX.
(Older one for
Win32. 1)
Check out the OpenBSD FAQ relating to
IPFilter for a VERY good and
clear example of IPFiltering, which is similar to the Linux IPTables or
IPChains, and which is a great example of firewall rules in action! See
also:
The OpenBSD project produces a FREE,
multi-platform 4.4BSD-based UNIX-like operating system. Our efforts
emphasize portability, standardization, correctness, proactive
security and integrated cryptography. OpenBSD supports binary
emulation of most programs from SVR4 (Solaris), FreeBSD, Linux,
BSD/OS, SunOS and HP-UX.
I have combined my LogSwap and CPFWBack tools, and added my
extract_patch tool into CPFW1TK–the Check Point Firewall-1
Tool Kit. LogSwap and CPFWBack work under both Windows and UNIX.
Extract_patch is unnecessary under UNIX.
CPFW1TK-3.2.0-2.exe (288,965
bytes) has the scripts and all other binaries needed to run. It also
includes the UNIX scripts just for fun, and it has some other bonus
stuff. It is a self-extracting ZIP archive.
CPFW1TK-3.2.0-1.tgz (10,251
bytes) just has the UNIX scripts and ReadMe files.
Extract_patch was created for extracting Check Point patches
under Windows, without installing WinZip, since Check Point are
now distributing all patches in TGZ
format.
But it will work for any TGZ (or .tar.gz, or .gz or .tar) you wish
to extract under Windows, without having to install WinZip. It
combines
Win32 ports of the GNU
tar.exe, gzip.exe and md5sum.exe utilities, so you can unpack and
verify *.tgz files.
LogSwap archives or “rolls” Firewall-1 logs. It includes
Logswap.cmd, obsolete.com, audit.com
and gzip.exe for Win32.
CPFWBack greatly automates the annoying process of backing up
Firewall-1 configurations. It includes CPFWBack.cmd,
zip.exe,
unzip.exe and
vdate.exe for Win32 and
CPFWBack.sh for UNIX.
I’ve also created an add-on called
jpcshrc for the default csh
configuration in Nokia’s IPSO 3.4.1-FCS5. It sets the csh prompt to
your current working directory, and add some aliases (mostly DOS
commands, since I can’t remember what OS I’m using).
Essential Check Point FireWall-1, ISBN 0201699508, written by
Dameon D.
Welch-Abernathy (AKA
PhoneBoy), owner/operator of the above FireWall-1 FAQ site. There is
also Essential Check Point FireWall-1 NG in the works, probably
available in early 2004.
Tom Horsley’s NTP
Time for Windows is
a nice NTP client program. It is free, but is a client only, and can
be configured to talk to only one NTP server at a time. NTP works
much better when referencing a pool of servers. BUT, it allows you
to use NTP to time-sync a hardened NT Firewall server. The NT
Resource kit TimeServ
will not run with the NT Workstation service disabled or removed
(which it should be on a firewall!!!)
fwlogsum “is a perl
script to summarise FW1 logs making it easier to see what services
are being blocked or allowed through your firewall.”
Securing IIS 5.0 Using Batch-Oriented Command
Files,
the tools. This package is essential to any attempt to secure Win
2000 or NT via script. It includes: auditpol.exe, CryptPwd.exe,
passprop.exe, Reg.exe, regini.exe, xcacls.exe.
A small
write-up
about the IIS 4 and IIS 5 Lockdown Tool and the
download
page for it.
HFNetChk,
the Microsoft Network Security Hotfix Checker, which is a
command-line tool that administrators can use to centrally assess a
computer or group of computers for the presence or absence of
security patches. You can use the Hfnetchk tool to assess patch
status for the Windows NT 4.0 and Windows 2000 operating systems, as
well as hotfixes for Internet Information Server 4.0 (IIS), Internet
Information Services 5.0 (IIS), SQL Server 7.0, and SQL Server 2000
(including Microsoft Data Engine [MSDE]), and Internet Explorer
5.01 or later.
Microsoft Personal Security Advisor
(MPSA)
“is an easy to use web application that will help you secure your
Windows NT 4.0 and Windows 2000 computer system. Simply navigate to
the MPSA site and press the Scan Now button to receive a detailed
report of your computer’s security settings and recommendations for
improvement.” More of a SOHO than corporate focus. (Curiously, this
does not seem to work too well using Netscape. I wonder why???)
John the Ripper password cracking
for UNIX and NT (need
pwdump,
pwdump2,
pwdump3 v2 (ZIP
or here), or
pwdump3e (ZIP)
for NT cracking), runs on UNIX, DOS or Win16.
Old Content
This content is old! It’s still useful, but it’s old, and there may be bit rot, newer/better tools or ways to do things. Sanity check and do your research.
Winpcap is a
libpcap-compatible library for Windows. Libpcap is the basis for most
UNIX sniffer and packet tools, such as namp, nc, tcpdump and dsniff. ↩︎↩︎↩︎↩︎↩︎
This content is old! It’s still useful, but it’s old, and there may be bit rot, newer/better tools or ways to do things. Sanity check and do your research.
Pause.c
A tiny C program that does the same thing as the DOS pause command.
UUConvert.c
An old program that reassembles and uu-decodes binary Usenet threads
saved by old mail readers such as Tin. I did not write this and I
don’t know who did.
Assembly SCR programs
WrmBoot.com and
CldBoot.com (16bytes)
A Warm Boot (CTRL-ALT-DEL) and Cold Boot (same as reset switch) on a
PC. Here is the C, BASIC, and debug .scr source
code.
This zip file is a collection of old “scr”
programs, many collected from old computer magazines. Here is a listing:
ADDTOIT.SCR Extend path environment variable to any length.
ALREADY.SCR Run programs or batch files only ONCE a day, at the first BOOT-UP.
ANSITEST.SCR Returns an ERRORLEVEL 0 if Ansi.sys is active, 1 if it is not.
ANTIBOOT.SCR Disables CTRL-ALT-DEL keyboard re-boot.
AUTOASK.SCR Returns the ASCII code of the character entered at the prompt.
AUTOCAPS.SCR Automatically turns the CAPs lock OFF when you press the shift key.
BIGBUF.SCR TSR to enlarge DOS's usual 15 character keyboard buffer to 127.
BUFFER.SCR Stuffs characters into the key buffer.
CLDBOOT.SCR Cold boot the PC.
CLEARKEY.SCR Empty the keyboard buffer
D2H.SCR Convert decimal numbers to hexadecimal.
ENCRYPT.SCR Simple file encryption (use PGP or something better instead).
ESC.SCR Inserts the escape character in batch files (eg. esc [10m).
FASTKEY.SCR Resets the keyboard repeat rate and delay rate to the fastest possible.
FULLSCR.SCR Script to switch DOS session to full screen in MS-Win.
H2D.SCR Convert hexadecimal numbers to decimal.
ISAREADY.TXT Detail and batch files with a utility to test if Drive a: is ready.
ISAREADY.SCR Utility to test if Drive a: is ready.
KEYCODE.SCR Returns the ASCII and scan codes for pressed keys.
MEGSFREE.SCR Returns an ERRORLEVEL = to the number of free megs.
NOPRSCR.SCR Turns OFF the prn-scr key.
NOSOUND.SCR Disable your PC's internal speaker.
NOWAIT.SCR Resets the number of re-tries BIOS makes to the printer.
NUMLOCK.SCR Toggles the numlock key.
PRSCR.SCR Turns ON the prn-scr key.
PRTLOG.SCR Toggle print logging of the command line.
REBOOT.SCR Warm boot the PC.
RENDIR.SCR Rename directories.
SHIFTKEY.SCR Change the bypass autoexec.bat key in Windows.
SHIFTMOD.SCR Make " , " & " . " keys "unshifted".
WAIT.SCR Timed pause.
Other
Here’s a very simple awk script to
prepare text for inclusion into a HTML file by changing URL’s in a
plain-text file into into HTML links to that URL. I’m not really sure
who wrote it, but I didn’t.
See this zip file for
some date and time related code and routines. I didn’t write any of this
either.
This content is old! It’s still useful, but it’s old, and there may be bit rot, newer/better tools or ways to do things. Sanity check and do your research.
Programs to print a BLACK page to an HP LaserJet. I plan to move this to https://github.com/vossenjp/
at some point…
C Source
#include<stdio.h>/* Program to print a BLACK page to an HP LaserJet */main()
{ /* begin of main */printf("\n\n\nBlack PC Computing October 1991");
printf("\n\nNow printing black page...\n");
fprintf(stdprn,"%c%s",'\33',"&l0E");
fprintf(stdprn,"%c%s",'\33',"&l0L");
fprintf(stdprn,"%c%s",'\33',"*p0x0Y");
fprintf(stdprn,"%c%s",'\33',"*c2400a3300B");
fprintf(stdprn,"%c%s",'\33',"*c0P");
fprintf(stdprn,"%c%s",'\33',"E");
} /* end of main */
BASIC Source
10 REM Program to print a BLACK page on an HP LaserJet20 PRINT:PRINT:PRINT"BLACK PC Computing October 1991"30 PRINT:PRINT"Now printing Black page..."40 OPEN"lpt1"FOROUTPUTAS#150 PRINT#1, CHR$(27) +"&l0E"60 PRINT#1, CHR$(27) +"&l0L"70 PRINT#1, CHR$(27) +"*p0x0Y"80 PRINT#1, CHR$(27) +"*c2400a3300B"90 PRINT#1, CHR$(27) +"*c0P"100 PRINT#1, CHR$(27) +"E"110 CLOSE#1
When saving, rename from *.pl.txt to *.pl or whatever you use for Perl.
If using on Unix, you may need to convert CRLF to LF, and modify
the “shebang” line (e.g. #!/usr/local/bin). “CleanUp”
can fix the line termination. Or find a good dos2unix and unix2dos
program like those found in the UNXUtils.
Perl (Practical Extraction and Report Language or Pathologically
Eclectic Rubbish Lister) is an interpreted language optimized for
scanning arbitrary text files, extracting information from those text
files, and printing reports based on that information. It’s also a good
language for many system management tasks. Start at
www.perl.com, and check getting the latest
Perl. See the
Win32 section especially if you use Win95 or WinNT (Hint: these versions
can access the NT event logs, making collection and auditing them
much easier…). I use
ActiveState’sActivePerl.
I’ve also used IndigoStar’sIndigoPerl (which has
Apache built in) and
Perl2Exe which can “compile”
Perl. And check out the Open Perl
IDE.
Scripts
Clicking on the name of a script will open the code here, so you probably
want to open ina a new tab instead.
General purpose utility to clean up messy text files. Can convert
tabs (see Tab above), UPPER or lower case the entire file, remove
leading “>”, trim, convert CRLF <–> LF, number output, etc.
Ironocally, the code for this is very messy too. One of these days I
need to start fresh, but…
Generate random passwords of arbitrary length and complexity. v3 is
a re-write to clean up the code and make it more modular. It also
now changes interface modes based on script name (.cgi or .pl, still
need to manually edit and add -T). New for 2.2 was dual
interfaces–GenPass can run as a CGI script or from the command
line. New for 2.1 was a Hex option, for generating WEP keys. Try the
CGI/web interface!
Count the frequency of a list of regular expressions in arbitrary
input data so that a file for use with ’egrep -f’ can have the most
frequent expressions first so it runs faster.
A more simple and generic program to merge lines two data files with
a common, unique key. This program is similar to the UNIX join
command, but is intended more to process TAB delimited files cut
from and pasted back into a spreadsheet. It creates a table or
matrix of data, where each line is merged on a unique key (hence
MergeL).
GenSite is the simple script (relative to “real” Content Management
Systems (CMS) anyway) that created the HTML code for this site up until 2026.
My main constraint is that the ISP where most of the site is hosted does not
support SSI, CGI, or anything else that would facilitate a modern site.
I have various reasons for staying there, not the least of which is
inertia. (That changed probably decades ago…)
So I needed a simple solution that would create a navigable
site without any those technologies. Also, after the last redesign
JavaScript fiasco, I wanted totally PURE HTML. Since the site is mostly
static, that was fine, except for being able to actually maintain it. So
I wrote this.
Generate a static HTML-only web site from templates. (See also my
GenSite wrapper web.)
OLD
Old Content
This content is old! It’s still useful, but it’s old, and there may be bit rot, newer/better tools or ways to do things. Sanity check and do your research.
Some Perl modules you might need (especially since I used them in some
of these scripts). Just extract the files (use WinZip) and copy *.pm to
\perl\lib\Number, \perl\lib\time or the correct path.
Fix-wls v1.0a 19-Mar-2000
Converts those annoying WatchGuard *.WLS files to Self-Extracting
archives. This is obsolete with current versions of Live Security.
LCaseHTML v1.1 05-Apr-1996
Converts HTML commands to lower case in HTML code, overwriting
original file. (I got tired of inconsistent case on my HTML code.
Talk about anal…)
URL2HTML v1.1b 15-Nov-2000
Converts URLs (in the form of URL{tab}description) to Bulleted HTML
lists.
REM From PC Mag, Vol11 Number 5DECLARE SUB ReBoot (Warm%)
CALL ReBoot(1) 'be sure to save this program before running it!SUB ReBoot (Warm%) STATICIF Warm%THEN'if they want a warm bootDEF SEG=0'assign the value 1234 HexPOKE&H473, &H12 'to address 0000:0473 HexPOKE&H472, &H34
ENDIFDEF SEG=&HFFFF 'either way call the BIOSCALL Absolute(0) 'routine at FFFF:0000 HexENDSUB
Assembly
To use these “scripts” cut&paste the source into a file, then issue the
following command “debug < cldboot.scr”.
CLDBoot.scr
A 0100
MOV AX,40
MOV DS,AX
JMP FFFF:0000
N CLDBOOT.COM
RCX
10
W
Q
WRMBoot.scr
A 0100
MOV AX,40
MOV DS,AX
MOV WORD PTR [72],1234
JMP FFFF:0000
N REBOOT.COM
RCX
10
W
Q
Note: Under DOS & Win9x/ME some of the simple STDOUT redirection and
pipes work, but none of the advanced STDERR or multiple command methods
are supported.
IO Redirection in Windows NT, 2000, XP, UNIX (sh, bash and variants)
File Descriptors
FD
Description
1
STDOUT
2
STDERR
3+
Additional files as opened by the process
Redirection
Command
Description
cmd1 | cmd1
Pipe STDOUT of cmd1 into STDIN of cmd2
\> file
Direct STDOUT to file, overwriting existing contents
\>\> file
Direct STDOUT to file, appending to existing contents
`>
file`
2\> file
Direct STDERR to file, overwriting existing contents
2\>\> file
Direct STDERR to file, appending to existing contents
\< file
Get STDIN from file
2\>&1
Direct STDERR to the same place as STDOUT
\>& file
Direct both STDOUT and STDERR to file
2\>&
Duplicate STDOUT to STDERR
echo 'foo' \>&2
Send output to STDERR instead of STDOUT
Notes:
Numbered file descriptions above may be used arbitrarily.
noclobber is a UNIX setting that prevents overwriting (clobbering)
existing files by redirection.
UNIX /dev/null is equivalent to Windows NUL. Windows NUL is not case
sensitive.
^ is the meta-character escape in DOS/Windows, so it may sometimes
be necessary to use ^| (e.g. when using egrep in a batch file). You
may use ^^ for a literal ^.
Examples:
Command
Description
dir c:\*.* > myls.txt
Redirect output of ls into myls.txt, overwriting or creating myls.txt if necessary
dir c:\winnt\*.* >> myls.txt
Append more output of ls into myls.txt
noisy_cmd > NUL
Make STDOUT output from noisy_cmd go away
noisy_cmd 2> NUL
Make STDERR output from noisy_cmd go away
noisy_cmd > NUL 2>&1
Make ALL output from noisy_cmd go away
noisy_cmd 2> NUL 1>&2
Make ALL output from noisy_cmd go away
noisy_cmd | more
Pipe noisy_cmd STDOUT into more (or less or whatever)
noisy_cmd 2>&1 | more
Pipe noisy_cmd STDOUT and STDERR into more (this is great for those “net” commands that scroll off the screen when you try to get help)
echo some message 1>&2
Use the echo command to send output to STDERR (it usually goes to STDOUT).
Running Multiple Commands in Windows NT, 2000, UNIX (sh, bash and variants)
Use parentheses to nest as needed.
Command
Description
cmd1 & cmd2
Run cmd1, then run cmd2
cmd1 ; cmd2
Run cmd1, then run cmd2 (UNIX only)
cmd1 && cmd2
Run cmd1. If it finishes successfully then run cmd2
This content is obsolete, but I am leaving it here as a historical reference.
Introduction
Does your computer ever do any of the following, “just for the heck of
it,” with no rhyme or reason? Does it do it a lot? More than once or
twice a day?
Lock up, freeze or otherwise crash unpredictably
Lock up, freeze or otherwise crash predictably
Often have Application Errors, Segmentation Faults (used to be GPFs
– General Protection Faults).
“Thrash” the hard drive – that is, have a lot of hard drive
activity (and sometimes even noise) with little result, or when you
are not actually doing anything? (This can also mean you need more
physical memory or RAM.)
If so, this page may be able to help. It is intended for novice
users, so hopefully everything will be clear. Let me know at JPATjpsdomainDOTorg
if it’s not.
Warning
This page is mostly oriented towards the Win9x code-base, which includes
Windows 95, 98 and ME. Some of the material is applicable to Windows
NT/2000/XP (such as clearing the temp directory) and some is not. This
is indicated in square brackets after each step.
Disclaimer
Some of the methods and techniques I talk about can be dangerous to your
data! Use this page at your own risk.
Make backups of your important data. I strongly recommend purchasing and
using a modern tape drive. They cost between $100 to $500 for typical
retail/consumer oriented units. Some of them even plug right into your
printer port, so you don’t even have to open up the computer to install
it, and they may be used on more than one computer. Another solution is
a CD Burner. They will not be able to back up the entire system, but
your data files will easily fit onto a CD-Rewritable or CD-R disk.
Consumer Tape Units
To be added.
CD Burners
To be added.
Hardware Upgrade/Restore Voodoo
Thanks to Leo for this great info.
Ever try to upgrade hardware on a Windows 2000 or XP system, especially
the motherboard or hard drive controller? Bet you regretted it, didn’t
you? Well, here are some MS TechNet articles that might help. Read
the
disclaimer!
“STOP 0x0000007B” Error After Moving Windows 2000/XP System Disk to
Another System
[Windows
2000]
[Windows
XP]
How to Move a Windows 2000/XP Installation to Different Hardware
[Windows
2000]
[Windows
XP]
Winsock Issues
The Windows TCP/IP networking stack is not the most stable thing in the
world. In particular, there is a ability for third-party applications to
tie themselves into the stack for various reasons. The advisability of
allowing this is questionable at best, but there it is. One of the
possible results of this “feature” is that Windows networking can
sometimes simply die. Often there is no remedy but reinstalling the
entire system from scratch.
However, since this is a known issue and such a giant pain there are a
couple of tools available to try and help. In particular,
LSP-Fix “is a free utility to repair
[problems in] Layered Service Provider or LSP, a piece of software
that can be inserted into the Windows TCP/IP handler like a link in a
chain. However, due to bugs in the LSP software or deletion of the
software, this chain can get broken, rendering the user unable to access
the Internet.”
Crapware & system slowdown or instability
There are programs out there that spy on you (spyware), monitor
keystrokes, bother you with advertisements (adware) and all kinds of
other things. They are often collectively known as “crapware” and they
are BAD. Many add-on tool or search bars for IE are crapware! Many of
the symptoms above can be cause by crapware, so it’s definitely worth it
to look into. I recommend reading the following links, then installing
and using the free
Ad-aware and
Spybot programs.
The PC Decrapifier will uninstall
many of the common trialware and annoyances found on many of the PCs
from big name OEMs (XP or newer only).
I call these techniques “voodoo” because it is often not clear that they
a) should work or b) why they work. They just do. This page is mostly
oriented towards the Win9x code-base, which includes Windows 95, 98 and
ME. Some of the material is applicable to Windows NT/2000/XP (such as
clearing the temp directory) and some is not. This is indicated in
square brackets after each step.
I’ve created a bootdisk that will automate some of these steps for you,
if you are using Win9x/ME. See the Boot Disk
section below.
Shutdown your computer. Turn it off, and unplug the AC cord. Try to
turn it on and off again a few time (this clears gremlins out of
components that normally get power even when the PC is “off”).
[All]
Boot into “DOS” mode. [Win9x/ME]
Clean out the “Temp” directory. [All]
Run ScanDisk and/or Chkdsk (technically, ScanDisk
is preferable, but I’ve had some luck with ChkDsk where ScanDisk
failed). [All]
Reboot into GUI (Graphical User Interface) mode. [All]
Defrag your hard drive. [All but NT]
Get rid of unnecessary “services” and background programs. [All]
Reboot and test to make sure everything works. [All]
Finally, the worst case is that you have to back up your data, reformat
your hard drive, and start over.
Boot into DOS Mode
Skip this step if you are using my boot disk.
This applies to Win9x/ME only. In NT, you can’t really do this at all,
especially if you are using NTFS (which you should be). (Technically,
that’s not true – there are ways. But they are far too complicated for
this page.) Win2K has something similar, but you probably don’t need or
want to mess with it.
Shutdown your computer. Turn it off, and unplug the AC cord. Try to
turn it on and off again a few time (this clears gremlins out of
components that normally get power even when the PC is “off”).
Wait about 5 minutes for it to really drain. Better yet, go to
lunch, or bed or go watch TV for awhile, or whatever.
Plug the AC cord back in and power up.
Watch for the screen to say “Starting Windows 9x.”
Hit the F8 button – quick – you have about 2 seconds. You should
get a menu.
Choose “Safe Mode - Command Prompt only”.
Continue on below and clean out your TEMP directory.
If you don’t get the “Starting Windows 9x.” or the menu, there are a
couple of reasons why that could be. Lots of computers have annoying
manufacturer logos that cover this stuff. Try hitting the ESC key when
the logo comes up. Other “F” keys sometimes work too.
You can also going to Start, Shutdown, “Restart in MS-DOS Mode,” which
will do pretty much the same thing. You type “exit” to end MS-DOS mode
and return to windows.
If you get the “Starting Windows 9x.” but pressing “F” keys does not
work, your c:\MS-DOS.sys file probably has BootKeys=0, BootMulti=0 or
something else screwy. Try rebooting and holding down one of the CTRL
keys. If that doesn’t work you have to edit c:\MS-DOS.sys, which is a
bit of a pain to correct that. See MS Technet “Contents of the Windows
Msdos.sys File
[Q118579]”
for details and instructions on how to fix it.
The “temp” directory is used to temporary files. Just about every
program uses temp files for something. The operating system uses then,
your applications like Word or Excel use them, and utilities use them.
What is supposed to happen is that when the program either shuts down,
or no longer needs the temp file – it is deleted. In practice this does
not always work. So over time you can build up amazing numbers of junk
temp files. I’ve seen computers with temp files dating back two and
three years, wasting hundreds of megs of disk space.
Of course, you can’t just randomly delete everything in the temp
directory. No. That would be too easy! The problem is that some of the
stuff in there might actually be in use and if you delete it, something
will break. Usually the system will not let you delete something that is
in use, but you can’t depend on that. There are two ways around this.
The first and best is to boot to DOS, which guarantees that nothing is
currently in use, then clear it. The second way is to reboot, and clean
out the temp directory before anything has a chance to get started. The
problem with that is that many people have all kinds of stuff running in
the background on startup, and those programs may open temp files right
away (see unnecessary services).
Boot to “DOS”
Find the TEMP directory. Usually c:\windows\temp (Win9x) or
c:\temp (NT).
Try to delete the TEMP directory with “deltree c:\windows\temp”
for Win9x or “deltree c:\temp” for NT. If it says “bad command or
filename” you do not have deltree in your path. Try
c:\windows\command\deltree. If that still does not work, cd into
the TEMP directory and type “del *.*”. In any case, when asked if
you really want to delete everything, make sure you are in the
correct place, and that it says to delete the correct things, and
say “yes.”
If deltree worked, you must re-create the temp directory. Type “md
c:\windows\temp” or “md c:temp” as needed.
While you’re at it, run ScanDisk (see below).
Run ScanDisk and/or Chkdsk
Do this after using the boot disk.
ScanDisk and Chkdsk (Check Disk) perform more or less the same function.
The difference is that Chkdsk is the old, DOS, command line tool, which
does not really know about long file names. ScanDisk is the more modern
graphical tool that can deal with a few more problems than Chkdsk. There
is also command line (non GUI) version of ScanDisk.
Non-GUI
Boot to “DOS”.
Type “scandisk” and let it run.
Reboot when finished.
GUI
Double-Click on “My Computer”.
Right-Click on the C: drive.
Choose “Properties”.
Find the “Tools” tab and choose it.
Click on the “Check Now” button.
Run Defrag (Win9x or Win2k only)
Read the
disclaimer! If the power goes out while defrag
is running, you can lose your data!
Do this after using the boot disk.
NT does not come with a defragger, so you are out of luck. Win9x and
Win2k do have them. You want to run it from the GUI (Graphical User
Interface) not the command line or DOS prompt.
Double-Click on “My Computer”.
Right-Click on the C: drive.
Choose “Properties”.
Find the “Tools” tab and choose it.
Click on the “Defragment Now” button.
Remove Unnecessary Services
Do this after using the boot disk.
This is easy to say, but hard to do. There are so many different tools
and configurations out there that it’s impossible to say what is really
needed or not. However, if your system tray (the little area next to the
clock) has more than 3-8 items in it, you probably have too many and
could get rid of some. Also, some incredibly annoying applications, such
as Netscape (which I otherwise like) install crap you probably don’t
want or need in the system tray – without asking! For example, Real
Player and AIM (AOL Instant Messenger) are often installed. Quicken is
another one that runs a bunch of crap most people don’t use.
I can’t even really tell you how to get rid of them, as they are all
different. However, I can tell you where to look.
Win98 (and I assume ME)
If you have Win98 (or I assume ME, but I don’t know for sure), go to
Start, Programs, Accessories, System Tools, System Information, then get
the Tools, System Configuration Utility. Better yet, go to Start, Run
and type “msconfig” (without the quotes) and hit enter. This gets you to
the same place, a lot faster.
Anyway, the System Configuration Utility has a very cool “Startup” tab.
This shows you all the stuff that starts up with your PC. You can go
in there and un-check things, then reboot and make sure everything still
works. You should check all the other tabs, but unless you see something
really obvious (like drivers for an old hardware device you no longer
have) you should probably not touch them too much.
NT
Go to Start, Settings, Control Panel, Services. See what is started, and
what is automatic. Try stopping things that you know you don’t need. For
example, you might be running a web server, FTP server, etc. without
even knowing it. This is a large security risk as well. If you can
stop services with no ill effects, you must change the startup type to
manual (which will still allow the service to be started under some
circumstances) or disabled (the service is now toast) or they will be
restarted the next time you reboot.
Win2K
Go to Start, Settings, Control Panel, Administrative Tools, Services.
See above NT section for the rest of the details.
Other Possibilities
Consider these after trying the above and/or using the boot disk.
Do you turn your PC off every ay or two? This is less of an issue
for NT/2000, but with Win9x, if I leave it on for more than 5-8
days, it gets really wacky and crashes. Get an “uptime” utility to
find out how long your machine has been up. See my
Favorite Utilities, Tools, Software for Windows
to get one.
Another possibility is a bad driver for some device. Video card
drivers are notorious for this. This one can look like either of the
next two.
A version conflict between some files can cause these kinds of
symptoms. It could be system DLLs or just about anything, and is
very hard to diagnose. Sometimes, if all the problem started when
you installed something (like the AOL client or IE, both of which
can really screw up your system) or if it always/only happens when
you run a particular program, you can figure out what did it and
maybe uninstall or fix it. But sometimes it’s more subtle than
that, as in the case where it’s a particular combination of thing,
that doesn’t happen often or is hard to reproduce. Then you’re kind
of stuck. Nuking the system and re-installing everything from
scratch might fix it, but that’s a lot easier said than done.
Finally, flaky hardware, new (and not quite compatible) hardware, or
hardware in the process of going bad. Systematically swapping out
hardware one bit at a time will usually find this kind of thing. Of
course, that takes a lot of time, and most people do not have spares
of everything. Your local computer shop might be able to help, but
don’t be surprised if they can’t, won’t or charge a lot for it.
The Boot Disk
This boot disk will work for Win9x/ME. It might work for NT/2000/XP, if
you are using the FAT file-system (which you shouldn’t be). This disk
uses only FREE programs – from the FREEDOS
Project!
Download the Boot Disk.
This boot disk will clean out the TEMP directory and some other junk
files, then try to run a command line ScanDisk. Since you are booting
from the disk, you do not have to mess with DOS mode – you’re already
where you need to be.
It will also clean up some other files that I didn’t mention above. If
any of the following exist, they will be deleted:
Lets you create DOS and Win95 boot sectors for the NT Loader
(NTLdr.exe). IT also lets you fix broken NT boot sectors. Very nice
little freeware tool. For more information about NT Boot Sectors,
you can check this direct boot
page.
Old Microsoft utility to delete partitions – ANY partitions. Great
for removing NTFS partitions from a bootable DOS floppy, or for
those times when FDisk confuses itself and will not let you remove
an extended partition because it says there are logical drives, but
when you try to delete the logical drives it says there aren’t
any…
From PC Magazine, allows you to compare two different directories
(see FreeCommander, below). Since this is so old (DIRMATCH 3.1 (c)
1989 Ziff Communications Co.) it doesn’t really work with long
filenames. (If you have problems downloading try
dirmatch.com.txt and rename it
to remove the trailing .txt.)
A dual-pane file manager for all 32-bit windows platforms (i.e.
95/98/NT/2000). It is a free program, similar to an updated Norton
Commander. Nicer and more current than 2xExplorer.
An obsolete dual-pane file manager for all 32-bit windows platforms
(i.e. 95/98/NT/2000). It is a free program, similar to an updated
Norton Commander. Use the previous one instead.
Really awesome “native” Win32 ports of some UNIX tools. Native
in this context means that no emulation layer (e.g. CygWin) is
needed. This avoids a lot a installation complexity. With these
native tools, you unzip ’em and run ’em!
bc-1.05, bison-1.28, bzip2-1.0.2, diffutils-2.7, fileutils-3.16,
findutils-4.1, flex-2.5.4, gawk-3.1.0, grep-2.4.2, gsar110,
gzip-1.2.4, indent-2.2.9, jwhois-2.4.1, less-340, m4-1.4,
make-3.78.1, patch-2.5, recode-3.6, rman-3.0.7, sed-3.02,
shellutils-1.9.4, tar-1.12, textutils-2.1, unrar-3.00, wget-1.8.2,
which-2.4.
It it listed as a beta, and for Windows 3.51, but it runs under NT
4. I have not tested it much though. See the
Readme.txt
or download the
archive
(~ 1.7 meg). Licensed under GNU (free), and stand alone (i.e. does
not need DJGPP or CygWin).
While the programs above are ports of various UNIX tools, the next four
items are complete UNIX environments for the PC. DJGPP is older, and has
not been updated too much (not that it needs it). DJ Delorie went to
work on the Cygwin project after he wrote DJGPP. Both environments are
very cool! I don’t know too much about the other two, and have
never used them.
A complete 32-bit C/C++ development system for Intel 80386 (and
higher) PCs running DOS. It includes ports of many GNU development
utilities. The development tools require a 80386 or newer computer
to run, as do the programs they produce. In most cases, the programs
it produces can be sold commercially without license or royalties.
Use the DJGPP Zip File
Picker to figure out
what you need to download.
A port of the popular GNU development tools/environment for Windows
NT, 95, and 98. It implements the Cygwin library which provides the
UNIX system calls and environment the programs expect. With these
tools installed, it is possible to write Win32 console or GUI
applications that make use of the standard Microsoft Win32 API
and/or the Cygwin API. As a result, it is possible to easily port
many significant Unix programs without the need for extensive
changes to the source code. This includes configuring and building
most of the available GNU software (including the packages included
with the Cygwin development tools themselves). Even if the
development tools are of little to no use to you, you may have
interest in the many standard Unix utilities provided with the
package. They can be used both from the bash shell (provided) or
from the standard Windows command shell.
MS UNIX environment for NT/2K; $99. Was Software Systems (dead line:
http://www.interix.com/); Microsoft Corporation acquired them on
September 16, 1999.
“NirCmd is a small command-line utility that allows you to do some
useful tasks without displaying any user interface. By running
NirCmd with simple command-line option, you can write and delete
values and keys in the Registry, write values into INI file, dial to
your internet account or connect to a VPN network, restart windows
or shut down the computer, create shortcut to a file, change the
created/modified date of a file, change your display settings, turn
off your monitor, open the door of your CD-ROM drive, and more…”
FreeDOS aims to be a complete, free, 100% MS-DOS compatible
operating system.
FreeDOS is ideal for anyone who wants to bundle a version of DOS
without having to pay a royalty for use of DOS. FreeDOS will also
work on old hardware, in DOS emulators, and in embedded systems.
FreeDOS is also an invaluable resource for people who would like to
develop their own operating system. While there are many free
operating systems out there, no other free DOS-compatible operating
system exists.
There are also a great number of free DOS tools in the FreeDOS
Software List, such as
deltree,
exe2bin,
fdisk,
tree,
nasm,
finger,
tail,
tee,
which,
and
more.
AWESOME if you are a CLI (command line interface) junkie like me. It
is a replacement for command.com or cmd.exe (there are also a
variety of other products).
I can’t even begin to list all the features (which are pretty much
common across all platforms/products)! Since I am not the best
typist, the file name completion is one of my favorites. The ability
to copy to the clipboard from the DOS prompt is another great thing
(i.e. type autoexec.bat > clip:). If you have a DOS Prompt in your
Startup group (or equivalent) you NEED this stuff! It is shareware,
but reasonably priced for all that you get out of it.
List is Vernon D. Buerg’s famous text viewer
I used to be unable to exist without this thing, but I use less these days!
I used
v7.7a because it works
with network drives, but there are others, including shareware
versions such as v9.3a.
If you want something similar to list, but for UNIX, try “less” or
“pilot” which is the browser built into
Pine. “Pico,” the Pine editor,
is also good for people who don’t like the typical UNIX editors.
Pine is free (but not GPL).
Check out the new Windows versions. 2.04g is still the latest
DOS version. Do not download anything other than that – it may be a
virus. Better yet, use
InfoZip, the free
GNU replacement for the PKWare command line
tools. All PkWare is shareware.
From the Readme: “NMAKE.EXE 1.50 is a the version of NMAKE that
ships with Visual C++ 2.X. It is used to build external projects. It
is a 32 bit version of NMAKE that was designed to run on Windows NT
version 3.5. It has been extensively used on Windows 95 and requires
a 32 bit operating system. It will not work on Windows 3.1.”
Sweepup helps you keep crap from building up in your temp and cache
directories. WARNING: It may delete things you don’t want to
delete! Read and understand the code before you run it. It is
well documented and simple. Also note that some programs will copy
files to the temp directory, then require you to re-boot. Files from
the temp directory are then used to replace other files that are in
use when the system is up. If you run Sweepup from your “Startup”
group like I do, you can nuke these programs before they work. You
have been warned!
Windows & Graphical Tools
VNC
Real VNC,
TightVNC,
UltraVNC,
WinVNC (with NTLM authentication),
OSXvnc,
Xvnc
(X server on one side, VNC server on the other, very cool), or even
PerlVNC
A remote display system which allows you to view a computing
‘desktop’ environment not only on the machine where it is running,
but from anywhere on the Internet and from a wide variety of machine
architectures. Essentially a FREE PC Anywhere on TCP/IP only,
supporting many different operating systems. (See also
Wikipedia
VNS details and history.)
A freeware telnet/dial-up application that supports Japanese fonts,
inline XModem, ZModem, Kermit, etc. and much more. It has its own
macro language too. It is not the easiest thing to set up
(especially the modem strings), but it works great. I wanted a
freeware application that could do ZModem and have the same
interface for both telnet and dial-in and had 16 and 32 bit
versions. This is a best one I’ve found. It also has a free Secure
Shell (ssh) client plug-in called
TTSSH.
An excellent shareware GUI-based full-featured compression
extraction program, that handles many formats (Zip, ARJ, LZH, ARC,
TAR, Z, GZ, TAZ, TGZ, UUencoded, XXencoded, BinHex, MIME , LZEXPAND
(MS *.??_)).
WinZipSE
creates Windows and/or DOS executable self-extracting archives that
can optionally install things (ala Win95). There are 16 and 32 bit
versions. Also see the
InfoZip graphical front end
WiZ. Last time I
checked, WinZip was still a lot better and easier than WiZ
though…
Uptime: See how long your Windows computer has been up.
The Uptime.exe that I like (looks cool, dynamic updates).
Vince Fatica’s
Uptimes (command line and GUI versions).
MS
Uptime.exe
(Microsoft’s feature bloated version – NT SP4 and better only).
This content is old! It’s still useful, but it’s old, and there may be bit rot, newer/better tools or ways to do things. Sanity check and do your research.
The term “shell script” comes from UNIX, the DOS term is “batch files.”
UNIX shell scripts are very powerful and flexible, they are essentially
programming languages unto themselves. Windows or more rightfully DOS
batch files are a pale imitation. However, sometimes you need to write
something that will just work on any plain old out-of-the-box
Windows install someone has–without adding all kinds of other tools.
Before getting too deep into this topic, consider if there is another
tool you might use. Here is a list of tools, all of which are far more
powerful, flexible and are probably easier to use than batch files:
Bash or other UNIX shells ported to DOS/Windows, e.g. the CygWin
project. It may not be totally
obvious how to get things at the Cygwin site. Either download the
“setup.exe” program,
which will guide you through everything, or use the mirrors
page to find a
mirror site and get everything yourself.
AutoIt Free scripting and
installation language for 95, 98, ME, NT4, 2000, XP, 2003. No
run-times and may be compiled into an EXE.
And there are tons of other free scripting tools for Windows out on
the ‘Net.
Powershell
If you follow Windows at all you will be aware that
Powershell is Microsoft’s
new command line tool, and that you will be required to use it more and
more with newer Windows versions. That is a Good Thing, in my opinion,
and it only took them about 20 years to realize, but that is not covered
here! I don’t really do Windows anymore and I have not bothered to
learn Powershell, which reminds me unpleasantly of Java’s verbose
ugliness. So this page is somewhat historical, though most everything
should work to at least Win7.
If you are interested in current Windows command line scripting and
Powershell (and if you like Windows you should be), there are any number
of other resources and books that will help. These are probably good but
I haven’t read them:
Did you know that Window’s cmd.exe has file and directory name
completion, like UNIX shells? It does, and that can be amazingly useful.
But in most versions of Windows it’s not turned on by default. (I
believe it may be on in Windows 2003, but can’t swear to it.)
To enable file and directory name completion under Windows, download
this registry file and remove the
.txt, then double-click on it and answer yes to the question about
importing into the Registry. If you can’t download for some reason you
can copy the text below into a new file and import it, or just open
regedit, navigate to the key, and change the values for CompletionChar
and PathCompletionChar to 9. One you’ve done that, open a new command
prompt and type dir c:\win then hit the TAB key and watch what
happens. Of course the up arrow and other command line editing functions
will still work as always.
REGEDIT4
; NT-TAB.reg -- Sets the NT Command Completion Character to TAB
; Use "RegEdit /s NT-TAB.reg" for silent installations
; v1.0 1998-10-22 JP Vossen http://www.jpsdomain.org/
; v1.1 2001-09-06 JPV Added PathCompletionChar
; v1.2 2003-03-30 JPV Added .DEFAULT and SOFTWARE sections
[HKEY_CURRENT_USER\Software\Microsoft\Command Processor]
"CompletionChar"=dword:00000009
"PathCompletionChar"=dword:00000009
; Can also do this if you have the permissions
;[HKEY_USERS\.DEFAULT\Software\Microsoft\Command Processor]
;"CompletionChar"=dword:00000009
;"PathCompletionChar"=dword:00000009
The Dirt
OK, if you are still going to go through with this, the first thing you
need is Tim Hill’s Windows NT Shell
Scripting,
otherwise you don’t have a chance. For Windows 9x/ME, you are still
toast, but for NT/2000 this book is really great. It’s the only way you
can navigate the bazaar, inconsistent, contradictory and often asinine
“scripting” language built into cmd.exe.
Using material from that book, plus my own almost 20 years experience
with DOS batch files, I still had a hell of a time writing the following
script. All it does is give you some basic file information (similar to
UNIX stat) and tell you if a file will fit on a floppy disk.
The Scripts
Clicking on the name of a script will open that script in a new
window.
Another of the many lacking tools is a simple “sleep” command, but you
can easily fake that using the “ping” command of all things. The
following will “sleep” for about 5 seconds, give or take:
C:\> ping -n 5 localhost \> NUL
Obviously you adjust the 5 as needed for the number of seconds. You can
even write a trivial “sleep” function in your scripts:
@echo off
REM sleep_demo.cmd--Simple "sleep" command demoREM 2012-07-26echo Before sleep
call :sleep 7
echo After sleep
REM End of Main programREM ###################################################################goto :EOF
REM +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++REM sleep for a specified number of seconds, more-or-less...REM Called like: call :sleep 7:sleep
set sleep_secs=%1
ping -n %sleep_secs% localhost > NUL
goto EoF
The Date
How to get and use dates and time in Windows scripts.
This is trivially easy in UNIX. You want to copy a log file to a dated
name? “cp mylog `date ‘+%Y-%m-%d’`-mylog” will copy mylog to
2002-11-27 (as of this writing). What could be easier? But in Windows,
it sucks.
There are two basic ways to approach this, both with advantages and
disadvantages. The native way is the “for” and “date /t” commands under
NT/2000/XP. These do NOT work under Windows 9x and they do not
consistently use 2 digit time fields, which totally screws you up if you
need the time. The second way is to use the UNIX date command, then do
whatever you please. This is very flexible, but requires you to download
and have the executable (date)
handy. You will also want to rename it (I use udate.exe) so you don’t
conflict with the built-in date command.
UPDATE (2012-07-26): All versions of Unix “date” commands that I have
tested under both WinXP and Win7 have a bug that causes them to skip
skip Mar-11 and/or
Apr-04!
That’s pretty annoying but has never been fixed as far as I know. Since
it affects both tools I’ve tested (UnxUtils and GNU Win32), I suspect
the Windows strftime lib is the problem. But I can’t prove it. And
someone else replied to the bug he could not reproduce the problem. So
I’d say it’s something I’m doing, but I find it off that both the WinXP
I’ve been using forever and a much newer Win7 do the same thing.
UPDATE (2003-06-07): Here is a third way that’s trivial! It seems
there are built-in but undocumented environment variables %time% and
%date% in Windows 2000. I have not tested other platforms (let me know
if you do). Due to the format, you can’t easily use the date in file
copy operations (for example), but the time should be OK. And it’s by
far the easiest option if you are just going to display (writing to a
log file or something).
C:\> echo %date% %time%
Sat 06/07/2003 18:32:30.52
Windows Trivial
@echo off
REM Play with W2K date/time env. vars.
echo The date: %date%
echo The time: %time%
UPDATE (2006-05-11): Here is a another trivial way! Thanks to
Richard Blake (RBlake {at} nea {DOT} org) for this great hack. In
addition to the above %time% and %date% variables, there is a
%VAR:offset,len% construct documented for the SET command, which works
elsewhere. As above, the use of a two digit time code can mess you up,
but for just the date it will work very well. Code to deal with non zero
padded hours is left as an exercise for the reader.
C:\tmp> set MyNewFileName=%DATE:~10,4%%DATE:~4,2%%DATE:~7,2%%TIME:~0,2%%TIME:~3,2%%TIME:~6,2%
C:\tmp> echo %MyNewFileName%
20060511 21921
This RedmondMag.com Backup Basics in Windows Server 2008
R2
article expands on the same method, but they are not portable because
they depend on how your system time is displayed, and that will vary
from machine to machine based on locale and user preference. For
example, I loath any date/time format except for
ISO8601 so I have my Windows
formats set as close to that as possible, which then breaks the
assumptions in the first block:
@echo off
REM date_demo.cmd--Simple date parsing demoREM 2012-03-10echo Current dates (Windows default date format for US)
set year=%date:~10,4%
set month=%date:~4,2%
set day=%date:~7,2%
set hour=%time:~0,2%
set min=%time:~3,2%
set sec=%time:~6,2%
echo Date: %date% Time: %time%
echo ISO-8601: %year%-%month%-%day%_%hour%:%min%:%sec%
echo.
echo Current dates (ISO-8601 date format)
REM 2012-03-10set year=%date:~0,4%
set month=%date:~5,2%
set day=%date:~8,2%
set hour=%time:~0,2%
set min=%time:~3,2%
set sec=%time:~6,2%
echo Date: %date% Time: %time%
echo ISO-8601: %year%-%month%-%day%_%hour%:%min%:%sec%
Other Ways
windate v1.0 sometime in 2001 or
2002
Native Windows Date commands
unixdate v1.0 sometime in 2001
or 2002
Using a UNIX date command in Windows
Getting Input
There are various tools like ask.exe and choice.exe that allow you to
get input. There there’s an even easier, although undocumented, way: set
/P. As in:
set/P MyAnswer=Your Prompt Here!
That prompts the user with “Your Prompt Here!” and puts whatever they
type into %MyAnswer%. Very cool.
Simple Utilities
Except for FindZero.bat, all of these batch files will work under DOS,
or any Windows.
DOS Commands are not case sensitive, unlike UNIX commands.
An “@” as the first character of a line prevents the command from
echoing whether echo is on or off.
echo. will echo a blank line (CRLF).
Command line parameters are specified with %1, %2, etc. not $1, $2
as in UNIX.
%0 is the name of the program, as invoked. In other words, if you
type “mybatch” %0 will be “mybatch”. If you type
“c:\utils\mybatch.bat” %0 will be “c:\utils\mybatch.bat”.
^G is a ‘control G’ which makes the console beep. This tiny batch
file has a ^G in in, which you can
cut & paste into scripts. There are lots of other ways to get
control characters into files, but they depend on your OS and text
editor. In most DOS windows, holding down the <ALT> key while
typing the ASCII code on the numeric keypad will produce that
character. ^G is 007, you you hold down <ALT>, type 007 on the
numeric keypad, then release <ALT> to get a beep.
Tim Hill’s Windows NT Shell
Scripting
for much more information and detail. Much of the book applies to
DOS and Windows (other than NT) as well.
Requires ANSI.sys, included with DOS & Windows, or PC Magazine’s free
AnsiCom.
@echo off
REM SPrompt.bat -- Dynamically Set PromptREM Created sometime in 1992REM 03-Mar-1998 JPVREM 19-Feb-1999 JPV Added "neat" prompt from JPS mail listREM Neat PROMPT `$+[%user@$P]%@EXECSTR[if %@LEN[%_CWD] GT 20 ECHOS $_:$s]`prompt $e[0;33;1;44m$P$e[36;44m$G $e[0;37;44m
ifnot"%1"==""prompt $e[32;1m%1 %2 %3 %4 %5 %6 %7 %8 %9$_$e[33m%prompt%
rem prompt $e[s$e[1;7f$e[0;45;37;1m$e[K($z) $d $t$e[u$e[1m$P$e[0m$G $e[0mrem set WINPMT=$e[0;33;1;44mEXIT to Windows$_$P$e[36;44m$G $e[0;37;44mif {%OS%}=={Windows_NT} prompt $P$G
WhoAmI.bat
Requires Microsoft Networking to be installed and active, and the DOS
find command. If you have a UNIX find command in the path, you’ll
probably get a “No such file or directory” error.
@echo off
echo.
net config /yes | find "name"echo.
pause
Sending e-mail
Something else that is taken for granted on UNIX is the ability to send
e-mail from the command line or a script. As usual, windows makes this a
challenge. There are a few free and commercial solutions for this,
including but not limited to the following list (I’ve only ever used
Blat):
Blat “is a Win32 command line utility that
sends eMail using the SMTP or NNTP protocols.”
NTsendmail “is Highly Acclaimed UNIX
Sendmail replacement for NT. NTsendmail is realeased under the GNU
Public License. NTsendmail was designed to enable script writers to
use their UNIX CGIs on Windows 95/98/NT/2000.”
And I’m sure there’s at least one or two Perl modules that can do
this.
Upload from Frontpage to your ISP
I used to use MS FrontPage to maintain this site (don’t ask me why). My
old ISP did not support FrontPage or its extensions, for excellent
security reasons. Using FrontPage to create pages, then uploading them
to a hosted site is a gigantic pain in the ass because of the way
FrontPage keeps all of its proprietary information in various
“_VTI_CNF” and other subdirectories. So simply zipping up the
directories and dumping them onto a host is not ideal. So I came up with
the following solution.
The old code I posted to a Netaxs news group was WRONG in places! THIS
stuff works.
I don’t use this any more, I use
Hugo and
Relearn.
Open issues are:
If you do a ZIP, and upload it, but do not delete it from the PC,
the next run will append, not overwrite. Your “diff” zip will just
keep growing. You need to manually delete the zip file after each
run is successful.
Since find is working off of the modification time, if you move a
file into the structure that has not been modified (i.e. a tool or
utility program) it will not be picked up by a diff. You have to add
that to the zip manually.
Unzip upload.zip and set permissions on www directory.
Windows Scripting Resources
Master and Command Line:
“Using the Windows GUI is fine–if you want to go slow. Learn to use
the command line and move into the administration fast lane.”
This content is old! It’s still useful, but it’s old, and there may be bit rot, newer/better tools or ways to do things. Sanity check and do your research.
This content is obsolete, but I am leaving it here as a historical reference.
Introduction
Welcome to the Windows port of Logcheck (now called LogSentry), the
famous UNIX log processing tool. Psionic was bought by Cisco who has
moved the cool Abacus tools, including LogSentry, to
http://sourceforge.net/projects/sentrytools/
As you probably know if you are bothering to read this, LogSentry helps
spot problems and security violations in your logfiles automatically and
will send the results to you in e-mail. However, it can only work with
what it’s given. I personally find the Windows Event Logs to be verbose,
yet un-informative. So when you get e-mailed messages with three or 4
lines on arcane gibberish, remember that it’s the same information as
you would see in the Event Log, except it’s in a slightly different
format, and you are actually SEEING it! (Of course, you would have
reviewed the Event Logs anyway, right?) :-)
The beginning of each log entry contains the name of the Event Log, the
date, and the time, like: “DIR,7/20/2001 11:52:12”. this is followed by
the event details. The three letter codes for the Event Logs are:
APP Application
SEC Security
SYS System
DNS DNS (Win2000 Server(?) only)
DIR Directory Service (Win2000 Server(?) only)
RPL File Replication (Win2000 Server(?) only)
It will help to look for those codes to isolate one event from the next,
since your mailer will probably wrap the lines.
Also, the MS API for the Event Logs will return the application log if
the Event Log being asked for does not exist. Once it returns the
handle, there isn’t any way that DumpEvt can tell which log is being
read. For example, if you dump RPL on an NT box, or DNS on W2k Pro it
dumps the APP log instead. This causes duplicate entries in the capture
file. The only work-around right now is to edit wrapper.cmd and REM out
the Event Logs that do not exist on that machine.
While the actual logcheck.sh script could have been ported to the
CMD.EXE shell, I thought it was much more efficient and effective to
make as few changes to that as possible, and instead create a “wrapper”
program to translate “Windows” into “UNIX.”
I gave the wrapper the highly imaginative and interesting name of
“wrapper.cmd.” Note that it only runs under Windows NT and 2000 (and
probably XP though I have not tested that). In short, logcheck is
pointless without logs, which means the Event Logs. Since the Win9x
series doesn’t have those…
See the “Tools-Readme.txt” for details about what tools are needed.
Paths/Dirs
I struggled with the default directory location for a while. I *hate*
programs that put themselves in the %SYSTEMROOT% (e.g. c:\winnt)
directory. However, I wanted to be a little obscure, so I thought
c:\etc was a little too obvious. And I wanted to be reliable, so
“c:\Program Files” or “c:\Documents and Settings” were both too long,
and had annoying spaces. C:\Progra~1 and c:\Docume~1 are not 100%
reliable. So %SYSTEMROOT%\etc it is… Of course, you can change that
if you want.
%SYSTEMROOT%\etc
%SYSTEMROOT%\etc\lcwin Keyword files
%SYSTEMROOT%\etc\bin Binaries
%SYSTEMROOT%\etc\tmp Secured Temp directory
%SYSTEMROOT%\etc\bin Documentation and help files
Note the word “secured” above. YOU need to set proper permissions on
those directories so that the account under which logcheck runs as the
proper access, but “Everyone” else does not. See the INSTALL-Windows.txt
file for more details.
At the heart of logcheck are two basic things, grep and mail. If they
fail, so will logcheck. While in the UNIX world you can assume the
ability to send mail from the command line, in the Windows world you
can’t. So make sure Blat is working. Again, see the INSTALL-Windows.txt
file for more details.
License
This software is released under the GPLv2. See the included LICENSE
file.
Credits
First, to Marcus J. Ranum and Fred Avolio for writing the deceptively
simple yet brilliant original frequentcheck.sh for TIS Gauntlet.
Second, to Craig Rowland for applying the idea to system logs.
Third, to all the people who wrote or ported the tools I needed to get
logcheck to run under Windows, notably K. M. Syring, the guys at
Somarsoft, and Tim Charron.
Fourth, to G.P. and I.P. who sent me logs to test to help me tune the
keyword files, and answered various other questions.
I should also mention the guys at Bastille Linux, whos style I’ve copied
here a bit in the Setup program “interview.”
Resources
Finally, a few notes about other tools or resources that may be of
interest, if you’ve managed to read this far.
I’ve found the following book to be essential in other Windows scripting
projects, and it proved helpful in this project as well. For anyone who
thinks you couldn’t possibly find enough material in the old DOS batch
file language to write a book about it – that’s what I thought when I
saw it. Check it out! While not nearly as powerful, flexible or easy to
use (though some would argue that last) as UNIX shells, Windows’s
CMD.EXE is actually a lot more powerful than you think. Forget about
Command.com though…
Windows NT Shell Scripting, by Tim Hill
New Riders Publishing, Paperback, Published April 1998, 377 pages, ISBN
1578700477
Other Methods of accessing the NT Event logs (not free):
Win2K Server ResKit Perl scripts: EventLog.pl & EventQuery.pl
NT (etc.) ResKit Elogdmp.exe event log dumper
Other Tools of interest (free):
Snare
is a Windows service to send NT Event Logs to a SysLog server. (This
used to be called Backlog, the latest version of which is Backlog
1.9b and which is archived here and
here.
Backlog is much more simple than Snare and may still be quite
useful.)
NTSysLog is another
Windows service to send NT/2000 Event Logs to a SysLog server, but
the latest release seems to be 1.13 from October 21, 2002.
NTLast
is a UNIX-like “last” command for NT Event Log (ntobjectives.com)
Log Analysis:
The Log Analysis site, maintained
by Tina Bird and Marcus Ranum. The whole site is great, but of
special interest in the context of Winlogcheck are the following:
