I often spend countless hours pouring through server logs. If I think there’s a possible application issue I’ll be sifting through system and application logs. If there’s a connectivity issue I may be digging through Apache logs. Either way, without the proper tools it can be a tedious job. Oh heck, who am I kidding? It’s tedious regardless.
Read More
Tag: awk
Linux Command Line and Shell Scripting Bible
This is the third edition of the book and I’ve been well pleased with it. The first part of the book is dedicated to the command line and gaining familiarization with the various commands, file system, navigation, etc. The real meat of the book begins with part 2 where you actually start scripting.
As a software developer by trade, Linux scripting is something that I find myself naturally drawn to. All of the systems I utilize at work aside from my desktop run Red Hat Enterprise Linux so I spend a great deal of time in terminal sessions. Many of the things I do are extremely repetitive and while I can’t (due to security limitations) put custom scripts in every environment I’m slowly working on a library that can help be do those mundane tasks quickly and with reproducible results.
Part 3 covers advanced scripting and the use of tools like sed and awk. Both of these along with regular expressions play a big role in my daily work. I really can’t imagine how difficult it would be for me to evaluate logs and generate reports for executive management without tools like these to make the job significantly easier and faster. I can easily scan massive business logs for key errors and use awk to generate an extract of the critical information I need to review. sed is a great tool when I need to make wholesale changes to configuration files as an example.
Part 4 gets you creating more real-world scripts. You’ll learn how to do things like archive logs. This is a big part of how our system is setup. We only keep a few days’ worth of logs “active” due to their size (we have an extremely high-volume application). Everything else gets compressed and archived to another location. You’ll also learn how to generate reports that can interact with the web, databases, etc.
There are a ton of books on scripting under Linux out there but this is one of my favorites.
# ./cli-book.sh I <3 the book 'Linux Command Line and Shell Scripting Bible'
sed & awk
One of the tasks I find myself doing a lot these days is reviewing web and application logs. Working in a Windows environment with a product responsible for literally billions of transactions each year it’s difficult if not impossible to do this. A single days’ log can easily exceed the capabilities of many Windows text editors and those that can manage it aren’t as fast as Unix tools.
I don’t have a huge need for sed, the Unix stream editor, but I make extensive use of awk. grep is also another tool that plays an important role in my toolbox. Since awk and grep both make use of regular expressions it was nice to find a chapter dedicated to the subject.
There are three chapters dedicated to sed and four geared towards awk. Each utility also has its own quick-reference guide.
The book hasn’t been updated in some time (pre-2000) but let’s face it, not a whole lot changes with basic Unix functionality. I found the examples to be concise and well-suited to helping me grasp the concepts. The material goes much deeper than what I need for my daily tasks but it’s nice to know that a lot of power is at my fingertips should I need it.
If you’re new to ‘nix system administration or like me and trying to clear off the dust this is a great reference book. Grab a copy!