Go to the first, previous, next, last section, table of contents.

"To boldly go where no man has gone before" is a Registered Trademark of Paramount Pictures Corporation.

Copyright (C) 1989, 1991, 92, 93, 96, 97 Free Software Foundation, Inc.

This is Edition 1.0.3 of Effective AWK Programming,
for the 3.0.3 (or later) version of the GNU implementation of AWK.

Published jointly by:

@multitable {Specialized Systems Consultants, Inc. (SSC)} {Boston, MA 02111-1307 USA}

- Specialized Systems Consultants, Inc. (SSC) @tab Free Software Foundation
- PO Box 55549 @tab 59 Temple Place -- Suite 330
- Seattle, WA 98155 USA @tab Boston, MA 02111-1307 USA
- Phone: +1-206-782-7733 @tab Phone: +1-617-542-5942
- Fax: +1-206-782-7191 @tab Fax: +1-617-542-2652
- E-mail: `sales@ssc.com` @tab E-mail: `gnu@prep.ai.mit.edu`
- URL: `http://www.ssc.com/` @tab URL: `http://www.fsf.org/`

  ISBN 1-57831-000-8
  Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies. Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one. Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modified versions, except that this permission notice may be stated in a translation approved by the Foundation.

  Cover art by Amy Wells Wood.

  To Miriam, for making me complete.

  To Chana, for the joy you bring us.

  To Rivka, for the exponential increase.

  To Nachum, for the added dimension.

  - Preface: What this book is about; brief history and acknowledgements.
  - What Is Awk: What is the `awk` language; using this book.
  - Getting Started: A basic introduction to using `awk`. How to run an `awk` program. Command line syntax.
  - One-liners: Short, sample `awk` programs.
  - Regexp: All about matching things using regular expressions.
  - Reading Files: How to read files and manipulate fields.
  - Printing: How to print using `awk`. Describes the `print` and `printf` statements. Also describes redirection of output.
  - Expressions: Expressions are the basic building blocks of statements.
  - Patterns and Actions: Overviews of patterns and actions.
  - Statements: The various control statements are described in detail.
  - Built-in Variables: Built-in Variables
  - Arrays: The description and use of arrays. Also includes array-oriented control statements.
  - Built-in: The built-in functions are summarized here.
  - User-defined: User-defined functions are described in detail.
  - Invoking Gawk: How to run `gawk`.
  - Library Functions: A Library of `awk` Functions.
  - Sample Programs: Many `awk` programs with complete explanations.
  - Language History: The evolution of the `awk` language.
  - Gawk Summary: `gawk` Options and Language Summary.
  - Installation: Installing `gawk` under various operating systems.
  - Notes: Something about the implementation of `gawk`.
  - Glossary: An explanation of some unfamiliar terms.
  - Copying: Your right to copy and distribute `gawk`.
  - Index: Concept and Variable Index.
  - History: The history of `gawk` and `awk`.
  - Manual History: Brief history of the GNU project and this book.
  - Acknowledgements: Acknowledgements.
  - This Manual: Using this book. Includes sample input files that you can use.
  - Conventions: Typographical Conventions.
  - Sample Data Files: Sample data files for use in the `awk` programs illustrated in this book.
  - Names: What name to use to find `awk`.
  - Running gawk: How to run `gawk` programs; includes command line syntax.
  - One-shot: Running a short throw-away `awk` program.
  - Read Terminal: Using no input files (input from terminal instead).
  - Long: Putting permanent `awk` programs in files.
  - Executable Scripts: Making self-contained `awk` programs.
  - Comments: Adding documentation to `gawk` programs.
  - Very Simple: A very simple example.
  - Two Rules: A less simple one-line example with two rules.
  - More Complex: A more complex example.
  - Statements/Lines: Subdividing or combining statements into lines.
  - Other Features: Other Features of `awk`.
  - When: When to use `gawk` and when to use other things.
  - Regexp Usage: How to Use Regular Expressions.
  - Escape Sequences: How to write non-printing characters.
  - Regexp Operators: Regular Expression Operators.
  - GNU Regexp Operators: Operators specific to GNU software.
  - Case-sensitivity: How to do case-insensitive matching.
  - Leftmost Longest: How much text matches.
  - Computed Regexps: Using Dynamic Regexps.
  - Records: Controlling how data is split into records.
  - Fields: An introduction to fields.
  - Non-Constant Fields: Non-constant Field Numbers.
  - Changing Fields: Changing the Contents of a Field.
  - Field Separators: The field separator and how to change it.
  - Basic Field Splitting: How fields are split with single characters or simple strings.
  - Regexp Field Splitting: Using regexps as the field separator.
  - Single Character Fields: Making each character a separate field.
  - Command Line Field Separator: Setting `FS` from the command line.
  - Field Splitting Summary: Some final points and a summary table.
  - Constant Size: Reading constant width data.
  - Multiple Line: Reading multi-line records.
  - Getline: Reading files under explicit program control using the `getline` function.
  - Getline Intro: Introduction to the `getline` function.
  - Plain Getline: Using `getline` with no arguments.
  - Getline/Variable: Using `getline` into a variable.
  - Getline/File: Using `getline` from a file.
  - Getline/Variable/File: Using `getline` into a variable from a file.
  - Getline/Pipe: Using `getline` from a pipe.
  - Getline/Variable/Pipe: Using `getline` into a variable from a pipe.
  - Getline Summary: Summary Of `getline` Variants.
  - Print: The `print` statement.
  - Print Examples: Simple examples of `print` statements.
  - Output Separators: The output separators and how to change them.
  - OFMT: Controlling Numeric Output With `print`.
  - Printf: The `printf` statement.
  - Basic Printf: Syntax of the `printf` statement.
  - Control Letters: Format-control letters.
  - Format Modifiers: Format-specification modifiers.
  - Printf Examples: Several examples.
  - Redirection: How to redirect output to multiple files and pipes.
  - Special Files: File name interpretation in `gawk`. `gawk` allows access to inherited file descriptors.
  - Close Files And Pipes: Closing Input and Output Files and Pipes.
  - Constants: String, numeric, and regexp constants.
  - Scalar Constants: Numeric and string constants.
  - Regexp Constants: Regular Expression constants.
  - Using Constant Regexps: When and how to use a regexp constant.
  - Variables: Variables give names to values for later use.
  - Using Variables: Using variables in your programs.
  - Assignment Options: Setting variables on the command line and a summary of command line syntax. This is an advanced method of input.
  - Conversion: The conversion of strings to numbers and vice versa.
  - Arithmetic Ops: Arithmetic operations (``+'`, ``-'`, etc.)
  - Concatenation: Concatenating strings.
  - Assignment Ops: Changing the value of a variable or a field.
  - Increment Ops: Incrementing the numeric value of a variable.
  - Truth Values: What is "true" and what is "false".
  - Typing and Comparison: How variables acquire types, and how this affects comparison of numbers and strings with ``<'`, etc.
  - Boolean Ops: Combining comparison expressions using boolean operators ``||'` ("or"), ``&&'` ("and") and ``!'` ("not").
  - Conditional Exp: Conditional expressions select between two subexpressions under control of a third subexpression.
  - Function Calls: A function call is an expression.
  - Precedence: How various operators nest.
  - Pattern Overview: What goes into a pattern.
  - Kinds of Patterns: A list of all kinds of patterns.
  - Regexp Patterns: Using regexps as patterns.
  - Expression Patterns: Any expression can be used as a pattern.
  - Ranges: Pairs of patterns specify record ranges.
  - BEGIN/END: Specifying initialization and cleanup rules.
  - Using BEGIN/END: How and why to use BEGIN/END rules.
  - I/O And BEGIN/END: I/O issues in BEGIN/END rules.
  - Empty: The empty pattern, which matches every record.
  - Action Overview: What goes into an action.
  - If Statement: Conditionally execute some `awk` statements.
  - While Statement: Loop until some condition is satisfied.
  - Do Statement: Do specified action while looping until some condition is satisfied.
  - For Statement: Another looping statement, that provides initialization and increment clauses.
  - Break Statement: Immediately exit the innermost enclosing loop.
  - Continue Statement: Skip to the end of the innermost enclosing loop.
  - Next Statement: Stop processing the current input record.
  - Nextfile Statement: Stop processing the current file.
  - Exit Statement: Stop execution of `awk`.
  - User-modified: Built-in variables that you change to control `awk`.
  - Auto-set: Built-in variables where `awk` gives you information.
  - ARGC and ARGV: Ways to use `ARGC` and `ARGV`.
  - Array Intro: Introduction to Arrays
  - Reference to Elements: How to examine one element of an array.
  - Assigning Elements: How to change an element of an array.
  - Array Example: Basic Example of an Array
  - Scanning an Array: A variation of the `for` statement. It loops through the indices of an array's existing elements.
  - Delete: The `delete` statement removes an element from an array.
  - Numeric Array Subscripts: How to use numbers as subscripts in `awk`.
  - Uninitialized Subscripts: Using Uninitialized variables as subscripts.
  - Multi-dimensional: Emulating multi-dimensional arrays in `awk`.
  - Multi-scanning: Scanning multi-dimensional arrays.
  - Calling Built-in: How to call built-in functions.
  - Numeric Functions: Functions that work with numbers, including `int`, `sin` and `rand`.
  - String Functions: Functions for string manipulation, such as `split`, `match`, and `sprintf`.
  - I/O Functions: Functions for files and shell commands.
  - Time Functions: Functions for dealing with time stamps.
  - Definition Syntax: How to write definitions and what they mean.
  - Function Example: An example function definition and what it does.
  - Function Caveats: Things to watch out for.
  - Return Statement: Specifying the value a function returns.
  - Options: Command line options and their meanings.
  - Other Arguments: Input file names and variable assignments.
  - AWKPATH Variable: Searching directories for `awk` programs.
  - Obsolete: Obsolete Options and/or features.
  - Undocumented: Undocumented Options and Features.
  - Known Bugs: Known Bugs in `gawk`.
  - Portability Notes: What to do if you don't have `gawk`.
  - Nextfile Function: Two implementations of a `nextfile` function.
  - Assert Function: A function for assertions in `awk` programs.
  - Round Function: A function for rounding if `sprintf` does not do it correctly.
  - Ordinal Functions: Functions for using characters as numbers and vice versa.
  - Join Function: A function to join an array into a string.
  - Mktime Function: A function to turn a date into a timestamp.
  - Gettimeofday Function: A function to get formatted times.
  - Filetrans Function: A function for handling data file transitions.
  - Getopt Function: A function for processing command line arguments.
  - Passwd Functions: Functions for getting user information.
  - Group Functions: Functions for getting group information.
  - Library Names: How to best name private global variables in library functions.
  - Clones: Clones of common utilities.
  - Cut Program: The `cut` utility.
  - Egrep Program: The `egrep` utility.
  - Id Program: The `id` utility.
  - Split Program: The `split` utility.
  - Tee Program: The `tee` utility.
  - Uniq Program: The `uniq` utility.
  - Wc Program: The `wc` utility.
  - Miscellaneous Programs: Some interesting `awk` programs.
  - Dupword Program: Finding duplicated words in a document.
  - Alarm Program: An alarm clock.
  - Translate Program: A program similar to the `tr` utility.
  - Labels Program: Printing mailing labels.
  - Word Sorting: A program to produce a word usage count.
  - History Sorting: Eliminating duplicate entries from a history file.
  - Extract Program: Pulling out programs from Texinfo source files.
  - Simple Sed: A Simple Stream Editor.
  - Igawk Program: A wrapper for `awk` that includes files.
  - V7/SVR3.1: The major changes between V7 and System V Release 3.1.
  - SVR4: Minor changes between System V Releases 3.1 and 4.
  - POSIX: New features from the POSIX standard.
  - BTL: New features from the Bell Laboratories version of `awk`.
  - POSIX/GNU: The extensions in `gawk` not in POSIX `awk`.
  - Command Line Summary: Recapitulation of the command line.
  - Language Summary: A terse review of the language.
  - Variables/Fields: Variables, fields, and arrays.
  - Fields Summary: Input field splitting.
  - Built-in Summary: `awk`'s built-in variables.
  - Arrays Summary: Using arrays.
  - Data Type Summary: Values in `awk` are numbers or strings.
  - Rules Summary: Patterns and Actions, and their component parts.
  - Pattern Summary: Quick overview of patterns.
  - Regexp Summary: Quick overview of regular expressions.
  - Actions Summary: Quick overview of actions.
  - Operator Summary: `awk` operators.
  - Control Flow Summary: The control statements.
  - I/O Summary: The I/O statements.
  - Printf Summary: A summary of `printf`.
  - Special File Summary: Special file names interpreted internally.
  - Built-in Functions Summary: Built-in numeric and string functions.
  - Time Functions Summary: Built-in time functions.
  - String Constants Summary: Escape sequences in strings.
  - Functions Summary: Defining and calling functions.
  - Historical Features: Some undocumented but supported "features".
  - Gawk Distribution: What is in the `gawk` distribution.
  - Getting: How to get the distribution.
  - Extracting: How to extract the distribution.
  - Distribution contents: What is in the distribution.
  - Unix Installation: Installing `gawk` under various versions of Unix.
  - Quick Installation: Compiling `gawk` under Unix.
  - Configuration Philosophy: How it's all supposed to work.
  - VMS Installation: Installing `gawk` on VMS.
  - VMS Compilation: How to compile `gawk` under VMS.
  - VMS Installation Details: How to install `gawk` under VMS.
  - VMS Running: How to run `gawk` under VMS.
  - VMS POSIX: Alternate instructions for VMS POSIX.
  - PC Installation: Installing and Compiling `gawk` on MS-DOS and OS/2
  - Atari Installation: Installing `gawk` on the Atari ST.
  - Atari Compiling: Compiling `gawk` on Atari
  - Atari Using: Running `gawk` on Atari
  - Amiga Installation: Installing `gawk` on an Amiga.
  - Bugs: Reporting Problems and Bugs.
  - Other Versions: Other freely available `awk` implementations.
  - Compatibility Mode: How to disable certain `gawk` extensions.
  - Additions: Making Additions To `gawk`.
  - Adding Code: Adding code to the main body of `gawk`.
  - New Ports: Porting `gawk` to a new operating system.
  - Future Extensions: New features that may be implemented one day.
  - Improvements: Suggestions for improvements by volunteers.

  # Preface

  This book teaches you about the `awk` language and how you can use it effectively. You should already be familiar with basic system commands, such as `cat` and `ls`,(1) and basic shell facilities, such as Input/Output (I/O) redirection and pipes.

  Implementations of the `awk` language are available for many different computing environments. This book, while describing the `awk` language in general, also describes a particular implementation of `awk` called `gawk` (which stands for "GNU Awk"). `gawk` runs on a broad range of Unix systems, ranging from 80386 PC-based computers, up through large scale systems, such as Crays. `gawk` has also been ported to MS-DOS and OS/2 PC's, Atari and Amiga micro-computers, and VMS.

  - History: The history of `gawk` and `awk`.
  - Manual History: Brief history of the GNU project and this book.
  - Acknowledgements: Acknowledgements.

  ## History of `awk` and `gawk`

  The name `awk` comes from the initials of its designers: Alfred V. Aho, Peter J. Weinberger, and Brian W. Kernighan. The original version of `awk` was written in 1977 at AT&T Bell Laboratories. In 1985 a new version made the programming language more powerful, introducing user-defined functions, multiple input streams, and computed regular expressions. This new version became generally available with Unix System V Release 3.1. The version in System V Release 4 added some new features and also cleaned up the behavior in some of the "dark corners" of the language. The specification for `awk` in the POSIX Command Language and Utilities standard further clarified the language based on feedback from both the `gawk` designers, and the original Bell Labs `awk` designers.

  The GNU implementation, `gawk`, was written in 1986 by Paul Rubin and Jay Fenlason, with advice from Richard Stallman. John Woods contributed parts of the code as well. In 1988 and 1989, David Trueman, with help from Arnold Robbins, thoroughly reworked `gawk` for compatibility with the newer `awk`. Current development focuses on bug fixes, performance improvements, standards compliance, and occasionally, new features.

  ## The GNU Project and This Book

  The Free Software Foundation (FSF) is a non-profit organization dedicated to the production and distribution of freely distributable software. It was founded by Richard M. Stallman, the author of the original Emacs editor. GNU Emacs is the most widely used version of Emacs today.

  The GNU project is an on-going effort on the part of the Free Software Foundation to create a complete, freely distributable, POSIX compliant computing environment. (GNU stands for "GNU's not Unix".) The FSF uses the "GNU General Public License" (or GPL) to ensure that source code for their software is always available to the end user. A copy of the GPL is included for your reference (see section GNU GENERAL PUBLIC LICENSE). The GPL applies to the C language source code for `gawk`.

  A shell, an editor (Emacs), highly portable optimizing C, C++, and Objective-C compilers, a symbolic debugger, and dozens of large and small utilities (such as `gawk`), have all been completed and are freely available. As of this writing (early 1997), the GNU operating system kernel (the HURD), has been released, but is still in an early stage of development.

  Until the GNU operating system is more fully developed, you should consider using Linux, a freely distributable, Unix-like operating system for 80386, DEC Alpha, Sun SPARC and other systems. There are many books on Linux. One freely available one is Linux Installation and Getting Started, by Matt Welsh. Many Linux distributions are available, often in computer stores or bundled on CD-ROM with books about Linux. (There are three other freely available, Unix-like operating systems for 80386 and other systems, NetBSD, FreeBSD,and OpenBSD. All are based on the 4.4-Lite Berkeley Software Distribution, and they use recent versions of `gawk` for their versions of `awk`.)

  This book you are reading now is actually free. The information in it is freely available to anyone, the machine readable source code for the book comes with `gawk`, and anyone may take this book to a copying machine and make as many copies of it as they like. (Take a moment to check the copying permissions on the Copyright page.)

  If you paid money for this book, what you actually paid for was the book's nice printing and binding, and the publisher's associated costs to produce it. We have made an effort to keep these costs reasonable; most people would prefer a bound book to over 330 pages of photo-copied text that would then have to be held in a loose-leaf binder (not to mention the time and labor involved in doing the copying). The same is true of producing this book from the machine readable source; the retail price is only slightly more than the cost per page of printing it on a laser printer.

  This book itself has gone through several previous, preliminary editions. I started working on a preliminary draft of The GAWK Manual, by Diane Close, Paul Rubin, and Richard Stallman in the fall of 1988. It was around 90 pages long, and barely described the original, "old" version of `awk`. After substantial revision, the first version of the The GAWK Manual to be released was Edition 0.11 Beta in October of 1989. The manual then underwent more substantial revision for Edition 0.13 of December 1991. David Trueman, Pat Rankin, and Michal Jaegermann contributed sections of the manual for Edition 0.13. That edition was published by the FSF as a bound book early in 1992. Since then there have been several minor revisions, notably Edition 0.14 of November 1992 that was published by the FSF in January of 1993, and Edition 0.16 of August 1993.

  Edition 1.0 of Effective AWK Programming represents a significant re-working of The GAWK Manual, with much additional material. The FSF and I agree that I am now the primary author. I also felt that it needed a more descriptive title.

  Effective AWK Programming will undoubtedly continue to evolve. An electronic version comes with the `gawk` distribution from the FSF. If you find an error in this book, please report it! See section Reporting Problems and Bugs, for information on submitting problem reports electronically, or write to me in care of the FSF.

  ## Acknowledgements

  I would like to acknowledge Richard M. Stallman, for his vision of a better world, and for his courage in founding the FSF and starting the GNU project.

  The initial draft of The GAWK Manual had the following acknowledgements:

  > Many people need to be thanked for their assistance in producing this manual. Jay Fenlason contributed many ideas and sample programs. Richard Mlynarik and Robert Chassell gave helpful comments on drafts of this manual. The paper A Supplemental Document for `awk` by John W. Pierce of the Chemistry Department at UC San Diego, pinpointed several issues relevant both to `awk` implementation and to this manual, that would otherwise have escaped us.

  The following people provided many helpful comments on Edition 0.13 of The GAWK Manual: Rick Adams, Michael Brennan, Rich Burridge, Diane Close, Christopher ("Topher") Eliot, Michael Lijewski, Pat Rankin, Miriam Robbins, and Michal Jaegermann.

  The following people provided many helpful comments for Edition 1.0 of Effective AWK Programming: Karl Berry, Michael Brennan, Darrel Hankerson, Michal Jaegermann, Michael Lijewski, and Miriam Robbins. Pat Rankin, Michal Jaegermann, Darrel Hankerson and Scott Deifik updated their respective sections for Edition 1.0.

  Robert J. Chassell provided much valuable advice on the use of Texinfo. He also deserves special thanks for convincing me not to title this book How To Gawk Politely. Karl Berry helped significantly with the TeX part of Texinfo.

  David Trueman deserves special credit; he has done a yeoman job of evolving `gawk` so that it performs well, and without bugs. Although he is no longer involved with `gawk`, working with him on this project was a significant pleasure.

  Scott Deifik, Darrel Hankerson, Kai Uwe Rommel, Pat Rankin, and Michal Jaegermann (in no particular order) are long time members of the `gawk` "crack portability team." Without their hard work and help, `gawk` would not be nearly the fine program it is today. It has been and continues to be a pleasure working with this team of fine people.

  Jeffrey Friedl provided invaluable help in tracking down a number of last minute problems with regular expressions in `gawk` 3.0.

  David and I would like to thank Brian Kernighan of Bell Labs for invaluable assistance during the testing and debugging of `gawk`, and for help in clarifying numerous points about the language. We could not have done nearly as good a job on either `gawk` or its documentation without his help.

  I would like to thank Marshall and Elaine Hartholz of Seattle, and Dr. Bert and Rita Schreiber of Detroit for large amounts of quiet vacation time in their homes, which allowed me to make significant progress on this book and on `gawk` itself. Phil Hughes of SSC contributed in a very important way by loaning me his laptop Linux system, not once, but twice, allowing me to do a lot of work while away from home.

  Finally, I must thank my wonderful wife, Miriam, for her patience through the many versions of this project, for her proof-reading, and for sharing me with the computer. I would like to thank my parents for their love, and for the grace with which they raised and educated me. I also must acknowledge my gratitude to G-d, for the many opportunities He has sent my way, as well as for the gifts He has given me with which to take advantage of those opportunities.

  Arnold Robbins
  Atlanta, Georgia
  February, 1997

  Go to the first, previous, next, last section, table of contents.