How to Use the AWK Command in Linux for Text Processing

You’ve got a 50 MB log file, a CSV export from some tool, or the output of a command, and you only need two columns, a total, or the lines that mention “error”. Opening it in a spreadsheet feels like overkill, and grep alone can’t do the math. This post shows you how to use the AWK command in Linux to pull out, filter and summarize text data with short one-liners and a small script.

What the AWK command in Linux does

AWK is a small programming language and command-line tool for processing structured text, like files made of lines (records) and columns (fields). It ships with almost every Unix-like system, including Linux and macOS. The name comes from the initials of its three creators at Bell Labs: Alfred Aho, Peter Weinberger and Brian Kernighan.

An AWK program is a list of pattern-action pairs. The pattern decides which lines to pick, and the action says what to do with them. If you skip the pattern, the action runs on every line. If you skip the action, AWK prints the matching lines.

You’ll usually write AWK as a one-liner right in the terminal. When the logic grows, you can save it in a separate file and run it as a script. I’d suggest starting with one-liners; you’ll be surprised how far they take you.

AWK features you’ll use most

AWK packs a lot into a tiny syntax. These are the parts that do the heavy lifting:

• Automatic field splitting: AWK splits each line into fields using a delimiter, which is whitespace by default. You get the fields as $1, $2 and so on, and $0 is the whole line.
• Built-in variables: NF holds the number of fields in the current line, NR is the current line number, and FS is the input field separator.
• Regular expressions: you can match lines with patterns like /error/, which makes filtering on complex conditions easy.
• Control structures: if/else, loops and other familiar logic let you go beyond simple filtering.
• User-defined functions: you can wrap reusable logic in your own functions once your scripts get bigger.

When to reach for AWK

AWK shines whenever your data has a predictable layout of lines and columns. In practice, that covers four common jobs:

• Data extraction: pick out only the records or fields that match your criteria.
• Data transformation: run calculations, reformat text, aggregate values or build quick reports.
• Log analysis: filter large log files, spot patterns and pull out the details you need for troubleshooting and monitoring.
• Text formatting: align columns, remove duplicates or convert data from one format to another.

If you spend a lot of time in the terminal, AWK pairs nicely with a good shell setup. My guide on setting up fzf with Zsh for faster terminal search is a good companion.

Print specific columns

Printing selected columns is the most common AWK job. To print the first and third columns of a space-separated file, run:

awk '{ print $1, $3 }' file.txt

AWK splits each line on whitespace, so $1 is the first column and $3 is the third. The comma between them adds a space in the output.

Filter rows based on a condition

A condition in front of the action tells AWK which lines to keep. To print only the lines where the second column is greater than 10, run:

awk '$2 > 10 { print }' file.txt

Lines where the condition is false are simply skipped. Check that the column really holds numbers, because text values won’t compare the way you expect.

Sum a column

AWK can do math as it reads the file. To calculate the sum of the third column, run:

awk '{ sum += $3 } END { print sum }' file.txt

The running total builds up line by line, and the END block prints it once AWK has read the last line.

Print lines that match a pattern

Regular expressions let AWK work like a smarter grep. To print every line that contains the word “error”, run:

awk '/error/ { print }' file.txt

The match is case-sensitive, so “Error” and “ERROR” won’t show up unless your pattern covers them.

Run a conditional action

You can put if/else logic inside the action itself. To print “OK” when the second column is greater than 10 and “Fail” otherwise, run:

awk '{ if ($2 > 10) print "OK"; else print "Fail" }' file.txt

Every line gets a result here, which is handy for quick pass/fail reports.

Count the lines in a file

The NR variable tracks the current line number, so at the end it equals the total. To count the lines in a file, run:

awk 'END { print NR }' file.txt

You get the same result as wc -l, but you can combine it with a pattern to count only matching lines.

Change the field separator for CSV files

Not every file uses spaces between columns. To process a comma-separated (CSV) file, set the field separator like this:

awk -F ',' '{ print $1, $3 }' file.csv

Now $1, $2 and friends refer to the values between commas. A quick warning: plain AWK doesn’t understand quoted fields that contain commas, so very messy CSV files may need a dedicated tool.

Write an AWK script for log analysis

Once a one-liner gets too long to read, move it into its own file. This script processes a log file, counts HTTP 500 errors and writes the results to a CSV file:

#!/usr/bin/awk -f

BEGIN {
    FS = " "   # Set the field separator to a space
    OFS = ","  # Set the output field separator to a comma
    count = 0  # Initialize the error count
}

/error 500/ {
    count++            # Increment the error count
    csv[count] = $0    # Store the error line in the CSV array
}

END {
    print "Error Count,Log Line"  # Print the header line in the CSV file

    # Loop through the CSV array and print error count and log line
    for (i = 1; i <= count; i++) {
        print i, csv[i]
    }
}

To run the script, use this command:

awk -f script.awk errors.log

Open the generated CSV file afterward to check the counts look right. If you leave long log jobs running, a terminal multiplexer helps a lot; here’s how I use tmux for terminal management.

Convert text to uppercase

AWK has built-in string functions for quick text cleanup. To convert all lowercase letters in the first column to uppercase, run:

awk '{ $1 = toupper($1) } { print }' file.txt

The toupper() function does the work, and tolower() does the opposite. For heavier string work inside a real program, check out my post on Python string manipulation.

AWK is small, fast and already on your machine, so the next time you’re about to open a spreadsheet for a quick column total, try a one-liner first.

Hope this saves you some time. 🐧