Read All Lines from File in Bash

When reading a file line by line in Bash, a common pitfall is mishandling the last line if the file does not end with a newline character.

Example: Faulty Way (misses last line if no trailing newline)

This common approach will fail to process the very last line if that line is not terminated by a newline character. This is because read returns a non-zero exit status when it encounters EOF before a newline, causing the while loop to terminate prematurely.

# FAULTY: Misses the last line if it's not newline-terminated.
while IFS= read -r line
do
    echo "Processing: ${line}"
done < /path/to/your/file.txt

Correct Ways to Read All Lines

Here are two primary, correct methods:

1. Robust `while read` Loop (Memory-Efficient, POSIX-friendly)

This is the most common and portable way to correctly read all lines, including the last line even if it's not newline-terminated.

Method: The key is to check the exit status of read or if the variable line was populated. If read reaches EOF without a newline but did read characters into line, the || [[ -n "$line" ]] condition ensures the loop body executes one last time for that line.

#!/bin/bash

input_file="/path/to/your/file.txt"


while IFS= read -r line || [[ -n "$line" ]]; do
    # IFS=    : Prevents trimming of leading/trailing whitespace. Remove if trimming is desired.
    # -r      : Prevents backslash interpretation (raw read).
    # || [[ -n "$line" ]]: Crucial for the last line if it has no trailing newline.
    #                     It ensures the loop continues if 'read' fails (EOF)
    #                     but 'line' still contains data.

    echo "Line=[${line}]"  # Process each line as needed
    # Example: your_command "${line}"
done < "$input_file"

Explanation:

IFS=: Preserves leading/trailing whitespace on the line. If you want to strip it, you can omit IFS=.
read -r line: Reads a line into the line variable. -r ensures backslashes are treated literally.
|| [[ -n "$line" ]]: This is the critical part.
- read returns 0 (success) if it reads a full line terminated by a newline.
- read returns non-zero (failure) if it hits EOF.
- If the last line has no newline, read reads the content, hits EOF, and returns non-zero.
- The || [[ -n "$line" ]] checks if line is non-empty. If read failed due to EOF but line still got populated (the last non-terminated line), this condition is true, and the loop body executes.
< "$input_file": Redirects the file content to the standard input of the while loop.

Pros:

Memory efficient (processes line by line).
Works correctly for files with or without a trailing newline on the last line.
Highly portable (though [[ ... ]] is a Bash extension, [ -n "$line" ] can be used for stricter POSIX).

Cons:

The || [[ -n "$line" ]] logic might seem slightly less intuitive at first glance compared to mapfile.

2. Using `mapfile` (or `readarray`) (Bash 4.0+)

If you're using Bash 4.0 or newer and the file isn't excessively large (as it reads the whole file into memory), mapfile is a very convenient and robust way.

Method: mapfile reads lines from standard input into an array. The -t option removes the trailing newline from each line. It inherently handles the last line correctly.

#!/bin/bash

input_file="/path/to/your/file.txt"

# -t: Removes the trailing newline character from each line read.
mapfile -t lines_array < "$input_file"
for line in "${lines_array[@]}"; do
    echo "Line=[${line}]" # Process each line as needed
done

Explanation:

mapfile -t lines_array < "$input_file": Reads all lines from $input_file into the lines_array. The -t option is crucial for stripping the newline characters from each line.
for line in "${lines_array[@]}"; do ... done: Iterates over each element (line) in the array.

Pros:

Simple and clean syntax.
Correctly handles files with or without a trailing newline.
Lines are easily accessible in an array.
-t conveniently removes newlines.

Cons:

Reads the entire file into memory, which can be an issue for very large files.
Requires Bash version 4.0 or newer.

Choosing the right method

For memory efficiency with large files or maximum portability (including older Bash versions or POSIX shells by adjusting [[ to [), use the robust while read loop.
For simplicity and convenience with files that fit comfortably in memory and when using Bash 4.0+, mapfile is an excellent choice.

Both methods will correctly process empty lines in the file (they will be read as empty strings). If you need to skip empty lines, you can add a condition like if [[ -z "$line" ]]; then continue; fi inside the loop.l

Correct Ways to Read All Lines

1. Robust while read Loop (Memory-Efficient, POSIX-friendly)

2. Using mapfile (or readarray) (Bash 4.0+)

Choosing the right method