Text processing and stream editing are core functionalities in Linux that empower users to manipulate text data efficiently. These operations are critical for tasks such as filtering logs, transforming data formats, automating edits, and extracting information.
Linux offers powerful tools like sed (Stream Editor) and awk (pattern scanning and processing language) that work on streams or files, enabling sophisticated text transformations and reporting, all from the command line.
What is Stream Editing?
Stream editing involves processing text sequentially from input (like files or pipelines), applying specified transformations or filters, and outputting the result without requiring manual intervention.
sed is a non-interactive stream editor, perfect for tasks like substitution, deletion, insertion, or selective print commands across large data sets.
Sed - The Stream Editor
Text manipulation in Linux can be efficiently handled using the sed stream editor. Following are essential sed commands used for editing and filtering text.
Basic Usage:
sed [options] 'command' fileCommon Commands:
1. Substitution:
sed 's/old-text/new-text/g' filenameReplaces all occurrences of old-text with new-text globally.
2. Deletion:
sed '3d' filenameDeletes the third line.
3. Printing Specific Lines:
sed -n '1,5p' filenamePrints lines 1 through 5.
4. In-place Editing:
sed -i 's/foo/bar/g' filenameReplaces text in the file directly (creates a backup if specified).
Appending and Inserting Lines:
1. Append after a match:
sed '/pattern/a New line text' filename2. Insert before a match:
sed '/pattern/i New line text' filenameSed reads line by line, applies the commands, then writes to standard output or files.
Awk - The Pattern Scanning and Processing Language
Awk processes input data record by record and field by field. Below is a list of practical awk commands for text analysis and reporting.
Usage:
awk 'pattern {action}' filenameProcesses files record by record (usually line by line) and field by field. Fields are represented by $1, $2, ..., and NF is the number of fields.
Common Actions:
1. Print specific fields:
awk '{print $1, $3}' filenamePrints first and third columns.
2. Pattern matching with conditional actions:
awk '/error/ {print $0}' filenamePrints all lines containing “error.”
3. Summation and aggregation:
awk '{sum += $3} END {print sum}' filenameSums values in the third column.
Awk also supports variables, arithmetic, built-in functions, control flow, making it a powerful data extraction and reporting tool.
Comparing Sed and Awk
Common Text Processing Workflow
1. Use grep to filter input lines.
2. Pipe output to sed for substitution or text cleanup.
3. Use awk to extract fields, perform calculations, or generate reports.
4. Output results or redirect to files.
Example:
grep 'ERROR' logfile | sed 's/ERROR:/Error:/' | awk '{print $1, $5}'