Linux Commands & Shell Scripting: A Complete Practical Guide for DevOps Engineers

Whether you're just starting your DevOps journey or brushing up on your Linux fundamentals — this guide covers everything you need to navigate and operate in a Linux environment confidently.

Table of Contents

Understanding Command Structure System Information Linux Directory Structure Listing Files & Directories File Operations — CRUD Copying, Moving & Deleting Downloading Files Searching with grep Piping — The Power of Chaining Commands head & tail cut — Field Extraction awk — Pattern Scanning & Processing Text Editors — vim File Permissions & Ownership Process Management Environment Variables Archiving & Compression Networking Essentials Quick Command Reference

1. Understanding Command Structure {#command-structure} Every Linux command follows a predictable pattern: bashcommand

command — the program to run (e.g., ls, grep, awk) options — flags that modify behavior, usually prefixed with - or -- inputs — files, directories, URLs, or text the command acts on

Examples: bashls -la /home # command + options + input grep -i "linux" file # command + option + pattern + file curl -s https://example.com | grep "text"

Tip: Use man or --help to read the manual for any command.

1. System Information {#system-information} bashuname # print system name uname -a # ALL system info: kernel, hostname, OS, architecture uname -r # kernel version only uname -m # machine hardware (e.g., x86_64) hostname # display system hostname whoami # current logged-in user id # user ID, group ID, and group memberships uptime # how long the system has been running Sample output of uname -a: Linux devserver 5.15.0-91-generic #101-Ubuntu SMP x86_64 GNU/Linux

2. Linux Directory Structure {#directory-structure} Linux follows a hierarchical Filesystem Hierarchy Standard (FHS). Understanding it is critical for every DevOps engineer. DirectoryPurpose/Root — the top-level directory (equivalent to "My Computer" on Windows)/homeHome directories for all Linux users (e.g., /home/john)/etcSystem-wide configuration files (nginx, ssh, hosts, cron, etc.)/tmpTemporary files — cleared on reboot/binEssential user binaries/commands (ls, cp, mv, cat, etc.)/sbinSystem binaries for root/admin tasks/usrUser programs, libraries, documentation/varVariable data — logs, spool files, databases/optOptional/third-party software/devDevice files (disks, terminals, USB)/procVirtual filesystem for process and kernel info/mntTemporary mount points for filesystems Path Types: bash/c/devops/daws-90s/repos # Absolute path — starts from root / repos/ # Relative path — relative to current location

Tip: Use pwd (print working directory) to see your current absolute path. Use cd - to go back to the previous directory.

1. Listing Files & Directories {#listing-files} bashls # basic list of files and directories ls -l # long format: permissions, owner, size, date, name ls -la # long format + ALL files (including hidden files starting with .) ls -lr # long format, reverse alphabetical order ls -lt # sort by modification time (newest first) ls -ltr # sort by modification time (oldest first / latest at bottom) ls -lh # human-readable file sizes (KB, MB, GB) ls -R # recursively list all subdirectories Understanding ls -l output: drwxr-xr-x 2 john devops 4096 May 28 10:00 projects -rw-r--r-- 1 john devops 512 May 28 09:45 devops.txt FieldMeaningd / -d = directory, - = file, l = symlinkrwxr-xr-xPermissions for owner, group, others2Number of hard linksjohnFile ownerdevopsGroup owner4096Size in bytesMay 28 10:00Last modification timeprojectsName Hidden files start with . (dot): bash.bashrc .gitignore .ssh/

2. File Operations — CRUD {#file-operations} CRUD = Create, Read, Update, Delete Create bashtouch devops.txt # creates an empty file (or updates timestamp if exists) touch file1 file2 file3 # create multiple files at once mkdir devops # create a directory mkdir -p a/b/c # create nested directories (no error if exists) Read bashcat devops.txt # print entire file to screen tac devops.txt # print file in reverse (last line first) less devops.txt # paginated view (press q to quit) more devops.txt # older paginator nl devops.txt # print with line numbers wc devops.txt # word count: lines, words, characters wc -l devops.txt # count lines only Update / Write bashcat > devops.txt # overwrite: type content, then Ctrl+D to save cat >> devops.txt # append: type content, then Ctrl+D to save

Redirect command output to a file

echo "Hello DevOps" > devops.txt # write (overwrite) echo "Another line" >> devops.txt # append

Merge files

cat file-1 file-2 > file-3 # merge file-1 and file-2 into file-3 Redirection operators: OperatorMeaning>Redirect stdout (overwrite)>>Redirect stdout (append)Redirect stderr2>&1Redirect stderr to stdout Delete bashrm devops.txt # delete a file rm -i devops.txt # interactive — ask before deleting rm -f devops.txt # force delete (no prompt) rm -r devops/ # recursively delete a directory and its contents rm -rf devops/ # force recursive delete (use with caution!) rmdir devops/ # delete only if directory is EMPTY

Warning: rm -rf is irreversible. Always double-check the path before executing.

1. Copying, Moving & Deleting {#copy-move-delete} bash# Copy cp source.txt destination.txt # copy file cp -r source_dir/ destination_dir/ # copy directory recursively cp -p file.txt backup.txt # preserve permissions and timestamps

Move / Rename

mv oldname.txt newname.txt # rename a file mv file.txt /home/john/documents/ # move file to another directory mv dir1/ /opt/newlocation/ # move a directory

Symbolic Links

ln -s /original/path /link/path # create a symlink (shortcut)

1. Downloading Files {#downloading-files} bash# wget — download files from the web wget # download file to current directory wget -O output.txt # save with a custom filename wget -c # resume incomplete download

curl — display or transfer data

curl # print response directly to screen curl -s # silent mode (no progress bar) curl -O # save file with original name curl -o output.txt # save with custom name curl -L # follow redirects When to use which? ToolBest For wget: Downloading files, recursive downloads, resuming downloads, curl API calls, scripting, HTTP headers, POST requests, piping output bash# curl for API calls curl -X POST https://api.example.com/data
-H "Content-Type: application/json."
-d '{"key": "value"}'

1. Searching with grep {#grep}. grep finds matching text inside files. bashgrep "linux" file.txt # case-sensitive search grep -i "linux" file.txt # -i: case-insensitive (matches Linux, LINUX, linux) grep -n "linux" file.txt # -n: show line numbers grep -c "linux" file.txt # -c: count matching lines grep -v "linux" file.txt # -v: invert — lines that do NOT match grep -r "linux" /etc/ # -r: recursive search in directories grep -l "linux" /etc/ # -l: list only filenames with matches grep -w "linux" file.txt # -w: whole word match only grep -A 3 "linux" file.txt # show 3 lines AFTER match grep -B 3 "linux" file.txt # show 3 lines BEFORE match grep -E "linux|unix" file.txt # -E: extended regex (OR match)

Remember: Linux is case-sensitive. Linux and Linux are different — use -i for case-insensitive matching.

1. Piping — The Power of Chaining Commands {#piping} The pipe | passes the output of one command as the input to the next. This is one of the most powerful concepts in Linux. bashcommand1 | command2 | command3 Real-world examples: bash# Fetch a remote file and search for a keyword curl -s https://raw.githubusercontent.com/daws-90s/notes/refs/heads/main/session-02.txt | grep linux

List all running processes and search for nginx

ps aux | grep nginx

Count how many lines in a file contain "error"

cat /var/log/syslog | grep -i "error" | wc -l

Get the top 5 largest files in a directory

ls -lhS | head -6

List all unique users from /etc/passwd

cut -d ":" -f1 /etc/passwd | sort | uniq

1. head & tail {#head-tail} bashhead file.txt # first 10 lines (default) head -n 20 file.txt # first 20 lines head -5 file.txt # shorthand for first 5 lines

tail file.txt # last 10 lines (default) tail -n 20 file.txt # last 20 lines tail -f /var/log/syslog # LIVE follow — watches file as it grows (perfect for logs!) tail -F /var/log/app.log # follow even if file is rotated/recreated

DevOps Tip: tail -f is invaluable for real-time log monitoring during deployments. Combine with grep for filtered live logs: bashtail -f /var/log/nginx/access.log | grep "500"

1. cut — Field Extraction {#cut} cut extracts specific columns/fields from text. bash# Syntax cut -d "" -f <field_number>

Extract usernames from /etc/passwd (delimiter is :, field 1)

cut -d ":" -f1 /etc/passwd

Extract user IDs (field 3)

cut -d ":" -f3 /etc/passwd

Extract multiple fields (1 and 3)

cut -d ":" -f1,3 /etc/passwd

Extract a range of fields (1 to 4)

cut -d ":" -f1-4 /etc/passwd

Extract the last segment of a URL using the delimiter /

echo "https://raw.githubusercontent.com/daws-90s/notes/refs/heads/main/session-02.txt"
| cut -d "/" -f9

Cut by character position (extract characters 1-5)

cut -c1-5 file.txt /etc/passwd structure: username:password:UID:GID:comment:home_directory:shell john:x:1001:1001:John Doe:/home/john:/bin/bash

1. awk — Pattern Scanning & Processing {#awk} awk is a powerful text-processing language. It processes files line by line and splits each line into fields. bash# Basic syntax awk -F "" '{ action }' file

Print last segment of a URL (NF = number of fields, $NF = last field)

echo "https://raw.githubusercontent.com/daws-90s/notes/refs/heads/main/session-02.txt"
| awk -F "/" '{print $NF}'

Print the first field from /etc/passwd

awk -F ":" '{print $1}' /etc/passwd

Print users with UID > 999 (system users have UID <= 999)

awk -F ":" '\(3 > 999 { print \)1 }' /etc/passwd

Print specific fields with a custom separator

awk -F ":" '{print $1, $3}' /etc/passwd

Print line number + content

awk '{print NR, $0}' file.txt

Count lines (like wc -l)

awk 'END {print NR}' file.txt

Sum a column of numbers

awk '{sum += $1} END {print sum}' numbers.txt

Print lines matching a pattern

awk '/error/ {print}' /var/log/syslog Key awk variables: VariableMeaning$0Entire current line$1, $2...Individual fields$NFLast fieldNRCurrent line numberNFTotal number of fields in current lineFSField separator (same as -F)

1. Text Editors — vim {#vim} Vim is the most widely used terminal text editor in Linux/DevOps environments. bashvim filename.txt # open or create a file vim has modes: ModeHow to EnterPurposeNormalEscNavigate, copy, paste, deleteInserti, a, oType/edit textVisualvSelect textCommand: Save, quit, search, replace Essential vim commands:

Navigation

h j k l → left, down, up, right gg → go to first line G → go to last line:20 → jump to line 20

Editing

i → insert before cursor a → insert after cursor o → new line below dd → delete current line yy → copy (yank) current line p → paste below u → undo Ctrl+r → redo

Saving & Quitting

:w → save :q → quit :wq → save and quit :q! → quit without saving:wq! → force save and quit

Search & Replace

/pattern → search forward ?pattern → search backward n → next match N → previous match :%s/old/new/g → replace all occurrences

Quick tip: Stuck in Vim? Press Esc, then type:q! and hit Enter to exit without saving.

1. File Permissions & Ownership {#permissions} Every file has three permission sets: owner, group, and others. -rwxr-xr-- 1 john devops 1024 May 28 file.sh ^^^ ^^^ ^^^ own grp oth PermissionSymbolNumericReadr4Writew2Executex1No permission-0 bash# Change permissions chmod 755 script.sh # rwxr-xr-x (owner: all, group/others: read+execute) chmod 644 config.txt # rw-r--r-- (owner: read+write, others: read only) chmod +x script.sh # add execute permission for everyone chmod -x script.sh # remove execute permission

Change ownership

chown john file.txt # change owner chown john:devops file.txt # change owner and group chown -R john:devops /opt/app/ # recursive ownership change

1. Process Management {#process-management} bashps aux # list all running processes ps -ef # full format process list top # live process viewer (press q to quit) htop # enhanced interactive process viewer

Find process by name

ps aux | grep nginx pgrep nginx

Kill a process

kill # send SIGTERM (graceful shutdown) kill -9 # send SIGKILL (force kill) killall nginx # kill all processes named nginx

Background & foreground

command & # run in background jobs # list background jobs fg %1 # bring job 1 to foreground bg %1 # resume job 1 in background nohup command & # run even after terminal closes

1. Environment Variables {#env-variables} bashenv # list all environment variables echo $HOME # print HOME variable echo $PATH # print PATH echo $USER # current user

Set a variable (current session only)

export MY_VAR="hello"

Make permanent (add to ~/.bashrc or ~/.bash_profile)

echo 'export MY_VAR="hello"' >> ~/.bashrc source ~/.bashrc # reload configuration

Common environment variables

\(HOME → /home/username \)USER → current user $PATH → directories searched for commands $SHELL → current shell (/bin/bash) $PWD → current directory $EDITOR → default text editor

1. Archiving & Compression {#archiving} bash# tar — tape archive tar -cvf archive.tar files/ # create archive tar -xvf archive.tar # extract archive tar -czvf archive.tar.gz files/ # create gzip compressed archive tar -xzvf archive.tar.gz # extract gzip archive tar -cjvf archive.tar.bz2 files/ # create bzip2 compressed archive tar -tzvf archive.tar.gz # list contents without extracting

zip / unzip

zip -r archive.zip files/ unzip archive.zip

gzip/gunzip

gzip file.txt # compress → file.txt.gz gunzip file.txt.gz # decompress

1. Networking Essentials {#networking} bashping google.com # test connectivity traceroute google.com # trace route to host netstat -tuln # list open ports ss -tuln # modern replacement for netstat ifconfig # network interface info (older) ip addr # modern way to show IP addresses ip route # display routing table nslookup google.com # DNS lookup dig google.com # detailed DNS lookup ssh user@hostname # connect to remote server scp file.txt user@host:/path/ # secure copy to remote

2. Quick Command Reference {#quick-reference} Command Purposes List directory contents cat Read/display file contents tac Display file in reverse order touch Create empty file or update timestamp rm Remove files/directories mkdir Create directories cp Copy files/directories mv Move or renamecurlTransfer data from URLs / API calls wget Download files from webheadShow first N linestailShow last N lines / live log followgrepSearch text patterns in filescutExtract fields/columnsawkText processing and pattern scanningvimTerminal text editorchmodChange file permissionschownChange file ownershippsList processeskillTerminate processestarArchive and compress filessshSecure remote loginunameSystem/kernel informationmanRead the manual for any command.

Tags: #Linux #DevOps #ShellScripting #SysAdmin #CloudComputing #Bash #CommandLine #LearnLinux #90DaysOfDevOps