Linux — Permissions, Pipes & the Environment¶

This article picks up where Linux Basics left off. You should already be comfortable navigating the filesystem and managing files. Here you'll learn how Linux controls access to files, how to chain commands together with pipes, and how to configure your shell environment.

The Terminal Tour¶

You've already set up your workspace from the basic tour. Now it's time to go deeper: control who can access your files, chain commands together, and manage software modules.

Enter your NetID below, then click SSH into Discovery → to begin.

File Permissions¶

Every file on Linux has a set of permissions that control who can read, write, or execute it. When you run ls -l, the leftmost column shows them:

-rw-r--r--  1 f00abc f00abc  312 Mar 10 14:22 README.txt
drwxr-xr-x  2 f00abc f00abc 4096 Mar 10 14:22 my_project/

The permission string has 10 characters:

- rw- r-- r--
│ │   │   └── others: read only
│ │   └────── group: read only
│ └────────── owner: read + write
└──────────── file type (- = file, d = directory)

Each group of three characters is rwx: read, write, execute.

Why this matters on HPC¶

On a shared cluster, files in your home directory are typically readable only by you. Files in a shared lab volume may be readable by your whole group. Scripts you want to run must be executable:

chmod +x run_analysis.sh   # make a script executable
chmod 750 my_project/      # owner: rwx, group: r-x, others: ---

Pipes and Redirection¶

One of Linux's most powerful ideas: the output of one command can become the input of another.

The pipe |¶

cat results.csv | grep "PASS" | wc -l   # count passing variants
ls -l | sort -k5 -rn | head             # 10 largest files

Redirect output to a file >¶

grep "PASS" results.csv > passing_variants.csv   # overwrite
grep "FAIL" results.csv >> all_variants.csv      # append

Redirect input from a file <¶

sort < unsorted.txt

Discard output¶

some_noisy_command > /dev/null 2>&1   # throw away stdout and stderr

Environment and Dotfiles¶

Environment variables¶

Your shell has environment variables — named values that affect how programs run. The most important one is $PATH, which tells the shell where to look for commands:

echo $PATH
echo $HOME    # your home directory
echo $USER    # your username

Dotfiles¶

Files starting with . are hidden from plain ls. Your home directory contains several of these dotfiles — configuration files that are loaded every time you open a shell:

To apply changes after editing ~/.bashrc without logging out:

source ~/.bashrc

Environment modules on Discovery¶

Now that you know about $PATH, here's the practical payoff: Discovery uses an environment modules system so you never have to edit $PATH by hand. When you load a module, the system updates your environment variables automatically:

module load python/3.11    # makes Python 3.11 available
module list                # see what's currently loaded

That's all you need for now. Modules are central to working on the cluster: Loading software, managing versions, and keeping your jobs reproducible are all important concerns in HPC land, so they have their own dedicated article.

Quick Reference¶

What's Next¶

• Storage on Discovery: Understand your home directory, lab volumes, and scratch space
• Submitting your first job: Put the scheduler to work with sbatch
• All about modules: Setting the stage by loading the right software for the job