Extra Unix Shell Material: Glossary

Key Points

Introduction

  • Familarity with the basic shell commands covered in the introductory lesson is assumed.

  • Nelle’s pipeline is available as a shell script which runs the goostats program on an entire dataset.
Manual Pages

  • man command displays the manual page for a given command.

  • [OPTION]... means the given command can be followed by one or more optional flags.

  • Flags specified after ellipsis are still optional but must come after all other flags.

  • While inside the manual page,use / followed by your pattern to do interactive searching.
Working Remotely

  • SSH is a secure alternative to username/password authorization

  • SSH keys are generated in public/private pairs. Your public key can be shared with others. The private keys stays on your machine only.

  • The ‘ssh’ and ‘scp’ utilities are secure alternatives to logging into, and copying files to/from remote machine
Working with archive files

  • Archive files are files which contain one or more other files. They are a convienient way to store or transfer multiple files and directory structures inside a single file.

  • Zip archives are (usually) compressed by default.

  • Zip files can be extracted with the unzip command.

  • Zip files can be created with the zip command.

  • Tar archives are not compressed.

  • Tar archives can be compressed using the gzip or bzip2 utilities.

  • Tar files can be extracted with the tar -xf option.

  • Modern versions of tar will automatically uncompress gzipped or bzipped archives.

  • Tar files can be created with the tar -cf option.
Transferring Files

  • wget is the default tool, available in most Linux distributions, to download files from web and FTP servers.

  • curl is another utility for downloading remote webpages. It defaults to outputting the result on screen, this can be piped to other programs.

  • rsync is a utility for transferring files. It can use the SSH protocol and is useful for mirroring complicated directory structures from one computer to another.
Disk Usage

  • The du command tells us how much disk space a directory is using.

  • The -h option to du gives us human readable units such a K, M and G.

  • The df command tells us how much space is in use on a disk.

  • The df command can also take a -h option for human readable units.

  • On some shared systems the quota command tells us how much space is left in our disk allocation.
Permissions

  • Correct permissions are critical for the security of a system.

  • File permissions describe who and what can read, write, modify, and access a file.

  • Use ls -l to view the permissions for a specific file.

  • Use chmod to change permissions on a file or directory.

  • Use chmod 777 carefully, as it grants all permissions to everyone.

  • Access Control Lists (ACLs) provide finer-grained permission control.
Processes and Job Control

  • When we talk of ‘job control’, we really mean ‘process control’

  • A running process can be stopped, paused, and/or made to run in the background

  • A process can be started so as to immediately run in the background

  • Paused or backgrounded processes can be brought back into the foreground

  • Process information can be inspected with ps

  • The top (or htop) command shows a live view of the resouces used by each process.

  • The tmux command allows us to leave a command running and disconnect from the system by press Ctrl+B d.

  • The tmux session can be reattached with tmux -a.
Aliases and Bash Customization

  • Aliases are used to create shortcuts or abbreviations

  • The .bashrc and .bash_profile files allow us to customize our bash environment.

  • The PS1 system variable can be changed to customize your bash prompt.
Shell Variables

  • Shell variables are by default treated as strings

  • The PATH variable defines the shell’s search path

  • Variables are assigned using “=” and recalled using the variable’s name prefixed by “$”
Command Substitution

  • We can substitue variables for the output of commands using the $(command) syntax.

  • We can loop through sets of values in a “parameter sweep”.

  • For loops can take a single variable with space separated arguments and treat each as a separate item to iterate over.
Streams

  • STDERR can be redirected by using 2>.

  • STDOUT can be redirected by using 1> or >.

  • tee can be used to duplicate STDOUT and send the second copy to a file.
AWK

  • awk can be used to manipulate and filter data, e.g. adding text or printing specific columns

  • NF is a variable that stores the number of fields in the current line

  • Field separator can be specified with the -F option, default is space

  • Matching patterns can be specified with /^PATTERN/ instruction

Glossary

remote login
The ability to log into a remote computer, typically over a network, as if you were physically present at the computer.
remote login server
A computer that allows remote login.
SSH daemon
A program that runs in the background on a server and waits for incoming SSH connections.
secure shell
A cryptographic network protocol for operating network services securely over an unsecured network.
SSH key
A cryptographic key pair used for authenticating users in the SSH protocol.
SSH protocol
A cryptographic network protocol for operating network services securely over an unsecured network.
command
A directive to a computer program acting as an interpreter of some kind, in order to perform a specific task.
user name
The name of a user account on a computer system.
user ID
A unique identifier assigned to a user account on a computer system.
user group
A collection of user accounts that share access rights and permissions.
user group name
The name of a user group.
user group ID
A unique identifier assigned to a user group.
access control lists
A list of permissions attached to an object that specifies which users or system processes are granted access to objects, as well as what operations are allowed on given objects.
search path
A list of directories that the operating system searches when looking for a file.