Workflow scripts

• A script is a file with a .sh extension. It contains a list of shell commands executed by an interpreter

• Shebang (#!) defines the interpreter on the first line

  • #!/bin/bash - commands interpreted by bash
  • #!/usr/bin/python - interpreted by Python

• A script file should have x permissions: chmod u+x hello_world.sh

• Running a script: ./hello_world.sh

Variables

• Set a variable: count_of_files=3

• Wrong set a variable: count_of_files = 3 (spaces)

• Quotes are optional. The following commands are equivalent:

  file="/home/mdozmorov/work/README.md"
  file=/home/mdozmorov/work/README.md

• Use a variable: echo $file, or echo ${file}

Capturing output of a command into a variable using backticks

• Wrap a command into backticks - the backwards apostrophes that appear on a US English keyboard at the upper left, under the ~ (tilde)

• Equivalent saying "get the output of the backticked command as a string variable"

echo `date` 
CURRENT_DIR=`pwd`
file_name=`basename /bin/mkdir`

Arguments of a script as variables

• Example of an argument: ./hello_world.sh "Hello World!"

• Within a script, special variables are reserved:

echo $0         - prints the script name
echo $1         - prints the first argument
echo $2         - prints the second argument
echo ${10} - prints the tenth argument
echo $#    - prints the number of arguments

Internal variables

Set system’s parameters. Can be defined in system’s configuration files .bashrc, .bash_profile

DISPLAY    - tells X11 on which display to open windows
EDITOR    - default text editor; usually emacs or vim
HOME    - path to user's home directory; same as ~
PATH    - path to executable programs
PWD        - current directory, same as pwd
SHELL    - path to the current shell
TERM    - current terminal type
USER    - account name of current user, same as whoami

Use, e.g., echo $DISPLAY to see their content

Aliases

To avoid retyping commands - use an alias. Can be defined in system’s configuration files .profile (Linux), .bash_profile, .bashrc (Mac)

alias lah='ls -lah'
alias ..='cd ..’
# get top process eating memory
alias psmem='ps auxf | sort -nr -k 4'
alias psmem10='ps auxf | sort -nr -k 4 | head -10'
# get top process eating cpu
alias pscpu='ps auxf | sort -nr -k 3'
alias pscpu10='ps auxf | sort -nr -k 3 | head -10'
# Find files eating space in the current directory
alias spacehogs='du -cks * | sort -rn'

Conditional execution (if .. then .. else)

if [ ! -e $results_dir ]; then 
    mkdir $results_dir; 
fi

Some popular operators for checking a condition include:

-e <file>                  - TRUE if a specific file/directory exists
–s <file>                  - TRUE if non-empty file
-z <string>                 - TRUE if the given string is empty
<string1> = <string2>  - TRUE if the two strings are equal

• help test - see all operators

Loops (for .. do .. done)

for file in `ls *.txt`; do 
    echo $file; 
    gzip $file;
done

• while-do-done construct also available

The PATH environment variable

• Unix executable commands are located in special folders

$ which ls
/usr/bin/ls
$ which cat
/usr/bin/cat
$ which head
/usr/bin/head

• Executables may be kept in many different places on the Unix system.

The PATH environment variable

• The PATH environmental variable is a colon-delimited list of directories where your shell will look to find exexutable commands

$ echo $PATH
/Users/mdozmorov/miniconda2/bin:/Users/mdozmorov/.rvm/gems/ruby-2.3.1/bin:/Users/mdozmorov/.rvm/gems/ruby-2.3.1@global/bin:/Users/mdozmorov/.rvm/rubies/ruby-2.3.1/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/TeX/texbin:/Users/mdozmorov/.rvm/bin

Exercise:

• Check the $PATH (or, pick any) variable (hint: use echo)

Expanding the PATH

• Often you need to install software as a user, i.e., not as root or sudo user

• Create user-specific bin, lib folders, like:

$ mkdir ~/.local/bin
$ mkdir ~/.local/lib

• .local is a hidden folder in your home directory (use ls -lah to see it)

• Add these folders to the search path: export PATH=$PATH:$HOME/.local/bin:$HOME/.local/lib - now, Unix will look there for executables

• Put the export ... command in bash_profile to automatically execute it every time you use shell

Installing software as a user

• Read README - each software is different
• When installing using make, typically:

  $ ./configure --prefix=$HOME/.local
  $ make
  $ make install

• When using Python setup.py, typically:

  $ python setup.py install --user

• When installing Python packages using pip

  $ pip install --user FOOBAR

https://unix.stackexchange.com/questions/42567/how-to-install-program-locally-without-sudo-privileges

Conda environment

Package, dependency and environment management for any language — Python, R, Java, and more. Install "miniconda" for your OS

• Environment - a place isolated from the operating system where one can install software without risking system's conflicts. If something will be wrong in the environment, it is easy to delete and start again. Think about environments like subfolders (they are).

By default, you are in "base" environment

https://docs.conda.io/en/latest/miniconda.html

Conda environment

Making new environment:

conda create -n new-env

Choose name new-env reflecting what you plan to install, e.g., conda create -n UCSC

Conda environment

Install any programming language within the environment (add -y arguiment to answer Yes to all prompts):

conda create -n new-env python=3.9
conda create -n new-env r

Activate/Deactivate environment:

source activate new-env
conda deactivate

Note the change in command prompt, when in an environment, e.g. (jupyterlab) mdozmorov@work:$

Conda environment

Install software per instructions. Also, use conda itself to install software:

conda install -c bioconda ucsc-bigwigtowig

Google "conda install 'software name'"

Create as many environments as you want. List them as:

conda info --envs
conda env list

Delete an environment

conda remove --name new-env --all