Korean, Edit

Chapter 1. Linux

Recommended Article: 【Operating System】 Operating System Table of Contents

1. Overview

2. Installation

3. Commands

a. Linux Server

b. Linux Trouble-shooting (01-20)

1. Overview

⑴ macOS, Ubuntu are part of the Linux family.

⑵ If you are not familiar with the Linux environment, use the Terminus program: it is provided for free to students.

2. Installation

⑴ Method 1. Ubuntu

① Since Ubuntu, etc., originally operate with Linux as the operating system, you can manipulate Linux by turning on the terminal.

⑵ Method 2. MacOS

① 2-1. You can remotely access a Linux server using the ssh command in the terminal.

② 2-2. Install Linux on an M1 MacBook with Parallels.

○ M1 MacBooks use the 64-bit ARM desktop image instead of the AMD64 desktop image.

○ Ryzen 7 does not support AMD-V enabling: you may need to use the ARM desktop image.

○ How to uninstall Parallels on a MacBook.

⑶ Method 3. Windows

① 3-1. The methods to access a Linux server are as follows:

○ mobaXterm

○ PowerShell

○ FireZilla

② 3-2. Install Linux on Windows with VMware.

○ Create a virtual environment and install Linux.

○ You can transfer files using mobaXterm.

○ You may need to explicitly enable AMD-V: the method to explicitly enable AMD-V on GIGABYTE is as follows:

○ Example 1

○ 1^st. Turn on the PC and press the Del key quickly several times until the BIOS screen appears.

○ 2^nd. Go to the ‘M.I.T.’ tab and move to ‘Advanced Frequency Settings’.

○ 3^rd. Move to ‘Advanced CPU Core Settings’.

○ 4^th. Switch ‘SVM Mode’ from ‘Disabled’ to ‘Enabled’.

○ Example 2

○ 1^st. Move to the BIOS screen.

○ 2^nd. Go to the ‘Tweaker’ tab and move to ‘Advanced CPU Settings’.

○ 3^rd. Switch ‘SVM Mode’ from ‘Disabled’ to ‘Enabled’.

3. Commands

※ Arrange commands from basic to advanced sequentially.

○ ctrl + shift + v: Paste code

○ Linux Emergency Escape

○ ctrl + c: Forcefully terminate code execution

○ ctrl + d

○ ctrl + \

○ ctrl + z: Pause code execution

○ Linux Terminal Clear

○ ctrl + l: Not commonly used due to imperfect clearing

○ clear

○ reset

○ echo -e '\033c'

○ tput reset

○ variable

○ $variable_name: Value of the variable.

○ export $variable: Used to hand over environment variable to another process.

○ export -f $function: Used to hand over a function to another process.

○ string variable: Declare string variable without quotes and include $variable_name

○ \: Execute a single command over multiple lines for better code readability.

○ Note that a space after \ is treated as an escape character

○ ;: Separate multiple commands within a single line

○ |: Connect output of the previous process to the input of the next process

○ ls: Display the list of files in the current directory

○ history: Past command history.

○ history | tail -20: Display only the last 20 entries of the past command history.

○ ssh-keygen: Generates RSA public and private keys. On a MacBook, the files id_rsa and id_rsa.pub are created in the ~/.ssh directory.

○ chmod

○ “change mode” abbreviated for permission management

○ Types of users: user (u), group (g), others (o)

○ Types of permissions: read (r; 4), write (w; 2), execute (e; 1), rw- (4+2=6), r-x (4+1=5), rwx (4+2+1=7)

○ Method 1. symbolic mode

○ Example 1. chmod u+x filename: Grant execute permission to the user

○ Example 2. chmod g-w filename: Remove write permission from the group

○ Example 3. chmod o=r filename: Set permissions for others to read-only

○ Method 2. numeric mode

○ Example 1. chmod 755 filename: Set permission 7 for the user, 5 for the group, and 5 for others

○ Application 1. Set permissions for an entire directory

○ Example 1. chmod -R 755 directory_name: Set 755 permissions throughout the directory_name.

○ chmod

○ Abbreviation for “change ownership”

○ chown [user]:[group] [Directory]

○ Linux-based text editor

○ open

○ nano: Install with the following command: sudo apt-get install nano

○ vim

○ emacs

○ zsh

○ bash

○ Directory command

○ open [DIR_NAME]: Navigate to a specific directory

○ cd: Navigate to a home directory.

○ cd [DIR_NAME]: Navigate to a specific directory.

○ cd ..: Move to the parent directory of the current directory.

○ ls -lh [FILE_NAME]: Checking the size of a specific file.

○ du -sh [DIR_NAME]: Checking the size of a directory.

○ mkdir [DIR_NAME]: Create a specific folder

○ mv [SOURCE] [DESTINATION]: Commands used in file moving, file replacement, and renaming include “mv”, which is short for “move”.

○ cp [OPTION] [SOURCE] [DESTINATION]: Copy files and paste

○ -r: Copy subdirectories and files

○ -p: Preserve ownership, group, permission, and time information when copying

○ rm [OPTION] [NAME]: Delete

○ rm -d [DIR_NAME]: Delete an empty directory

○ rm -r [DIR_NAME]: Delete a non-empty directory

○ File compression command

○ zip -r test.zip ./*: Compress all files (./*) in a specific directory including subfolders into test.zip

○ unzip FILE.zip: Extract FILE.zip in the current folder

○ gzip [FILE_NAME]: Compress to .gz

○ gzip -d [FILE_NAME].gz: Decompress .gz file

○ gunzip [FILE_NAME].gz: Decompress .gz file

○ tar [OPTION] [FILE.tar.gz]

○ [OPTION]: -f, -c, -x, -v, -z, -j, -t, -C, -A, -d, -r, -u, -k, -U, -w, -e, etc.

tar [OPTION...] [FILE]...
-f : Specify target tar archive. (Default option)
-c : create tar archive. overwrite existing archive. (used to bind files)
-x : Extract files from tar archive. (Use to pull files)
-v: detailed list of processes being processed (file information).
-z : gzip compression application option.
-j : bzip2 compression application option.
-t : Check what is included in the tar archive.
-C : Specifies the destination directory path.
-A : Adds the specified file to the tar archive.
-d : search for differences between tar archive and file system.
-r : Add files to the end of the tar archive.
-u : Add files to the end of the tar archive.
-k : Retain existing files when extracting tar archives.
-U : Delete existing files before extracting tar archive.
-w : Request confirmation of all progress. (interactive)
-e : Stop when the first error occurs.

○ Example 1: Compress: tar -zcvf archive.tar.gz ./MyFolder

○ Example 2: Decompress: tar -xzvf spaceranger-2.0.1.tar.gz

○ The same commands apply to .tar files as well as .tar.gz files, but -z should be omitted for .tar files

○ xz -z filename: Offers quite a good compression ratio but is slow. Uses LZMA, LZMA2 algorithms.

○ xz -d filename.xz: Decompresses .xz files.

○ 7z a outputfile.7z inputfile: Offers quite a good compression ratio but is slow. Uses various algorithms including LZMA. Available on Linux through the p7zip package.

○ 7z x outputfile.7z: Decompresses .7z (7-Zip) files.

○ zstd filename: Balances compression ratio and speed.

○ zstd -d filename.zst: Decompresses .zst files.

○ brotli -Z filename: Compresses using Brotli.

○ brotli -d filename.brotli: Decompresses .brotli files.

○ pigz filename: Compresses using PIGZ.

○ unpigz filename.gz: Decompresses .gz files.

○ echo

○ echo [STR]: Print specified string

○ echo $(($(find . -type d -maxdepth 1 | wc -l) - 1)): Print the number of folders in a specific directory.

○ find

○ find . -type d -maxdepth 1 | wc -l: Print the number of folders in a specific directory.

○ symbolic link (soft link)

○ ln -s [Source_Directory] [Target_Directory]

○ ln -sf [Source_Directory] [Target_Directory]

○ unlink [Target_Directory]: Deactivate the link

○ ifconfig: Check IP address

○ traceroute [IP_ADDRESS]: Display IPs reached to reach a specific IP address

○ View system speculations

○ lshw | less: Provides information about system specs such as CPU, RAM, etc.

○ lshw -C display: Provides information about the GPU

○ nvidia-smi: Provides information about the GPU and GPU usage status

○ watch -d -n 0.5 nvidia-smi: GPU usage monitoring

○ sudo: Allow Linux users to run commands as other users.

○ Sudo permissions can be granted selectively for different users.

○ su [User_ID]: Command for switching to a different user account in Linux. Often used with sudo su.

○ reboot: Rebooting.

○ Status command

○ free -h: Checking memory and swap usage.

○ top: A command to check command execution and thread usage status.

○ htop: Must be installed. Superior to top.

○ df: Command to view system status.

○ df -h: Command to view system capacity.

○ Remote data transfer command

○ Method 1: scp: Low possibility of file corruption

○ scp -P ${port} ${srcFile} ${id} @ ${destIP} :${destPath}

○ ${port}: Port number of the destination

○ ${srcFile}: Path of the file to be moved from the source. Example: “~/Downloads/file.txt”

○ ${id}: User ID on the destination

○ ${destIP}: IP address of the destination. Example: #.#.#.# (IPv4)

○ ${destPath}: Directory address on the destination to store the file. Example: ~/DATA1/1.txt

○ Code Generator

○ Method 2: rsync (remote sync): Faster speed

○ Example: rsync --rsh=’ssh -p ${port}’ -avzhP ${srcFile} ${id} @${destIP} : ${destPath}

○ ${port}: Port number of the destination

○ ${srcFile}: Path of the file to be moved from the source. Example: “~/Downloads/file.txt”

○ ${id}: User ID on the destination

○ ${destIP}: IP address of the destination. Example: #.#.#.# (IPv4)

○ ${destPath}: Directory address on the destination to store the file. Example: ~/DATA1/1.txt

○ Code Generator

○ Method 3. sftp

## Install NCFTP
sudo apt-get install ncftp


## Connect to the FTP Server
ncftp -u [username] [host address]


## Navigate and Manage Files
# List files and directories.
ls 

# Change directory.
cd [directory] 
lcd [directory] 

# Download a file from the FTP server to your local machine.
get [filename] 

# Upload a file from your local machine to the FTP server.
put [filename] 

# Create a new directory on the FTP server.
mkdir [directory_name]

# Remove a directory on the FTP server.
rmdir [directory_name]

# Delete a file on the FTP server.
rm [filename]

# Exit NCFTP
quit


## Use the ncftpput Command
ncftpput -R -u [username] -p [password] [host] [remote-dir] [local-dir]

# Example 
ncftpput -R -u username -p password ftp.example.com /remote/directory /local/directory

○ Method 4. Client software for SFTP: FileZilla, Cyberduck, etc.

○ File execution command

○ .sh file: To run a .sh file, use ./[My_File].sh, sh [My_File].sh, or bash [My_File].sh.

○ .py file (ref)

○ ctrl + z → bg: Run in the background

○ [command] + &: Run in the background (ref)

○ Note: This method leads to processes being terminated when the session ends (e.g., terminal closure)

○ Background run using nohup

○ step 1. Prepare a .sh or .py file

○ step 2. Open the terminal

○ step 3. Navigate to the directory where the .sh file or .py file you want to execute after environment setup is located (e.g., app.py)

○ step 4. Grant permission: chmod 755 app.py

○ step 5. nohup python app.py \

○ step 6. Press Enter once more for execution

○ step 7. (optional) Check execution: ps -ef | grep app.py → Check nohup.out for “app.py” keyword

○ step 8. (optional) Forcefully terminate background execution: kill {ProcessID}

○ Background run using tmux

○ tmux → Linux Command → Terminal Window / Mobaxterm Off

○ docker: Supports background run

○ while command

○ while read

○ while ··· do ~ done

○ cat command: Related to file output.

○ cat FILE: Display file content.

○ cat > FILE: Create a file.

○ cat -n FILE: Display line numbers.

○ cat -E FILE: Display $ at end of lines.

○ cat -T FILE: Display tabs as ^I.

○ cat -s FILE: Ignore repeated blank lines.

○ cat FILE1 FILE2 > FILE_merge: Merge FILE1 and FILE2 into FILE_merge.

○ cat FILE1 - FILE2: Append contents of FILE1 to FILE2.

○ cat FILE | more: Display file content one screen at a time.

○ cat FILE | grep "STR": Filter file content.

○ cat *: Display content of all files.

○ cat *.txt: Display content of files with a specific extension.

○ qsub [-OPTION STR] [SCRIPT]: Submit PBS job to the queue.

○ [SCRIPT]: .sh file

○ Specify arguments for the .sh file after the .sh filename.

○ qsub [-e STR] [SCRIPT]: Define the path for standard error stream of the batch job.

○ qsub [-o STR] [SCRIPT]: Define the path for standard output stream of the batch job.

○ qsub [-N STR] [SCRIPT]: Define the job name.

○ grep [STR]: Used to find a specific string in the content of a file passed as input.

○ $ sudo command to be executed

○ When executed like this, commands run with root privileges, avoiding errors.

○ md5sum

○ Generates code to verify file integrity between sender and receiver.

$ md5sum /home/parallels/Downloads/metadata.xlsx

○ Security-related: Finding MD5 collision pairs is relatively easy, but a method for second-preimage attacks on MD5 is not known.

○ sed

○ netcat

○ awk

○ ssh: Protocol used for remote connection to Linux servers.

○ Troubleshooting 1. If “command not found” appears after entering ssh, follow this procedure for macOS Terminal.

○ Display when ssh is installed

○ Windows 10 onwards, you can use Windows PowerShell for terminal (ref.)

○ Troubleshooting 2. ssh: connect to host #.#.#.# port §: Connection refused

○ Where #.#.#.# is the IP address.

○ Cause 1. Remote login on MacBook is blocked.

○ Solution: System Preferences > Sharing > Remote Login > ON

○ Cause 2. openssh is not installed (ref.)

○ Cause 3. Host IP address has changed.

○ Troubleshooting 3. key_exchange_identification: read: Connection reset by peer Connection reset by #.#.#.# port §

○ Cause: Temporary server issue.

○ Solution: Re-enter the code.

○ Troubleshooting 4. When attempting VPN connection on Windows

○ Xshell from NetSarang allows VPN connection on Windows.

○ Troubleshooting 5. Can’t operate. Failed to connect to bus: Host is down

○ Unsuitable for ssh server.

○ Ensure you intended to install an ssh server, not just a client.

○ curl: Download files from a specific URL.

curl -o my.sh "https://repo.anaconda.com/miniconda/Miniconda3-py38_23.3.1-0-Linux-x86_64.sh"

○ wget command: Enables downloading of various files.

○ Example: wget <https://raw.githubusercontent.com/jdblischak/singleCellSeq/master/code/run-md5sum.sh>

○ If wget is not found: $ sudo apt-get install -y wget

○ Typically, fetches entire command: $git clone <https://github.com/jdblischak/singleCellSeq.git>

○ sudo netstat -lntp: Shows port-specific PID, protocol, etc.

○ Parallel processing through GNU Parallel

## install for Devian/Ubuntu
sudo apt-get update
sudo apt-get install parallel

## run 
process() {
	... 
}
parallel process

Input: 2021.11.04 23:23

Modified: 2023.06.02 14:21