#readwise # Linux Fundamentals - Filter Contents  ## Metadata - Author: [[Hack The Box]] - Full Title: Linux Fundamentals - Filter Contents - URL: https://academy.hackthebox.com/module/18/section/80 ## Summary This section teaches how to filter and manipulate text outputs from commands using various tools. Key tools include `less`, `head`, `tail`, `grep`, `cut`, `tr`, `column`, `awk`, `sed`, and `wc`. These tools help you efficiently process large files and search for specific patterns. Practice using these commands will improve your filtering skills in the terminal. ## Highlights There are two powerful tools for this - `more` and `less`. These are known as pagers, and they allow you to view the contents of a file interactively, one screen at a time. While both tools serve a similar purpose, they have some differences in functionality, which we'll touch on later. ([View Highlight](https://read.readwise.io/read/01jm4dec0328zfspjhachwecdn)) --- When closing `less` with the `[Q]` key, we will notice that the output we have seen, unlike `more`, does not remain in the terminal. ([View Highlight](https://read.readwise.io/read/01jm4dmt01fxgyg87ts9rzm0vj)) --- Sometimes we will only be interested in specific issues either at the beginning of the file or the end. If we only want to get the `first` lines of the file, we can use the tool `head`. By default, `head` prints the first ten lines of the given file or input, if not specified otherwise. ([View Highlight](https://read.readwise.io/read/01jm4dn6m59bds6b3hyjytqv97)) --- If we only want to see the last parts of a file or results, we can use the counterpart of `head` called `tail`, which returns the `last` ten lines. ([View Highlight](https://read.readwise.io/read/01jm4dndra78efpq9fyw2pwv45)) --- Depending on which results and files are dealt with, they are rarely sorted. Often it is necessary to sort the desired results alphabetically or numerically to get a better overview. For this, we can use a tool called `sort`. ([View Highlight](https://read.readwise.io/read/01jm4dp0vqmz8hsssv4xfk1a1k)) --- In many cases, we will need to search for specific results that match patterns we define. One of the most commonly used tools for this purpose is grep, which provides a wide range of powerful features for pattern searching. For instance, we can use grep to search for users who have their default shell set to `/bin/bash`. `cat /etc/passwd | grep "/bin/bash"` ([View Highlight](https://read.readwise.io/read/01jm4dvz5vxcwj3hbtv5nxnsz1)) --- This is just one example of how grep can be applied to efficiently filter data based on predefined patterns. Another possibility is to exclude specific results. For this, the option "`-v`" is used with `grep`. In the next example, we exclude all users who have disabled the standard shell with the name "`/bin/false`" or "`/usr/bin/nologin`". `cat /etc/passwd | grep -v "false\|nologin"` --- Specific results with different characters may be separated as delimiters. Here it is handy to know how to remove specific delimiters and show the words on a line in a specified position. One of the tools that can be used for this is `cut`. Therefore we use the option "`-d`" and set the delimiter to the colon character (`:`) and define with the option "`-f`" the position in the line we want to output. `cat /etc/passwd | grep -v "false\|nologin" | cut -d":" -f1` ([View Highlight](https://read.readwise.io/read/01jm4dy5f84bqyt1w5afcf66f8)) --- Another possibility to replace certain characters from a line with characters defined by us is the tool `tr`. As the first option, we define which character we want to replace, and as a second option, we define the character we want to replace it with. In the next example, we replace the colon character with space. `cat /etc/passwd | grep -v "false\|nologin" | tr ":" " "` ([View Highlight](https://read.readwise.io/read/01jm4dz9rsjqrs27tzg3vfeync)) --- Since search results can often have an unclear representation, the tool `column` is well suited to display such results in tabular form using the "`-t`." `cat /etc/passwd | grep -v "false\|nologin" | tr ":" " " | column -t` ([View Highlight](https://read.readwise.io/read/01jm4e089cpqadsb5tsf7yqnnn)) --- As we may have noticed, the line for the user "`postgres`" has one column too many. To keep it as simple as possible to sort out such results, the (`g`)`awk` programming is beneficial, which allows us to display the first (`$1`) and last (`$NF`) result of the line. `cat /etc/passwd | grep -v "false\|nologin" | tr ":" " " | awk '{print $1, $NF}'` ([View Highlight](https://read.readwise.io/read/01jm4e4ssz0a4cpvd7t9q7vhjg)) --- There will come moments when we want to change specific names in the whole file or standard input. One of the tools we can use for this is the stream editor called `sed`. One of the most common uses of this is substituting text. Here, `sed` looks for patterns we have defined in the form of regular expressions (regex) and replaces them with another pattern that we have also defined. Let us stick to the last results and say we want to replace the word "`bin`" with "`HTB`." The "`s`" flag at the beginning stands for the substitute command. Then we specify the pattern we want to replace. After the slash (`/`), we enter the pattern we want to use as a replacement in the third position. Finally, we use the "`g`" flag, which stands for replacing all matches. `cat /etc/passwd | grep -v "false\|nologin" | tr ":" " " | awk '{print $1, $NF}' | sed 's/bin/HTB/g'` ([View Highlight](https://read.readwise.io/read/01jm4e6getg8xprjy18aw2em52)) --- Last but not least, it will often be useful to know how many successful matches we have. To avoid counting the lines or characters manually, we can use the tool `wc`. With the "`-l`" option, we specify that only the lines are counted. ([View Highlight](https://read.readwise.io/read/01jm4e7jx3gb8ks2c9jav7wxak)) --- 1. A line with the username `cry0l1t3` - `cat /etc/passwd | grep cry0l1t3` 2. The usernames. - `cat /etc/passwd | cut -d ":" -f1` 3. The username `cry0l1t3` and his UID. - `cat /etc/passwd | cut -d ":" -f1,3 | grep cry0l1t3` 4. The username `cry0l1t3` and his UID separated by a comma (`,`). - `cat /etc/passwd | cut -d ":" -f1,3 | grep cry0l1t3 | tr ":" ","` 5. The username `cry0l1t3`, his UID, and the set shell separated by a comma (`,`). - `cat /etc/passwd | awk -F ":" '{print $1 "," $3 "," $NF}' | grep cry0l1t3` 6. All usernames with their UID and set shells separated by a comma (`,`). - `cat /etc/passwd | awk -F ":" '{print $1 "," $3 "," $NF}'` 7. All usernames with their UID and set shells separated by a comma (`,`) and exclude the ones that contain `nologin` or `false`. - `cat /etc/passwd | awk -F ":" '{print $1 "," $3 "," $NF}' | grep -v "false\|nologin"` 8. All usernames with their UID and set shells separated by a comma (`,`) and exclude the ones that contain `nologin` and count all lines of the filtered output. - `cat /etc/passwd | awk -F ":" '{print $1 "," $3 "," $NF}' | grep -v "nologin" | wc -l` ([View Highlight](https://read.readwise.io/read/01jm4m5n38s8jsvhah4gj3kfhp)) ---