Windows Migration: Terminal - The Power Of The Terminal

In the previous article, I showed you how to access a terminal, how to type in some very simple commands, and what results you could expect to see from them. Nothing very impressive or spectacular, but it worked. It showed that using the terminal in Linux is no black art. It is not something reserved only for computer geeks or the very knowledgeable, but it is something that can be used by anybody that can enter a few simple commands on the keyboard.

So far, you have used only two commands, ls & pwd, but I am sure that you can see that, although these are strange words, they are not mystical incantations from Hogwarts. Rather, these are but a simple way of instructing the system to perform a function on some file or data or to provide some information. There is no sorcery involved in using the terminal, but there is a little learning curve.

To get any significant benefit from using the terminal, you need to be comfortable in its use, and be able to perform actions at least as easily as you do in a graphical environment. You don't need to master it, just be comfortable with it. The X windowing system, the graphical system used by Linux, was not meant to replace the text based command line, but to complement it, and this is still true today. Although you can get along just fine using the graphical interface only, making use of both interfaces gives you access to the total power and flexibility of the Linux operating system.

Under the hood, Linux is different from Windows, and it will take a little while to get used to those differences. I can show you what is available and how to use it, but I shall leave you to decide just how far you want to go and let you explore the possibilities. There are many sources of reference to get a deeper understanding of the topics presented here. The intention of these articles is to make you aware of what is possible, and of how 'Linux - the operating system' works behind the graphical desktop you usually see.

Don't be put off by this; it is by no means as difficult as is often described. A good understanding of just a few of the many hundreds of available commands will get you started on a new level of control of your Linux environment. The real power of the terminal is not knowing all of the commands, but knowing how to use them. In the terminal, you can have total control of all of the features of the commands, not just a sub-set as deemed adequate by the designer of a higher level graphical application. You can make your own commands by combining existing commands to achieve exactly the result you require. There is always more than one way to do something and there is little that is more satisfying than to do it 'your way.' That is the real power of the terminal.

Linux is different

When the Unix operating system, on which Gnu/Linux is based, was devised, a few very important, visionary ideas were introduced. Some equally important concepts were also established.

1. All of the tools and utilities should be designed to do just one thing, but to do it well.

2. There should be a way of combining the effects of applying multiple tools to a set of data. This led to the introduction of the 'pipe' to allow data to flow between tools and through filters to provide a compound, transitional effect to the data. This is known as a data flow or stream.

3. The resultant modified data stream should be re-directed to one or more new destinations, and the input data should be available to be read from any selected source.

4. Errors and system information should have a distinct, concurrent output path that may be logged, reported or combined with the data output stream.

Add to this the fact that Linux/Unix treats everything as a file -- even your mouse is treated as a file -- and you can start to see that Linux is a different beast. The fact that it has been around for over four decades, (dating back to its Unix roots), and that it has been used by many millions of people, and by many thousands of major organisations during that period, is testament to its reliability and usefulness.

If you have used the Windows command line or MS-DOS, then you may well have used the wildcards ? & * to represent unknown characters in filenames, and you may even have written or used batch files. If you have used the new Windows "powershell," then you will have had access to regular expressions, to a command shell, and to an advanced scripting language. Linux has had these features since day one, and a whole lot more.

During the course of these introductory articles, I will show you some commands that are often used, and how to combine them. Occasionally, I shall break off to explain an important concept which you will not have encountered when using Windows.

To start you off, I will show you what can be done in the terminal with the simple ls command and three or four other very simple commands.

Keep it simple, do it well

The ls command lists the contents of a directory (or directories). That's it. That is what the ls command was designed to do, and that is what it does. It does, however, do it extremely well. By applying certain options, we can adjust the output. We can expand or limit the amount of data output by the command. We can produce it in different formats. We can sort the output. We can include subdirectories, and we can include or exclude certain types of files. In short, we can demand to be shown exactly the data we want and only that data.

When you typed in the command ls, the terminal responded by printing a list of the contents of your home directory. Well actually, it only printed out some of the contents. Some files and directories are hidden. There is nothing sinister here. They are your files and you have a right to see them. They are hidden because you rarely need to see them. Hiding them simply reduces clutter under normal use.

A hidden files name begins with a period. To show these, we need to change the ls (list) command, and we can do this by adding an option to the command. There are many options, and they usually begin with one or two hyphens. The one we are looking for is -a. The command ls -a translates as 'list all.'

Now you can see why they were hidden, most of those hidden files will mean absolutely nothing to you -- yet!

Another useful option is -l, and this tells the ls command to produce a long listing.

That gave a lot more information but omitted the hidden files. To include them, we can combine the options like this.

Type ls --help and you will see that there are many more options that you can use. Some of the options may be meaningless to you. Just ignore them. If you don't understand them, then you probably don't need them. Experiment with those that interest you, it is perfectly safe to do so as the command only prints information to the screen. Most commands include this basic usage help.

Data flow & pipes

When you typed in ls --help, some of the information scrolled off the screen, as there was quite a lot of it. The standard tool for displaying a text file on the terminal screen in Unix is a command named more. It reads the data from the file, and then displays one screenful of information at a time, waiting until you press the spacebar. When the spacebar is depressed, it will give you another page, or 'more.' Unfortunately there was no way of going backward to re-read the text, and so a new tool was introduced to include this functionality. What else could it be called but less (modern versions of more do allow bi-directional scrolling, but for many reasons, I would suggest that you stick to using less. After all, "less is more, more or less.") To use these commands, we introduce the vertical bar, or 'pipe' character "|." The data that is output from the first command is piped through to the next command as it is required.

ls --help | less

Now we can use the arrow and page-up/page-down keys to read the information at our own pace. Press 'q' to exit the command.

In this manner complex commands can be constructed, adding commands and filters to produce exactly the results we require. Look at the following example.

The ls command is passed options to produce a long listing of all files with sizes in human-readable form. This is piped to the tr command, which squeezes out extra spaces. This is then passed to the cut command, which removes all columns except columns 5, 6, 7 and 9. Finally, this is again given to the tr command which now converts spaces to tabs. This is a rather complicated example, but it demonstrates the principle of chaining together multiple commands to arrive at just the result you require. Ignore the details for now; just try to understand the basic idea.

Redirection

Commands in Linux have an input and an output even though, at first glance, some seem not to have. The cd command, which is used to change your working location to another directory, appears to have no input and no output. But it does have both, even though the input is not used. Where this input and output come from or go to is arbitrary. When input or output is unspecified, a command will use what are known as 'standard input,' stdin, usually the keyboard, and 'standard output,' stdout, usually the terminal screen. We have seen an example of stdout with the ls command. When we used the pipe symbol '|' the commands output was redirected to another command.

We can also use the < and > characters to redirect stdin and stdout, respectively. It works like this. Suppose we want to save a listing of the files in our Documents directory to a file named docs-list, instead of just printing it to the screen. We would then issue the command ls Documents > docs-list.

If the file doesn't exist, then it will be created, and then the output from the ls command will be written to it. If the file exists, then the contents will be overwritten with the new data. To get around this, we can use two greater than characters, thus ls Documents >> docs-list.

The new output is then appended to the file. If we want to sort the contents of the file, then we can feed the file to the input of the command, which will do our sorting. Unsurprisingly the name of the command we need is sort.

sort < docs-list

This sorts the contents, but prints them out to the screen, leaving the original file untouched. To produce a sorted file in one step we combine redirection ( <,> ) and pipes ( | ) like this:

ls Documents | sort > docs-list

The ls command produces output, which it pipes to the sort command, which then sorts the data and re-directs its output to the destination file.

Error messages

In Linux, stdin and stdout are known as file descriptors, which are a bit like channels that carry data. There can be many active file descriptors, but these are reserved for the operating system, which refers to them as file descriptor 0 and 1. There is a third file descriptor reserved for system use, known as 'standard error' stderr, and this carries the number 2.

Commands used in Linux are programmed to output data to stdout and to output error messages and other information that is not part of the commands intended output to stderr. By default, stderr writes any output to the terminal screen, but that is often inconvenient. You probably don't want error messages mixed with valid data, and if you are running a long process in your absence, perhaps a backup, you may want to save any error messages to a log file that you can refer to when the process completes. This can easily be done by redirecting stderr, or file descriptor 2. If you want to log the output, use a command like this:

command 2> logfile

command 2> logfile > datafile

To simply get rid of any error messages, Linux has a special device called /dev/null. Although a device is usually perceived as a physical device, such as a keyboard or a network card, Linux has some devices that have no physical presence. The device /dev/null/ is one of these. As Linux treats everything, including devices, as files, you can write to /dev/null, which sends all data it receives to irretrievable oblivion.

The Shell

All of this makes for an extremely flexible and powerful set of tools that you can use to work with your files and with the system configuration files. The text based environment in which you use these tools is controlled by a program known as a shell, which interprets what you type before deciding how to execute it. There are many different shells available but the most common one, and the default shell in PCLinuxOS, is named bash.

Bash is extremely powerful and very friendly. Over the many years of its use, lots of features have been added to assist and accelerate your work in the terminal. One of these helpful features is known as command completion.

Type ls D

Press the tab key. Bash is intelligent enough to realise that you have entered the command ls and are now trying to enter the name of a file or directory so that you can list out the relevant information. The shell, bash, responds to the tab key by listing all of the files and directories, that it can see from here, which begin with an uppercase D. The directories Desktop, Downloads and Documents were listed on my system. Typing an additional ,b>o and hitting tab reduces the list to two, as Desktop is not a match. Enter a ,b>c and there is now only one possibility, Documents, and so, by pressing tab again, that is automatically filled in for me at the prompt ready for me to hit Enter, which produces my listing.

This is a real time saver, and can also help when you are unsure of the name of a command. For example, I know that there is another command that begins with ls, and that will give me information about the machines processor.

I type ls and hit Tab. As this is the first thing after the command prompt bash realises that a command is being attempted, and I am shown all of the commands that the system knows and that begin with ls. The one I want is named lscpu (I should have guessed that!). I type c, hit the Tab key and the command name lscpu is completed for me at the prompt. I then press the Enter key, and there is my information.

When you have typed in a command and want to repeat it, you do not have to retype it. Press the up arrow key, and your previous command will appear ready for you to edit or use. Continue pressing the up and down arrow keys to scroll through your previously typed commands.

There is also a feature called reverse search. To use it, press Ctrl + R and start typing what you can remember of a previous command. As you type, the most relevant previous command is placed at the prompt. When you have found what you want, press Enter to execute the command, or the left or right arrow key to edit it.

Finally there is command line history. Type history (his then Tab will finish the typing for you). You will be presented with a numbered list of all of your previous commands. Type an exclamation point (!) followed by the number of the command that you want to repeat, and then press the Enter key to immediately execute that command.

When you find that you often use a command in the same way, with the same options then you can create a shortcut, known as an alias, to save a little typing.

To generate a long listing with human-readable file sizes and sorted by file size the command would be ls -lhS.

Although this is not difficult, can you always remember that the S must be uppercase? The command alias lh='ls -lhS' makes this available by simply typing the new command lh. Unfortunately, this has to be done every time that you open a terminal and start the bash shell. To make this happen automatically, you can add it to a hidden file in your home directory, named .bashrc.

There are many text editors available for the terminal in Linux, and PCLinuxOS has an excellent and easy one to use, named nano.

Type nano ~/.bashrc (there's that squiggle again) and your own copy of this file will open in the editor. You will most likely already have some aliases in here provided by the developers of PCLinuxOS, and a good place to put your new one would be with those, but it doesn't really matter. Type in your new alias on a blank line and then press Ctrl + X. Answer yes when prompted to save. Restart the terminal and your new command is available every time.

To see what other aliases the good folks over at PCLinuxOS development have added for you, type the command alias.