Administrating and Monitoring Linux Process

Home / Administrating and Monitoring Linux Process

Administrating and Monitoring Linux Process

December 7, 2015 | Article | No Comments

Whoever can conclude that kernel is the core / brain of Operating System. Yet, computer with kernel only is useless because operating system only do managing computer while task we need such as writing document, watching video, listening to music, etc are a user programs. All of our concern is programs, not the kernel. Operating System just provide environment , where user programs can run.

In today era, people forget the difference between programs and operating system, simply because modern operating system installs a good number of programs along with the operating system itself.

When we running a program, the program will be loaded to memory and working. We can run two or more identical program, but in memory they are treated as different program. For example, you open notepad twice, one for editing a.txt and other for editing b.txt. Although both are notepad but they work for different purpose. And the program running in memory is referred as a process.

A program, can start one or multiple process to complete the task it was designed to do. Processes which get started will execute and die on its own without any user intervention. And the one who has responsible for such thing is kernel itself.

In this article we will discuss about different aspects of a process in Linux. As Linux is a mimic of Unix Operating System, you may find it similar to other Unix-based Operating System.

Process Identification (PID)

Each and every process in Linux Operating System is identified by a number which is referred as process id.

Process ID, or PID, is nothing more than a unique identification number to identify a process in Linux. When we say unique, it means two process running at same time cannot have same PID. PIDs are 16-bit number that are sequentially assigned to different process when they are spawned.

After the execution (when process exit), the PID number is released and can be reused by other process. In Linux, to handle processes Kernel maintains a table of process. The name is simply Process Table. Process table store any information of process and of course they are identified by their PID. The PID number start from 1.

Like I said before, when a process exit its PID is released and can be reuse by other process. We also cannot guarantee a process can get same PID when it is executed (ex: today A got PID number 1653, tomorrow it can be 1435, 1206, or anything). But, we can guarantee that there is only one process that get same PID number each and everytime. This process is INIT.

INIT is the first program that’s run by Linux system. Therefore, Init will always get PID 1. It is also parent of all other process in Linux. Here is some point to make the concept clear:

  1. Init is the parent and the first process, which always gets the PID number 1
  2. There is a limit to the maximum number of processes that can be run on a Linux operating system.

The maximum number of PID can be found from the file /proc/sys/kernel/pid_max. Commonly the maximum PID is set as 32768, which means 32768 number of process can be run simultaneously at a time in system. Let’s remember that Linux is assigning PID sequentially which means a PID number 435 won’t be given unless PID number 434 already given. However, this won’t always true. When PID counting reach maximum PID allowed, Linux will start from beginning and search for available PID number.

Higher PID number such as 30000 on a system does not mean 30000 process are running on system. It means 30000 process has been run so far (which implied some might already dead). And also, a PID number of 1000 does not always mean that it was started before the PID number 30000

Listing Running-Process

There are different tools available in Linux. A common tool used is ps. Here is the example of its usage:

[email protected]:/# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.1  0.1   2064   624 ?        Ss   08:16   0:00 init [3]
root         2  0.0  0.0      0     0 ?        S<   08:16   0:00 [migration/0]
root         3  0.0  0.0      0     0 ?        SN   08:16   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S<   08:16   0:00 [watchdog/0]
root         5  0.0  0.0      0     0 ?        S<   08:16   0:00 [events/0]
root         6  0.0  0.0      0     0 ?        S<   08:16   0:00 [khelper]
root         7  0.0  0.0      0     0 ?        S<   08:16   0:00 [kthread]
root        10  0.0  0.0      0     0 ?        S<   08:16   0:00 [kblockd/0]
root        11  0.0  0.0      0     0 ?        S<   08:16   0:00 [kacpid]
root       175  0.0  0.0      0     0 ?        S<   08:16   0:00 [cqueue/0]
root       178  0.0  0.0      0     0 ?        S<   08:16   0:00 [khubd]
root       180  0.0  0.0      0     0 ?        S<   08:16   0:00 [kseriod]

There we got list of running process. Let’s inspect each column given.

USER
Gives us clue about the name of the user who started the process. In other words, the owner of process.
PID
The PID number of process. And yes we see init has PID number 1.
%CPU
The amount of CPU usage in percentage for process.
%MEM
The amount of memory utilized by process, in percentage.
VSZ
The amount (size) of virtual memory used.
RSS
Resident Set Size. Portion of RAM (Physical memory) that a process is using.
TTY
The terminal name from where the process started. The ‘?’ mark indicated that the process is not run from terminal. It can be either a daemon, or process started by the system, or a process related to a cron job.
STAT
Will be discussed in next section
START
The time when process started.
TIME
a
COMMAND
Command or the name of running program.

The Process State (in ps)

A process has a state which obviously state the condition of process. A process have different state option and it can be classified into different states, depend on its current status. For example, a process can be idle, waiting for a user input, a process can be waiting for an IO request to be completed, A process can be zombie, or even orphan. Let’s go through different states of a process in Linux.

Let’s first understand interruptible and uninterruptible processes in Linux.

Processes which are not currently doing any task, and are waiting for any interrupt, or say for example a process which is waiting for an input from the user is normally in the interruptible sleep state. This state of a process is denoted by “S”. You will find a large number of processes in the system with an S state. This is because, most of the processes are in the started state, and are waiting for something to execute as required.

Processes that are waiting for something to get completed are said to be in uninterruptible state. For example, a process that is waiting for an IO request to be completed goes into an Uninterruptible state. Such a state is denoted by D in the ps command output. You cannot send any message, to a process that is in uninterruptible state, because it will not accept any signal or message. You will normally not see, too many processes in D state.

Another state here is Zombie which represented by Z. This kind of process will be discussed later.

Process status column also tells the priority of the process over other processes. There are two symbols that indicate the priority of the process.

“<” indicate’s that the process is having a higher priority than other processes. In other words, this kind of process having VIP access and will be treated special than other process. On the other hand N, indicates that the process is nice to other’s. Which means a process with the status of has a lower priority compared to other processes running on the system. If no < or it means the process is “regular” process and will be treated using usual handle.

The Zombie Process

We have discussed that INIT process is the first program called during boot time. INIT is the parent process of all other processes in Linux. All processes that run on Linux has a parent process to it (directly or indirectly).

A zombie process is a process which has characteristics of a real zombie (dead body without soul is called as a zombie). It is a dead process without any task left to do (without utilizing any system resource). When a child process completes its execution, it informs its parent process that it has completed its execution. After informed, parent has job to get the complete status of the process and then finally remove it from the process table so the PID Is available for reuse. But there are some irresponsible parent process that takes a lot of time to collect complete status of its children, even after being informed. Due this reason, the children becomes zombies till the parent does its job of collecting information and removing them form the list.

The “irresponsible” parent is the process who are responsible for the existence of Zombie Process. This might happen due to programming inefficiencies and bugs. But generally a zombie cannot cause any sort of harm, because they are not consuming any system resource, just sitting in the process table. But if a large number of zombie process occurred, we might runs out of PID number.

A parent can be forced to reap their child processes. This can be done by sending SIGCHLD signal to parent. This signal will ask the parent to reap their children who are zombies. Suppose 1234 is the parent process of a zombie process, we can send signal to that parent by:

kill -s SIGCHLD 1234

Note that kill command is not exactly killing a target (in this case 1234), but sending a signal to that target. A quick command to list all zombie process and also their parent would be:

ps aux | awk '{ print $8 " " $2 }' \
| grep -w S | awk '{ print $2 }' \
| while read line; do
echo $line " has parent " `ps -p $line -o ppid=`;
done

The zombie process is on the left side while the parent is on the right side of “ has parent “.

Orphan Process

Orphan process is literally a process which has became orphan. It is unfortunate process whose parents die leaving the children behind. Parents can die due to accident (ex: crash). However some process immediately adopts the children whose parents have been died. In this example, INIT adopts almost all orphan process immediately. In such case, we will see PID Number 1 will be their parent process.

In Linux and Unix, whenever parent get killed, all children get destroyed. This normally happen, until and unless, a process is intentionally made orphan, so that init adopts it as its child. This process of intentionally making a process orphan, is useful when we want to run a program from a shell, for a very long period of time, and it also does not require any user intervention in between. Normally when we run a program from a command line shell, the program becomes the child of the shell process. So when we logout from the shell, all child gets killed. But in some cases, we want the process to run until it gets completed by itself.

This can be achieved by purposely making a process orphan, so that the INIT process immediately adopts it. This is done with the help of a command called as nohup in Linux.

nohup sh sample.sh &

The above command starts the sample.sh process with nohup, so that it does not get killed even after the shell is exited. And the final & indicates that the process should be run in background.

Foreground and Background Process

Previously when we were running our example script using & at the end of the command, it makes that process go in the background. Now let’s understand what’s a background and foreground process.

To understand what is background and foreground, let’s see an example. If we are working on a linux terminal, then each and every command we execute will first need to complete, before we get the shell prompt. For example, if we want to run a command like yum update, from terminal, we need to first wait for the command to complete, before we can run another command on the same terminal.

What if we want to run multiple commands one after the other at the same time, without waiting for each of them to complete. In order to do this, we need to send a command (the process), to the background. A background process will run concurrently with the foreground process. Whenever you run a command, its by default run on the foreground. But there are methods that can be used to send a foreground process to background.

apt-get update

When we run the above command, it will run the foreground by default, and we will have to wait till the command completes, to get the shell prompt back again to run another command. But we can pause the process or suspend the process by pressing CTRL + Z. If we do that, some words will be prompt back to shell, something like this:

[1] + Stopped                 yum update

As soon as we press CTRL + Z, the process was suspended/stopped. Which is clear from the above message [1]+ stopped yum update. Pressing CTRL+Z, is not a good solution to run another command or to get the shell prompt back, because it suspends the already running process. We need the system to run this process, as well as allow us to run another commands, by releasing the shell prompt. Moreover, we cannot say we are running multiple program if we suspend one, right? 😀

On the previouse prompt, [1] indicates the job number id. This number is differ with PID number. As we only have one job here, which is suspended it is given the job id number of 1. To resume the suspended process, we can execute fg followed by it’s job number id.

[email protected]:~/# fg %1
yum update

But even if we resume the process, with the help of fg %1, it will once again come to the foreground, and we still need to wait till the process gets completed, to run another commands. A good solution to this problem is to send the process to background, so that the process will continue to run, as well you will get the shell prompt back, to run another commands.

[email protected]:~/# bg %1
[1]+ yum update &

The above command, will resume the yum update process and will also put that process in the background, so that you can run other commands. If you have multiple processes, running in the background, then you can view their status, with the help of jobs command, as shown below.

[email protected]:~/# jobs
[1]+  Running                 yum update &

You can anytime switch between background and foreground with the help of fg and bg command in linux, by simply giving the exact job number as the argument(%1,%2 etc)

Alternatively you can always use the symbol &, immediately after the command, to send it to the background. So for example, let’s send the same yum update command to background during run time.

[email protected]:~/# yum update &
[1]  1234

As you can see, using the &, symbol will also do the same thing of assigning a job id number and sending it to background.

Monitoring a Process

We have used one process-monitor command, ps. Ps can list process and gives details. However we cannot see it realtime as ps only show all process running when ps command execute. In other word, it is static. If we want to monitor process for any single second, we cannot execute ps consecutively. Also, ps is static which means we can not see the dynamic of process. The dynamic of process for example: a process with PID x suddenly consume much CPU. For that purpose, we cna use another tool, top.

When we execute top command, we will be prompted with a screen and some information. For example:

top - 05:23:08 up  7:09,  4 users,  load average: 0.00, 0.11, 0.10
Tasks:  94 total,   1 running,  93 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.7%us,  0.7%sy,  0.0%ni, 97.0%id,  0.0%wa,  0.3%hi,  0.3%si,  0.0%st
Mem:    515444k total,   441964k used,    73480k free,    73364k buffers
Swap:        0k total,        0k used,        0k free,   276772k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
31571 root      15   0  2196 1008  804 R  0.7  0.2   0:00.02 top
4535 root      15   0  1968  644  564 S  0.3  0.1   0:23.77 hald-addon-stor
1 root      15   0  2064  624  536 S  0.0  0.1   0:00.94 init
2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/0
3 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0

top command will keep on updating it stats (by default: 3 seconds). However we can change this value by:

top -d 1

which top will updating every 1 second.

Although top command is a pretty nice tool to monitor process details on a linux system, it also provides us the complete overview of the system in one interface. If you see the top area of the command output, you will be able to see the complete details of a running system.

Some details we can get from above state:

  • There is no user logged in
  • System Load average
  • Uptime
  • Total number of process, with how many are sleeping, running, stopped, or zombie.
  • CPU statistics, memory statistics

You can sort the output of the top command, based on memory usage, cpu usage, swap usage etc. With the help of sorting option’s available in the command. You can get the options by pressing SHIFT + O, which will display the list of sorting options. Pick one and we can get sorted process based on profile we chose.

Another interesting command for monitoring a process is pstree. This will display the parent child relationship of all process in the system using tree graph. For example this is the snapshot of the result of my pstree.

init-+-NetworkManager-+-dhcpcd
|                `-2*[{NetworkManager}]
|-acpid
|-6*[agetty]
|-akonadi_control-+-4*[akonadi_agent_l---{akonadi_agent_}]
|                 |-akonadi_archive
|                 |-akonadi_maildis
|                 |-akonadi_mailfil
|                 |-akonadi_nepomuk---{akonadi_nepomu}
|                 |-akonadiserver-+-mysqld---35*[{mysqld}]
|                 |               `-21*[{akonadiserver}]
|                 `-{akonadi_contro}
|-amarok---18*[{amarok}]
|-atd
|-bluedevil-monol
|-bumblebeed
|-2*[conky---3*[{conky}]]
|-console-kit-dae---64*[{console-kit-da}]
|-crond
|-2*[dbus-daemon]
|-dbus-launch
|-gam_server
|-gconfd-2
|-gpm
.
.
.
|-syslogd
|-udevd---2*[udevd]
|-udisks-daemon-+-udisks-daemon
|               `-2*[{udisks-daemon}]
|-upowerd---2*[{upowerd}]
`-wpa_supplicant

Now you can see that Ini is at the top, which means it is the parent of all process.

Most of the command line utilities that shows us the process details, actually fetch their information from the /proc. This is node created by filesystem to maintain process. If we go to /proc, we will be able to see separate directories for each running process. For example, this is my result:

dr-xr-xr-x 240 root       root                     0 May 13  2013 ./
drwxr-xr-x  28 root       root                  4096 May  9 18:56 ../
dr-xr-xr-x   8 root       root                     0 May 13  2013 1/
dr-xr-xr-x   8 root       root                     0 May 13  2013 10/
dr-xr-xr-x   8 root       root                     0 May 13  2013 1024/
dr-xr-xr-x   8 root       root                     0 May 13  2013 1070/
dr-xr-xr-x   8 root       root                     0 May 13  2013 1071/
dr-xr-xr-x   8 root       root                     0 May 13  2013 11/
dr-xr-xr-x   8 root       root                     0 May 13  2013 1141/
dr-xr-xr-x   8 root       root                     0 May 13  2013 1142/
dr-xr-xr-x   8 root       root                     0 May 13  2013 1143/
dr-xr-xr-x   8 root       root                     0 May 13  2013 1146/
dr-xr-xr-x   8 root       root                     0 May 13  2013 1150/
dr-xr-xr-x   8 root       root                     0 May 13  2013 12/

If we go to one directory, let say 1141 and list the inside, we can get something like this:

attr             cpuset   limits      net            root       status
auxv             cwd      loginuid    ns             sched      syscall
cgroup           environ  maps        oom_adj        sessionid  task
clear_refs       exe      mem         oom_score      smaps      wchan
cmdline          fd       mountinfo   oom_score_adj  stack
comm             fdinfo   mounts      pagemap        stat
coredump_filter  io       mountstats  personality    statm

We have short time here so I won’t cover each directory here. At least, let’s understand some of them.

    1. cmdline file tells about the command line arguments that were used while running this process
    2. cwd links the process to the working directory of the process
    3. exe is the executable of the process.
    4. environ file contains the environment variables of the process

status

    contains a complete information about the process. Most of the system monitoring tools fetch process information form this location only.

Sending Signal to Process

We have seen SIGCHLD before. We send signal using kill. Most of the people who use Linux in day to day are aware of this command. Like I said, although it has name kill, it can achieve more than killing. It is used for sending a signal. If no signal specified, killing-signal will be used.

There are various signal can be sent to a process. The way the process responds to a particular signal, depends upon how that process was programmed. Let’s discuss some commonly used kill signal’s

We have discussed that programs run from a terminal get’s exited, when you exit the terminal. Basically when you exit the terminal, the program’s run from that terminal receives a SIGHUP signal. SIGHUP signal tell’s the process to hang up. Basically this signal will terminate the process.

What happens when you type CTRL + C or CTRL + Z, to a running process. CTRL + C will send a SIGINT, signal to the running process (Most of the times, it will terminate the running application). We have previously seen that a running program can be suspended using a CTRL + Z. The shell will send a SIGSTOP signal to a running program when you press CTRL + Z.

Kill command is used with different signal level’s as an argument. The signal type can be specified as word such as SIGHUP, SIGINT, SIGKILL  or as their equivalent number such as 1, 2, 9 etc. Let’s see the following snippet. The first two commands are the basic syntax which are equivalent to each other. The last two are the working example of the first two.

kill -s <SIGNAL NAME> <PID NUMBER>
kill -<SIGNAL NUMBER> <PID NUMBER>
kill -s SIGINT 1234
kill -2 1234

About Author

about author

xathrya

A man who is obsessed to low level technology.

Leave a Reply

Your email address will not be published. Required fields are marked *

Social media & sharing icons powered by UltimatelySocial