Actions

Difference between revisions of "Specifying memory requirements"

From ALICE Documentation

(Created page with "Specifying memory requirements In many situations, it is useful to monitor the amount of memory an application is using. You need this information to determine the characteri...")
 
Line 1: Line 1:
Specifying memory requirements
+
==Specifying memory requirements==
  
 
In many situations, it is useful to monitor the amount of memory an application is using. You need this information to determine the characteristics of the required compute node, where that application should run on. Estimating the amount of memory an application will use during execution is often non-trivial, especially when one uses third-party software.
 
In many situations, it is useful to monitor the amount of memory an application is using. You need this information to determine the characteristics of the required compute node, where that application should run on. Estimating the amount of memory an application will use during execution is often non-trivial, especially when one uses third-party software.
  
Available Memory on the machine
+
===Available Memory on the machine===
  
 
The first point is to be aware of the available free memory in your computer. The“free” command displays the total amount of free and used physical and swap memory in the system, as well as the buffers used by the kernel. We also use the options “-m” to see the results expressed in Mega-Bytes and the “-t” option to get totals.  
 
The first point is to be aware of the available free memory in your computer. The“free” command displays the total amount of free and used physical and swap memory in the system, as well as the buffers used by the kernel. We also use the options “-m” to see the results expressed in Mega-Bytes and the “-t” option to get totals.  
Line 13: Line 13:
 
Important is to note the total amount of memory available in the machine (i.e., 16 GB in this example) and the amount of used and free memory (i.e., 4.7 GB is used and another 11.2 GB is free here). It is not a good practice to use swap-space for your computational applications. A lot of “swapping” can increase the execution time of your application tremendously.
 
Important is to note the total amount of memory available in the machine (i.e., 16 GB in this example) and the amount of used and free memory (i.e., 4.7 GB is used and another 11.2 GB is free here). It is not a good practice to use swap-space for your computational applications. A lot of “swapping” can increase the execution time of your application tremendously.
  
Checking the memory consumption
+
===Checking the memory consumption===
  
 
To monitor the memory consumption of a running application, you can use the “top” or the “htop” command.
 
To monitor the memory consumption of a running application, you can use the “top” or the “htop” command.
Line 19: Line 19:
 
top provides an ongoing look at processor activity in real time. It displays a listing of the most CPU-intensive tasks on the system, and can provide an interactive interface for manipulating processes. It can sort the tasks by memory usage, CPU usage and run time. htop is similar to top, but shows the CPU-utilisation for all the CPUs in the machine and allows to scroll the list vertically and horizontally to see all processes and their full command lines.
 
top provides an ongoing look at processor activity in real time. It displays a listing of the most CPU-intensive tasks on the system, and can provide an interactive interface for manipulating processes. It can sort the tasks by memory usage, CPU usage and run time. htop is similar to top, but shows the CPU-utilisation for all the CPUs in the machine and allows to scroll the list vertically and horizontally to see all processes and their full command lines.
  
$ top  
+
  $ top  
 
+
  $ htop
$ htop
 
  
 
Setting the memory parameter
 
Setting the memory parameter
Line 27: Line 26:
 
Once you gathered a good idea of the overall memory consumption of your application, you can define it in your job script. It is wise to foresee a margin of about 10%. Sequential or single-node applications: The maximum amount of physical memory used by the job can be specified in a job script as:
 
Once you gathered a good idea of the overall memory consumption of your application, you can define it in your job script. It is wise to foresee a margin of about 10%. Sequential or single-node applications: The maximum amount of physical memory used by the job can be specified in a job script as:
  
<nowiki>#</nowiki>PBS -l mem=4gb
+
  # PBS -l mem=4gb
 
+
  $ qsub -l mem=4gb
$ qsub -l mem=4gb
 
  
 
This setting is ignored if the number of nodes is not 1. Parallel or multi-node applications: When you are running a parallel application over multiple cores, you can also specify the memory requirements per processor (pmem). This directive specifies the maximum amount of physical memory used by any process in the job. For example, if the job would run four processes and each would use up to 2 GB (gigabytes) of memory, then the memory directive would read:
 
This setting is ignored if the number of nodes is not 1. Parallel or multi-node applications: When you are running a parallel application over multiple cores, you can also specify the memory requirements per processor (pmem). This directive specifies the maximum amount of physical memory used by any process in the job. For example, if the job would run four processes and each would use up to 2 GB (gigabytes) of memory, then the memory directive would read:

Revision as of 13:34, 17 April 2020

Specifying memory requirements

In many situations, it is useful to monitor the amount of memory an application is using. You need this information to determine the characteristics of the required compute node, where that application should run on. Estimating the amount of memory an application will use during execution is often non-trivial, especially when one uses third-party software.

Available Memory on the machine

The first point is to be aware of the available free memory in your computer. The“free” command displays the total amount of free and used physical and swap memory in the system, as well as the buffers used by the kernel. We also use the options “-m” to see the results expressed in Mega-Bytes and the “-t” option to get totals.

free -m -t

total used free shared buffers cached Mem: 16049 4772 11277 0 107 161 -/+ buffers/cache: 4503 11546 Swap: 16002 4185 11816 Total: 32052 8957 23094

Important is to note the total amount of memory available in the machine (i.e., 16 GB in this example) and the amount of used and free memory (i.e., 4.7 GB is used and another 11.2 GB is free here). It is not a good practice to use swap-space for your computational applications. A lot of “swapping” can increase the execution time of your application tremendously.

Checking the memory consumption

To monitor the memory consumption of a running application, you can use the “top” or the “htop” command.

top provides an ongoing look at processor activity in real time. It displays a listing of the most CPU-intensive tasks on the system, and can provide an interactive interface for manipulating processes. It can sort the tasks by memory usage, CPU usage and run time. htop is similar to top, but shows the CPU-utilisation for all the CPUs in the machine and allows to scroll the list vertically and horizontally to see all processes and their full command lines.

 $ top 
 $ htop

Setting the memory parameter

Once you gathered a good idea of the overall memory consumption of your application, you can define it in your job script. It is wise to foresee a margin of about 10%. Sequential or single-node applications: The maximum amount of physical memory used by the job can be specified in a job script as:

 # PBS -l mem=4gb
 $ qsub -l mem=4gb

This setting is ignored if the number of nodes is not 1. Parallel or multi-node applications: When you are running a parallel application over multiple cores, you can also specify the memory requirements per processor (pmem). This directive specifies the maximum amount of physical memory used by any process in the job. For example, if the job would run four processes and each would use up to 2 GB (gigabytes) of memory, then the memory directive would read:

#PBS -l pmem=2gb

or on the command line

$ qsub -l pmem=2gb

(and of course this would need to be combined with a CPU cores directive such as nodes=1:ppn=4). In this example, you request 8 GB of memory in total on the node.