forked from adaptivecomputing/torque
-
Notifications
You must be signed in to change notification settings - Fork 0
/
README.new_in_4.0
218 lines (157 loc) · 8.97 KB
/
README.new_in_4.0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
TORQUE 4.0 What is new
November 2011
*******************************************************
The readme file contains the following sections:
+ New Feature Highlights +
+ New Feature Details +
+ Upgrading and Backward Compatibility +
+ Known Issues +
*******************************************************
+ New Feature Highlights +
*******************************************************
The following is a summary of key new features in TORQUE 4.0
- Multi-threaded pbs_server
-- Faster response to client commands. No more waiting for someone else's call to qstat to
finish before your request can run.
-- Higher job throughput
-- More robust
- New trqauthd client daemon
-- trqauthd replaces pbs_iff
-- trqauthd is multi-threaded and able to reliably handle multiple simultaneous client requests.
-- Run as root. Gives Administrators more control
- Communication
-- RPP (reliable packet protocol) removed
-- UDP removed
-- All communication done using TCP/IP
- Mom Hierarchy
-- Reduces overall compute node update traffic to pbs_server
-- Allows administrators to manage network communications
- Scalability
-- Ability to support over 15,000 compute nodes in a cluster
-- Job radix allows users to submit jobs with very large node needs
*******************************************************
+ New Feature Details +
*******************************************************
This section contains detailed information pertaining to specific new features.
---------------------------------
=== Multi-threaded pbs_server ===
---------------------------------
pbs_server is multi-threaded starting with TORQUE 4.0. By default the number of available
threads is 2*(number of cores)+1. This value can be modified using a set of three
server parameters; min_threads, max_threads and thread_idle_seconds. Each of these
parameters take an integer as an argument and can be set using qmgr.
min_threads: Sets the minimum number of threads that will be available in pbs_server
max_threads: Sets the maximum nuber of threads that can be on pbs_server
thread_idle_seconds: The number of seconds a thread can be idle before it is removed from
the system threadpool. Note the number of threads will never fall below the minimum.
thread_idle_seconds by default is -1. This indicates to TORQUE to never remove a thread
from the threadpool.
By setting max_threads greater than min_threads pbs_server is able to dynamically add threads
as the server load increases. The thread_idle_seconds parameter is used to detect a drop
in the server load and removes threads as they are idle for the number of seconds given in
the parameter.
Setting max_threads equal to min_threads will keep the number of threads static.
---------------------
=== Mom Hierarchy ===
---------------------
The Mom Hierarchy is a new optional feature in TORQUE 4.0 which is designed to improve the
efficiency of communications between pbs_server and the compute nodes (pbs_mom). By default
all compute nodes send status updates directly to pbs_server. As cluster sizes increase the
need to reduce the traffic and time required to keep the cluster status up to date becomes
more important. The Mom Hierarchy allows administrators to configure compute nodes in a way
where each node sends its status update information to another compute node. The compute nodes
pass the information up a tree or hierarchy until eventually the information reaches a node
that will pass the information directly to pbs_server.
Setting up the Mom Hierarchy
The name of the file that contains the configuration information is named mom_hierarchy. By
default it is located in the /var/spool/torque/server_priv directory. The file uses an XML like
syntax. By the time TORQUE 4.0 is released this will be an XML compatible syntax. For now
they syntax is as follows:
<path attr=val>
<level attr=val> comma separated node list </level>
<level attr=val> comma separated node list </level>
...
</path attr=val>
<path attr=val>
<level attr=val> comma separated node list </level>
...
</path attr=val>
...
The <path> </path> tag pair identifies a group of compute nodes. The <level></level> tag pair
contains a comma separated list of compute node names. Multiple paths can be defined with
multiple levels within each path.
Within a <path> tag pair the levels define the hierarchy. All nodes in the top level communicate
directly with the server. All nodes in lower levels communicate to the first available node
in the level directly above it. If the first node in the upper level goes down the nodes in the
subordinate level will then communicate to the next node in the upper level. If no nodes are
available in an upper level then the node will communicate directly to the server.
If an upper level node has fallen out and it is back in again the lower level nodes will eventually
find that the node is available and start sending their updates to that node.
----------------
=== trqauthd ===
----------------
trqauthd is a new daemon starting in TORQUE 4.0. It replaces pbs_iff which is used by
TORQUE client utilities to authorize user connections to pbs_server. Unlike pbs_iff
which is executed by the TORQUE client utilities each time the utility is run, trqauthd
is started once and remains resident. TORQUE client utilities then communicate with
trqauthd on port 15005 on the loopback interface. Unlike pbs_iff trqauthd is multi-threaded
and is able to successfully handle a greater volume of simultaneous requests than pbs_iff.
Running trqauthd
trqauthd MUST be run as root. It must be running on any host where TORQUE client commands
will execute.
By default trqauthd is installed to /usr/local/bin.
trqauthd can be invoked directly from the command line or by the use of init.d scripts
which are located in the contrib/init.d directory of the TORQUE source.
There are three init.d scripts for trqauthd in the contrib/init.d directory of the
TORQUE source tree.
debian.trqauthd
Used for the apt based systems (debian, ubuntu are the most common variations of this)
suse.trqauthd
Used for the rpm based systems. (redhat, suse, scientific, centos, fedora, are some
common examples)
trqauthd
An example for other packages managers. (anything that doesn't use rpm or apt)
Inside each of the scripts are the variable PBS_DAEMON and PBS_HOME. These two variables
should be updated to match your torque installation. PBS_DAEMON needs to point to the
location of trqauthd. PBS_HOME needs to match your TORQUE installation. For more information
about PBS_HOME please see the TORQUE documentation at www.adaptivecomputing.com.
Choose the script that matches your dist system and copy it to /etc/init.d. If needed,
renamed it to trqauthd
To start the daemon type:
/etc/init.d/trqauthd start
To stop the daemon type:
/etc/init.d/trqauthd stop
You can also use the following:
service trqauthd start/stop
---------------------------------------------------
=== Upgrading to 4.0 and Backward Compatibility ===
---------------------------------------------------
Because TORQUE 4.0 has removed all use of UDP/IP and moved all communication to use
TCP/IP previous versions of TORQUE will not be able to communicate with the components
of TORQUE 4.0. However, all files in the /var/spool/torque ($TORQUE_HOME) directory
and all subdirectories are forwardly compatible.
Job Arrays
Job arrays from TORQUE version 2.5 and 3.0 are compatible with TORQUE 4.0. Job arrays
were introduced in TORQUE version 2.4 but modified in 2.5. If upgrading from TORQUE
2.4 you will nned to make sure all job arrays are complete before upgrading.
serverdb
The pbs_server configuration is saved in the file $TORQUE_HOME/server_priv/serverdb.
When TORQUE 4.0 is run for the first time this file will be converted from a binary
file to an XML like format. This format can be used by TORQUE versions 2.5 and 3.0
but not earlier versions. It is recommended this file be backed up before moving to
TORQUE 4.0.
Upgrading
Because TORQUE 4.0 will not communicate with previous versions of TORQUE it is not
possible to upgrade one component and not upgrade the others. Rolling upgrades will
not work.
Before upgrading the system all running jobs must complete. To prevent queued jobs
from starting nodes can be set to offline or all queues can be disabled. Once all
running jobs are complete the upgrade can be made. Remember to allow any job arrays
in version 2.4 to complete before upgrading. Queued array jobs will be lost.
-------------------
=== Know Issues ===
-------------------
As of this writing TORQUE 4.0 is not code complete. All features are complete and have
been addressed in this document but code changes are expected. The success of TORQUE
is directly tied to the involvement of the community. Please feel free to offer
suggestions to improve the documentation.