Idea Transcript
Practical Sessions
Authors: I.M. Smith L. Margetts
NSF/MRCCS Summer School in Parallel Finite Element Analysis 1-5 September 2003
1 Contents This workbook contains the following: a. Logging on instructions. b. A description of the parallel versions of Smith and Griffiths1 (1998) example programs. c. Instructions for compilation and interactive execution. d. Instructions for batch execution using the Load Sharing Facility (LSF). e. Exercises based on Smith and Griffiths1 (1998) programs. f. Crib sheets for UNIX, VI, EMACS and LSF. Further exercises will be provided separately.
2 Logging on 2.1 Overview Users not familiar with the Manchester facilities may find the following explanation helpful: The supercomputers are accessed through a PC that may be running WINDOWS or LINUX. Depending on the room or cluster, a username and password may be required for the PC. Once logged on to the PC, access to the supercomputers is achieved through an intermediate machine named WREN. Users will be issued a separate username and password for WREN. Logging on to WREN will be via a secure terminal on the PC. In this practical session, parallel jobs will be “batch” executed on GREEN, a 512 processor ORIGIN 3800. The submission of jobs to GREEN is handled by a “batch script” and a further username and password for GREEN is unnecessary. Two sets of logging on instructions follow. These are for two distinct clusters. The demonstrator will indicate which instructions should be followed on the day of the practical. You will also be provided the following: • •
Username and password for the PC (if necessary). Username and password for WREN
Note: The following logging on instructions are generic and apply for all practicals.
1
Smith and Griffiths (1998), ‘Programming the Finite Element Method’, 3rd Edition, Wiley
2
2.2
The Lascelles-Williams Room
a. Power up the machine (switch at front) and turn monitor on. b. PAY ATTENTION TO THE SCREEN. c. When the two login options appear, immediately select LINUX and press RETURN. d. When the GNOME DESKTOP MANAGER login window appears, with the left mouse button, double-click on the head labelled USER. e. Click on the left mouse button, highlight XSHELLS then GNOME TERMINAL and press the mouse button again to select this option. A terminal window should now appear. In this window, type: • •
ssh -l wren.cfs.ac.uk Press RETURN. Note will be assigned to you.
f. If asked: • •
Are you sure you want (yes/no?) Type yes and press RETURN.
to
continue
connecting
g. Type your password (provided by the demonstrator) when prompted and press RETURN h. You are now logged on to WREN. Steps 5 to 7 can be repeated to open as many windows on WREN as desired. i. Note that it is not necessary to type any special commands to allow WREN to open XSHELLS on the workstation.
3
2.3 Computer Science Department - Rooms LF34, LF35 and LF39 a. The machine should already be powered up. If you find that it isn't, then power it up (switch at front) and turn monitor on. Go to step 4. b. If your machine shows a LINUX login prompt (i.e. Login: on the command line), go to step 5. Otherwise, re-boot the machine using . c. Wait until a list of boot options appear. Select LINUX and press RETURN. d. When the login prompt appears, type in the username you have been given for the PC's (starts “mcc”) and press RETURN. e. Type your password (provided by the demonstrators) when prompted and press RETURN. f. A menu of choices should now appear. Unless you already have a preference for one of the other LINUX desktops listed, select "2 – USE GNOME". Press RETURN once to choose this option and then again to start up the desktop. g. Wait until the "START HERE" window appears. Remove this by clicking with the left mouse button on the cross in the top-right corner of the window. h. On the panel at the bottom of the screen is a series of icons. The icon that looks like a monitor with a footprint on starts a terminal window running. Do so now by clicking on it with the left mouse button. i. In the terminal window, type: • •
ssh -l wren.cfs.ac.uk Press RETURN. Note will be assigned to you.
j. If asked: • •
Are you sure you want (yes/no?) Type yes and press RETURN.
to
continue
connecting
k. Type your WREN password (provided by the demonstrator) when prompted and press RETURN. l. You are now logged on to WREN. Steps 10 to 13 can be repeated to open as many windows on WREN as desired.
4
3 The Example Programs 3.1 Overview Ten example programs have been written to enable those attending the workshop to experience the results of parallelisation quickly. Inevitably, the examples are rather simple geometrically, and are restricted to cuboidal shapes. The programs are: par510 par67 par73 par87 par88 par95 par96 par105 par117 par118
Static equilibrium: linear elasticity Static equilibrium: elastic/Mohr Coulomb plastic Steady state flow: Laplace Implicit transient flow: “Conduction” or “Consolidation” Explicit transient flow: “Conduction” or “Consolidation” Implicit coupled transient deformation/flow (Biot) Steady state coupled velocity/pressure flow (Navier Stokes) Eigenvalues/vectors of linear elastic solids Implicit dynamic transient equilibrium of linear elastic solids Explicit dynamic transient equilibrium of elastoplastic (von Mises) solid
3.2 Programs par510 and par67 Geometry allowed is cubical and a symmetric quarter is analysed. It represents a vertical load on a square surface patch of an elastic or elastoplastic half-space represented by 20 node elements. z
Top face free to move
y
x Uniformly loaded patch 1/5 of cube side
All 4 side faces are rollers
Base fixed (x,y,z freedoms are zero) Centre Line
Figure 1 Example problem for par510 and par67
5
It is implied that the front and left side vertical faces are planes of symmetry while the back and right side vertical faces are on rollers. In par510, a simple static load is applied while in par67, the load is applied incrementally until failure. The loaded patch is arbitrarily chosen to be 1/5 of the cube’s extent. Thus 5x5x5, 10x10x10, 20x20x20 element meshes would be used. Students could be assisted in to generalise this problem if necessary (and all problems).
par510.dat
NELS NXE NZE NIP AA BB CC E V TOL LIMIT
number of elements in mesh number of elements in x direction number of elements in z direction number of integrating points x dimension of elements y dimension of elements z dimension of elements Young’s modulus Poisson’s ratio Tolerance specified for PCG convergence Iteration limit (number of iterations allowed)
par67.dat
PHI Angle of internal friction (Mohr Coulomb) C Cohesion PSI Dilation angle E Young’s modulus V Poisson’s ratio CONS Isotropic starting stress assumed NELS,NXE,NZE,NIP,AA,BB,CC as above PLASITS Plastic iteration limit CJITS Conjugate Gradient iteration limit PLASTOL Plastic convergence tolerance CJTOL Conjugate Gradient convergence tolerance
6
3.3 Programs par73, par87 and par88 Geometry allowed is a cuboidal box, symmetric quarter analysed. It represents a flow or temperature analysis of a region represented by 8 node elements.
Top face zero boundary conditions
y
z
Rear face zero boundary conditions
x
Centre Line
Zero boundary conditions
Figure 3 Example test problem for par73, par87 and par88
It is implied that the top horizontal, right hand vertical and back vertical faces have boundary condition zero while the bottom horizontal, front vertical and left hand vertical faces are planes of symmetry. In par73 (steady state) a flux of 10 units can be applied, or a potential of 10 units specified, at node C (the “centre” of the box). In par87, par88 the initial conditions are of x units everywhere except at the zero boundaries, and the variation of the result with time is computed at node C.
par73.dat
NELS NXE NZE NIP AA BB CC KX KY KZ TOL LIMIT
Number of elements in mesh Number of elements in x-direction Number of elements in z direction Number of integrating points x dimension of elements y dimension of elements z dimension of elements Conductivity (permeability etc) in x direction Conductivity (permeability etc) in y direction Conductivity (permeability etc) in z direction Tolerance specified for PCG convergence Iteration limit for PCG
7
par87.dat
NELS,NXE,NZE,NIP.AA,BB,CC as above PERMX,PERMY,PERMZ as KX,KY,KZ above DTIM Time step in analyses NSTEP Number of timesteps required THETA ϑ in implicit method 0.5 ≤ ϑ ≤ 1 NPRI Number of steps before results are printed (e.g. every 10th) TOL Tolerance for PCG convergence LIMIT Iteration limit for PCG VAL0 Uniform initial condition par88.dat
NELS, NXE, NZE, NIP, AA, BB, CC, PERMX, PERMY, PERMZ, DTIM, NSTEP, NPRI, VAL0 As par87 above
3.4 Program par95 For coupled (Biot) analysis, this program effectively combines par510 and par87. The same cubical elastic solid described under par510 is coupled with a laminar fluid. 20 nodes are used to represent solid freedoms and 8 to represent fluid freedoms. Zero fluid potential is imposed at the top horizontal boundary only. z
Top face free to move and zero fluid pressure
y x Uniformly loaded patch 1/5 of cube side
All 4 side faces are rollers
Base fixed (x,y,z freedoms are zero) Figure 3 Example problem for par95
8
Again 5x5x5, 10x10x10 etc meshes are implied.
par95.dat
NELS,NXE,NZE,AA,BB,CC,NIP,PERMX,PERMY,PERMZ E,V,DTIM,NSTEP,THETA,CJITS,CJTOL All have the same meanings as previously described.
3.5 Program par96 This analysis is of a complete 3D cubical ‘lid-driven cavity’. The number of elements making up the sides of the cube is arbitrary but the steep gradients in the solution near the lid, and the uniform mesh, mean that at least 10x10x10 elements are advisable.
z
y
All nodes on top face driven with uniform velocity in the xdirection x
No flow (zero velocity) through all 4 side faces and base
Figure 4 Example test problem for program par96
The lid (top of the box) is driven with uniform unit velocity in the x-direction. 20 node elements are used to represent velocities and 8 to represent pressures. A “no-slip”, i.e. zero velocity, condition is applied at all boundaries except at the lid. The pressure is of course not computed at non-corner nodes but exists at all corners except at the left hand edge of the lid (to prevent a singularity).
9
par96.dat
NELS,NXE,NZE,AA,BB,CC As previous. VISC Viscosity of fluid RHO Density of fluid TOL Tolerance in convergence of the “external” solution LIMIT Limit of external iterations CJTOL Tolerance in convergence of BiCGStab CJITS Limit of BiCGStab iterations PENALTY Penalty function for lid movement ELL “l” in BiCGStab(l) KAPPA k in BiCGStab 3.6 Program par105 A simple cantilever made up of 8-node elements has been assumed.
y
z x
Front face is fixed in x, y, and z directions Figure 5 Example test problem for par105
The front x-z face is fixed in all directions and the lowest few eigenvalues/vectors computed. par105.dat
NELS,NXE,NZE,NIP,AA,BB,CC As before RHO Mass density of the elastic solid E,V As before NMODES Number of eigenmodes wanted EL Lower limit of eigenvalue wanted ER Upper limit of eigenvalue wanted IFLAG Error flag (set to -1)
10
EP LALFA LEIG LX LZ ACC
Channel for output information (11) Workspace for eigensolver Workspace for eigensolver Workspace for eigensolver Workspace for eigensolver Accuracy wanted (1.E-8)
3.7 Programs par117 and 118 A simple cantilever is assumed using 20-node elements. Applied load y
z x
Front face is fixed in x, y, and z directions Figure 6 Example test problem for par117 and par118
The front face is fixed in the 3 x, y, z directions. A load is applied at the mid-point of the top of the free end of the cantilever. In the case of par117, the load varies sinusoidally whereas in the case of par118 it is an impact load applied instantaneously at time zero and held constant thereafter. (An even number of elements in the xdirection us necessary for the mid-point load). par117.dat
NELS,NXE,NZE,NIP,AA,BB,CC,RHO,E,V ALPHA1 Rayleigh damping parameter BETA1 Rayleigh damping parameter NSTEP Number of timesteps NPRI Number of steps between printing THETA Implicit timestepping parameter 0.5 ≤ ϑ ≤ 1 OMEGA Frequency of sinusoidally applied force TOL,LIMIT As before
11
As before.
par118.dat
NELS,NXE,NZE,NIP,AA,BB,CC,RHO,E,V SBARY von Mises yield stress PLOAD Value of impact force NSTEP,DTIM,NPRI As before.
12
As before
4 Running the Manchester Example Programs 4.1 Overview The purpose of this practical session is for the student to experience at first hand the benefits of running in parallel. The first three exercises cover the basics, compiling the programs and running them, both interactively and using a batch queueing system. In the remaining exercises, the student is encouraged to investigate the performance characteristics of the programs provided. The programs cover a wide range of engineering problems and the student may select those relevant to their own area of expertise. However, it is suggested that par510 be the first choice as the solution is reasonably short for the resources available.
Exercise 1 – Compilation a. Login to WREN using instructions A or B depending on which training room you are using. . b. Check your user area on WREN contains a directory called ParFE by typing: • •
ls –lrt and change to it by typing: cd ParFE
c. Check the directory contains 10 programs, 10 data files and 10 results files (*.f90 *.dat and *.res) and a file called makefile by typing: •
ls –lrt
d. Select the program you wish to try, eg. par510.f90 and make copies of the data and results files provided by typing: • •
cp par510.dat par510.old.dat cp par510.res par510.old.res
These copies may prove useful if you make a mistake later. e. You now need to edit the makefile using a text editor such as vi or emacs. If you’ve never used vi before, don’t worry. It’s a little tricky to get used to, so you may need a few attempts to get the hang of it. Again, save a good copy: • •
cp makefile makefile.old then open the file using vi: vi makefile
The first line of makefile reads something like prog=par67 where par67 is the program name. Using the instructions below, change the program name to the program you are working on.
13
• • • • • • •
Use the cursor (arrow) keys to position the cursor. Hit ESC to enter command mode. Hit x to delete characters (one hit = one deleted character) Hit ESC to enter command mode Hit i or a to insert characters (many characters can be inserted after one hit of i or a) Type the name of the program Hit ESC to enter command mode
To finish and save changes • •
Hit ESC Hold down the SHIFT key and type zz
Want to abort and have another go? • •
Hit ESC Type quit! and hit ENTER.
Really made a mess of it? Then start again. •
Type cp makefile.old makefile (remember the good copy!)
f. Compile the program •
Type make
g. After compilation, check there is an executable. •
Type ls –lrt
If the compilation failed, consult a demonstrator.
Exercise 2 - Interactive Execution The purpose of this exercise is to test the students understanding of the items in the data file, e.g. par510.dat. a. Referring to the description of the example programs presented earlier, edit the data file as required to run a very small test problem (