Tutorial
If you haven't already, read through the Getting_started page. Skip the Getting_started#Compiling_libraries section if someone has set up the libraries and Makefile for you.
The following assumes that the code has been downloaded (Main_Page#Download), and that libraries have been correctly installed (Getting_started#Compiling_libraries), so that the command 'make' does not exit with an error.
Where to start - initial conditions and the main.info file
The best input initial condition is usually the output state from another run, preferably from a run with similar parameter settings. Output state files are named state0000.cdf.dat, state0001.cdf.dat, state0002.cdf.dat, and so on. Any of these could be used as a potential initial condition. If resolution parameters do not match, then they automatically interpolated or truncated to the new resolution (the resolution selected at compile time).
Download the following file from the Database: File:Re2400a1.25.tgz and extract the contents (replace '...' with appropriate paths):
> cd .../openpipeflow-1.02b/ > mv .../Re2400a1.25.tgz . > tar -xvvzf Re2400a1.25.tgz
This should produce a directory Re2400a1.25/ containing an output state file state0010.cdf.dat and a main.info file. The main.info file is a record of parameter settings that were used when compiling the executable that produced the state file.
A more computationally demanding example: You may wish to start from File:Re5300.Retau180.5D.tgz and compare data with Eggels et al. (1994) JFM. To run at a reasonable pace try running on 5-10 cores.
Set your parameters
We will assume serial use (for parallel use see Getting_started#Typical_usage).
The number of cores is set in parallel.h. Ensure that _Nr and _Ns are defined to be 1 (number of cores _Np=_Nr*_Ns):
> head parallel.h ... #define _Nr 1 #define _Ns 1 ...
If not, edit with your favourite text editor, e.g.
> nano parallel.h [OR] > pico parallel.h [OR] > gedit parallel.h
Next, in another terminal window, take a look at the main.info downloaded a moment ago
> less Re2400a1.25/main.info
('less' is like 'more', but you can scroll and search (with '/'). Press 'q' to exit.) In the previous terminal window, edit the parameters so that they are the same as in the given main.info file
> nano program/parameters.f90
You should ignore from 'i_KL' onwards.
Compile and setup a job directory
After setting the parameters, we need to create an executable that will run with the settings we've chosen. To compile the code with the current parameter settings
> make > make install
If an error is produced, go back to the top and check that libraries and Makefile are set up correctly. The second command creates the directory install/ and a new main.info file. Often its a good idea to check for any differences in parameters between jobs:
> diff install/main.info Re2400a1.25/main.info
We'll create a new job directory with an initial condition in there ready for the new run
> cp Re2400a1.25/state0010.cdf.dat install/state.cdf.in > mkdir ~/runs/ > mv install ~/runs/job0001 > cd ~/runs/job0001 > ls -l -rw-rw-r-- 1 949 Sep 9 08:50 main.info -rw-rw-r-- 1 3679640 Sep 9 08:50 main.out -rw-rw-r-- 1 3177740 Jun 10 2013 state.cdf.in
Start the run
To start the run
> nohup ./main.out > OUT 2> OUT.err &
'&' puts the job in the background. 'nohup' allows you to logout from the terminal window without 'hangup' - otherwise, a forced closure of the window could kill the job. Output and errors normally sent to the terminal window are redirected to the OUT files.
A few seconds after starting the job, press enter again. If there is a message like
'[1]+ Done nohup ./main.out ...'
then it is likely that there was an error. In that case, try
> less OUT > less OUT.err
If there is a message about MPI libraries, and earlier you changed _Np from another value to 1, then you could try running instead with
> mpirun -np 1 ./main.out > OUT 2> OUT.err &
You might need to include a path to mpirun; search your Makefile for the mpirun command.
Monitor the run
Let the code run for a few minutes, then type
> ls -l -rw-rw-r-- 1 18 Sep 9 08:56 HOST -rw-rw-r-- 1 949 Sep 9 08:54 main.info -rw-rw-r-- 1 15443 Sep 9 09:16 OUT -rw-rw-r-- 1 0 Sep 9 08:56 OUT.err -rw-rw-r-- 1 106 Sep 9 08:56 RUNNING -rw-rw-r-- 1 3177740 Jun 10 2013 state.cdf.in -rw-rw-r-- 1 3177740 Sep 9 08:56 state0000.cdf.dat -rw-rw-r-- 1 3177740 Sep 9 09:00 state0001.cdf.dat -rw-rw-r-- 1 3177740 Sep 9 09:04 state0002.cdf.dat -rw-rw-r-- 1 22821 Sep 9 09:16 tim_step.dat -rw-rw-r-- 1 32431 Sep 9 09:16 vel_energy.dat -rw-rw-r-- 1 32431 Sep 9 09:16 vel_friction.dat -rw-rw-r-- 1 2504 Sep 9 08:56 vel_prof0000.dat -rw-rw-r-- 1 2504 Sep 9 09:00 vel_prof0001.dat -rw-rw-r-- 1 2504 Sep 9 09:04 vel_prof0002.dat -rw-rw-r-- 1 2801 Sep 9 08:56 vel_spec0000.dat -rw-rw-r-- 1 2801 Sep 9 09:00 vel_spec0001.dat -rw-rw-r-- 1 2801 Sep 9 09:04 vel_spec0002.dat
A number of new files should have appeared. The code outputs snapshot data, which includes a 4-digit number e.g. vel_spec0003.dat and state0012.cdf.dat, and time-series data, e.g. the energy as a function of time vel_energy.dat. Snapshot data is saved every i_save_rate1 (typically 2000) timesteps, and time-series data is saved every i_save_rate2 (typically 10) timesteps .
To see how far it has run, try
> tail OUT step= 4050 its= 1 step= 4060 its= 1 step= 4070 its= 1 step= 4080 its= 1 step= 4090 its= 1 step= 4100 its= 1
Or
> tail vel_energy.dat 0.405000000000E+02 0.237759801033E+00 0.214047103034E+00 0.176073100060E+00 0.406000000000E+02 0.237720816537E+00 0.214012492019E+00 0.176086005951E+00 0.407000000000E+02 0.237683667806E+00 0.213980401111E+00 0.176099745534E+00 0.408000000000E+02 0.237648349327E+00 0.213950865483E+00 0.176114304963E+00 0.409000000000E+02 0.237614853619E+00 0.213923919413E+00 0.176129666838E+00 0.410000000000E+02 0.237583171343E+00 0.213899596127E+00 0.176145810402E+00
Column 1 of vel_energy.dat is the time, column 2 is the energy in the perturbation to the mean flow. Columns 3 and 4 are the energies in the axially averaged (k=0) and azimuthally averaged (m=0) components respectively. Sometimes the time-series files are buffered, i.e. no output is written until the buffer is filled. The OUT file is less likely to be buffered.
Let the code run for a few minutes, give it a chance to save a few state files. To see the simulation times when the state were saved,
> grep state OUT | less loading state... saving state0000 t= 0.000000000000000E+000 saving state0001 t= 20.0000000000003 saving state0002 t= 40.0000000000006
Let's plot the energy as a function of time:
> gnuplot > plot 'vel_energy.dat' w l [with lines] > plot 'vel_energy.dat' u 1:($2-$3) w l [using column1 and column2-column3]
The last line here plots the energy in the axially dependent modes only (k non-zero). This quantity decays rapidly after relaminarisations, and the simulation will stop if it drops below the parameter value d_minE3d, e.g. 1d-5. Note that this parameter has no effect if it is set to e.g. -1d0.
It is a good idea to keep track of the resolution. Still in gnuplot
> set log > plot 'vel_spec0001.dat' u ($1+1):2 w lp [with lines and points] ... > quit
This is a rough plot indicating the drop-off in amplitude of coefficients (the drop-off in energies will be the square of these values), defined by E_k = max_{nm} a_{nkm}, E_m = max_{nk} a_{nkm}, E_n = max_{km} a_{nkm}, where n is the index of axial resolution, k for axial and m for azimuthal.
Ideally, the lines should all drop by around 3-4 orders of magnitude or more. If there is an upward spike at the end of one of the lines, then this is usually the signature of a timestep that is too large.
Ignore the zig-zag for the tail of the line for radial resolution, focus on the upper values in the zig-zag only. It arises because a finite difference scheme is used, but data has been transformed onto a spectral basis, purely for the purpose of gauging the quality of resolution.
End the run
Type
> rm RUNNING
and press enter. This signals to the job to terminate (cleanly). Wait a few seconds then press enter again. There should be a message like, '[1]+ Done nohup ./main.out ...', to say that the job has has ended.
To see how long and how fast it was running, do
> tail OUT step= 10480 its= 1 step= 10490 its= 1 step= 10500 its= 1 RUNNING deleted ! cleanup... sec/step = 0.1118212 CPU time = 19 mins. saving state0006 t= 105.100000000017 ...done!
Make a util
Here we'll build a utility code to process data for visualisation.
Let's return to the code, cd .../openpipeflow-1.02b/
The core code in program/ rarely needs to be changed. Almost anything can be done by creating a utility instead. There are many examples in utils/. Further information can be found on the Utilities page.
In Makefile, put
UTIL = prim2matlab [ommitting a .f90 extension]
In the utils/ directory there is a corresponding file prim2matlab.f90.
> make > make install > make util
Really, only the last command is necessary, which creates the executable prim2matlab.out. It is good practice, however, to do the previous commands to generate a main.info file to keep alongside the executable.
Visualise
Fourier coefficients are stored in the state files. (It takes much less space and is more convenient should resolution be changed.) The utility utils/prim2matlab.f90 converts data to real space data, and outputs it to a NetCDF file that is easily read by Matlab and Visit software packages.
The output mat_vec.cdf:
- contains variables A (the data) and x,y,z (grid points).
- x has dimension x(nx).
- A has dimension A(nx,ny,nz,Ads).
- Ads=1 for scalar data, Ads=3 for a vector.
- nz=1 if data is confined to a cross section.
This section is to be extended. For the moment, please see scripts that accompany the code in the directory matlab/, in particular the file matlab/Readme.txt .