Configuration
=============
This documention provides instructions on how to configure a Folding\@home
Work Server and the projects on it.
Global Configuration
--------------------
During install a default configuration file will be created at
``/etc/fah-work/config.xml``. The contents should look something like this:
.. code-block:: xml
ws.example.com
``xxx.xxx.xxx.xxx`` should match your WS' public IP address and
``ws.example.com`` its domain name. Edit these if they are wrong.
Replace ``change-me@example.com`` with your email address.
If you installed the GROMACS tools you may also have a ``path`` entry. This
replaces the executable ``PATH`` environment variable for project commands
run by the WS.
Start the WS
------------
Once the global configuration is correct you can start the WS with:
::
sudo service fah-work start
Verify that it is really running by searching for it in the list of running
processes:
::
ps aux | grep fah-work
If the above command yields no response then the WS is not running.
Accessing the Log
-----------------
The WS writes a log file which details it's activities, error and warnings. You
can follow this log with:
::
sudo tail -n 1000 -F /var/log/fah-work/log.txt
Accessing the Web Interface
---------------------------
The WS web interface provides usefull information about the status of the WS
and the projects running on it. To access the web interface the IP address of
your computer must be listed in the WS' ``web-allow`` configuration option.
Add your IP address:
.. code-block:: xml
xxx.xxx.xxx.xxx
or add a CIDR_ IP address range eg:
.. code-block:: xml
123.456.798.0/24
The above would allow any IP address in the range 123.456.798.0 to
123.456.798.255.
Visit the web interface at https://ws.example.com/, where ``ws.example.com`` is
the domain name of your WS.
Accept WS certificate
---------------------
When the WS first starts up it will automatically contact the Assignment Servers
(AS) a thttps://assign1.foldingathome.org/ andhttps://assign2.foldingathome.org/
and submit a Certificate Signing Request (CSR) to the AS. You must login to
the AS and approve your WS to give it access to Folding\@home.
With your WS running logon to the AS by going to
https://assign1.foldingathome.org/ and clicking on the Folding\@home icon. Once
you are logged in find your WS in the list on the ``WS`` page and click the
``Accept`` button. After a few moments your WS should communicate with the WS
and acquire a certificate which allows it to participate in the Folding\@home
network.
You should be able to see the WS connect to the AS or discover any errors by
looking at the WS' log. You can also see the AS connection status on the
web interface.
Global CS
---------
All WS should have at least one Collection Server (CS). A CS provides a backup
in case your WS goes down but you must configure it ahead of time and keep it
running. A CS uses the same software but may have no projects. One CS may
serve many WS. A WS can act as both a WS and a CS at the same time.
In the default setup, WS are configured as "global CS" with this option:
.. code-block:: xml
Normally, this is all you need to do. Your WS will act as a CS to other WS
on Folding\@home home and vice versa.
For global CS the ``collect-allow`` option is not needed. WS are authenticated
using the Folding\@home certificate system and any WS in the network is allowed
to use any global CS.
Manual CS
---------
To manually configure a CS, install the WS software as normal. Then add the
following to its ``config.xml``:
.. code-block:: xml
xxx.xxx.xxx.xxx
Where ``xxx.xxx.xxx.xxx`` is the IP address or IP address ranges (in CIDR_
format) for the WS that are allowed to use the CS. Restart the CS to reload
the configuration.
Next, on each of the WS which will use the CS add the following to their
``config.xml``:
.. code-block:: xml
xxx.xxx.xxx.xxx
Where ``xxx.xxx.xxx.xxx`` is one or more IP addresses, IP address ranges or
hostnames. Restart the WS for the changes to take effect. Check the WS log
file for errors and the WS web interface to make sure it is successful talking
to the CS.
Project Configuration
---------------------
A WS may run one or more folding projects. These are added to the WS by
creating project configuration files named ``project.xml`` in subdirectories
under ``/var/lib/fah-work/projects``. This directory may be a symbolic link
to somewhere else on the file system. For example, you may have:
::
/var/lib/fah-work/projects/p12345/project.xml
There are currently two main types of simulation project which can be run on
Folding@home. These use either the GROMACS_ or OpenMM_ molecular dynamics
simulation software.
If you are using GROMACS_ you will need to install the appropriate tools.
See :ref:`GROMACS install `.
Acquire a Project Number
^^^^^^^^^^^^^^^^^^^^^^^^
Each project needs a unique project number. These numbers are tracked in a
document which can be found on the `project numbers`_ page on GitHub.
If you have not already been allocated a range please contact
projectmanager@foldingathome.org.
In the following sections we will refer to the project number as ``1234``
Configure the Project
^^^^^^^^^^^^^^^^^^^^^
In the project directory create a file called ``project.xml`` with the following
contents:
.. code-block:: xml
Use ``0xa8`` for GROMACS_ projects and ``0x22`` for OpenMM_. Enter your email
address for the contact.
Next set the number of ``runs`` and ``clones``. This tells the WS how many
jobs to create. For each run there will be a number of clones.
Generally, clones are identical but start with a different random seed whereas
runs start with different initial positions and velocities.
.. code-block:: xml
``gens`` determine the maximum number of Work Units or WUs in a trajectory.
A job trajectory is made up of multiple consecutive WUs or generations.
.. code-block:: xml
The number of atoms must be specified but is for display purposes only:
.. code-block:: xml
``credit`` is the base credit per WU.
.. code-block:: xml
``timeout`` determines the amount of time in days a client has to return the
WU and still be eligible to receive bonus points. A related option is
``timeout-factor`` which determines when the WS will give up on waiting for the
WU results and potentially reissue the WU. It is not normally necessary to
set the ``timeout-factor``. The default should be sufficient.
``deadline`` should be longer than ``timeout`` and is the maximum number of days
in which a WU can be returned for credit.
.. code-block:: xml
``send`` is a list of file glob patterns which tell the WS which files to send
to the client with a WU. ``return`` similarly tells the core which files to
send back from the client. Variables such as ``$gen`` may be used in file
names. See the :doc:`help` system for which variables are defined where and
when.
.. code-block:: xml
frame$gen.tpr
frame*.trr md.log
``core-args`` can be used to pass additional arguments to the core. These
arguments are core dependent. In the case of Gromacs, ``core-args`` are
arguments you would normally pass to ``mdrun`` on the command line.
.. code-block:: xml
-s frame$gen.tpr
-o frame$gen.trr
``create-command`` and ``next-gen-command`` are shell commands run by the WS
to create the first and next generation WUs. See the WS online :doc:`help` for
more information about the variables defined for these commands.
.. code-block:: xml
gmx grompp \
-f $home/mdp/nav.mdp \
-c $home/runs/$run.gro \
-p $home/runs/$run.top \
-o $jobdir/frame0.tpr \
-po $jobdir/mdout.mdp \
-maxwarn 2
gmx convert-tpr \
-s $jobdir/frame0.tpr \
-f $results/frame$prev-gen.trr \
-o $jobdir/frame$gen.tpr \
-extend $(mul $gen 160)
Other commands include ``cleanup-command`` and ``test-command``. Which may
be used to cleanup old files and run further tests on WUs. The following
can be used to automatically delete old ``.tpr`` files:
.. code-block:: xml
$(if $(neq $prev-gen 0) "rm -f $jobdir/frame$prev-gen.tpr")
The above command removes the previous ``frame###.tpr`` if it is not
``frame0.tpr``.
Configuration Variables
~~~~~~~~~~~~~~~~~~~~~~~
Many configuration options reference variables or call configuration
functions. The variables defined for an option vary. Consult the WS :doc:`help`
system to determine which variables are defined where and when.
Variables may be referred to in one of two ways:
::
$name
or
::
$(name)
Both are equivalent but the later form is useful for separating variable names
from content. Note that variable names always start with a letter but may
contain numbers, dashes or underscores. Variable and function names are
generally all lower case but they are case sensitive.
Configuration Functions
~~~~~~~~~~~~~~~~~~~~~~~
Functions are called in much the same way but functions have arguments which are
separated by white-space. Arguments containing white-space may be escaped
using double or single quotes or parenthesis.
Sometimes it is desirable to compute a value from other variables in an option.
This may be achieved using math functions like ``mul``, ``div``, ``sub``,
``add``. For example:
::
$(mul 4 $(add 3 3))
The above would compute the value 24.
The conditional function ``if`` takes 2 or 3 arguments. If the first argument
evaluates as true then the result is the value of the second
argument otherwise the result is the value of the third argument or empty.
Note that any non-empty string other than ``false`` is considered
true. For example:
::
$(if $(eq $run 0) first_run.xml other_runs.xml)
The above would result in ``first_run.xml`` when ``$run`` is ``0`` otherwise
``other_runs.xml``.
If the dollar sign is needed in an option it can be escaped with ``\$``.
Likewise, the backslash is escaped with ``\\``.
Example Gromacs Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is for 0xa8 and newer GROMACS_. Using GROMACS core 0xa7 is not recommended
for new projects. Note that 0xa8 used Gromacs 2020.
.. code-block:: xml
frame$gen.tpr
frame*.trr md.log
-s frame$gen.tpr
-o frame$gen.trr
gmx grompp \
-f $home/mdp/nav.mdp \
-c $home/runs/$run.gro \
-p $home/runs/$run.top \
-o $jobdir/frame0.tpr \
-po $jobdir/mdout.mdp \
-maxwarn 2
gmx convert-tpr \
-s $jobdir/frame0.tpr \
-f $results/frame$prev-gen.trr \
-o $jobdir/frame$gen.tpr \
-extend $(mul $gen 160)
$(if $(neq $prev-gen 0) "rm -f $jobdir/frame$prev-gen.tpr")
Example OpenMM Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: xml
$home/core.xml
$home/RUN$run/system.xml
$home/RUN$run/integrator.xml
state.xml
ln -s -f $home/RUN$run/state$clone.xml $jobdir/state.xml
mv -f $results/checkpointState.xml $jobdir/state.xml
Project description
^^^^^^^^^^^^^^^^^^^
Each project has a public description which is shown to F@H users. This
description should be as non-technical as possible and should give the user some
idea of the goals of your simulation project. To edit your project's
description navigate to https://apps.foldingathome.org/projects/
You will need to login using credentials acquired above.
Benchmark the Project
^^^^^^^^^^^^^^^^^^^^^
In order to set a fair amount of points for your project's WUs you need to
benchmark. Benchmark your project against a comparable project already running
on FAH. For example:
- Run a general FAH WU (Work Unit)
- Note the PPD (Points Per Day)of that project
- Run your project
- Configure base-credit, timeout, and deadline to match your project's PPD to
that of the production WU.
A typical GPU project runs for about half a day and will give around 175k points
per day (PPD) on the Nvidia GTX 780 and 55k PPD on the AMD Radeon HD 7970.
The PPD formula is:
::
PPD = [Base Credit] / 1000 * sqrt(0.75 * [Deadline] / [Average TTP]^3)
where:
- ``Base Credit`` is the minimum points awarded for completing the WU
- ``Deadline`` is the time in days before the WU expires
- ``Average TPP``, or time-per-percent, is the mean amount of time in
days needed to complete 1% of a WU.
Please note that this is a slight overestimate of PPD, and that various factors
including file download times, etc. may limit actual WU performance.
Setup Internal Testing
^^^^^^^^^^^^^^^^^^^^^^
Next logon to the AS and configure the project's assign constraints and weight.
Use the AS' online documentation by clicking on the book icons.
Initially, you should assign a project key to your project. A project key is a
31-bit random integer. You can run ``gen-project-key`` to generate a valid
random project key.
Note, the project key must not start with zero as this will make the AS think
it is in octal format. Make sure you set a non-zero weight for the WS as well as
the project. Then double check your setup by viewing the project's
*Full Constraints* on the AS.
Set the Project Cause
^^^^^^^^^^^^^^^^^^^^^
Decide which of the cause categories, if any, your project falls in to then add
a constraint to your project on the AS. For example:
::
Cause=Cancer
This will *not* restrict your project to only users who specify a matching cause
but it will give priority to your project when assigning to users who have
specified a matching preference. If your project does not fit in to one of the
available causes, do not add this constraint. For the current list of causes see
the **Constraint Variables** section in the AS online help.
Note that the *Disease* category in the project description is for display
purposes only and is independent of the cause constraint.
Self Test the Project
^^^^^^^^^^^^^^^^^^^^^
Next test your project using the F\@H client software. Specify the project key
you choose above. Make sure you have either a CPU (GROMACS) or GPU (OpenMM) slot
configured on your client. Then you run the client as follows:
::
./FAHClient --project-key=########
You can find more information about running the F\@H client at
https://foldingathome.org/
Make sure you can download a WU, upload the results, create the next generation
and then download and upload the next generation WU to make sure all of the
project commands are working correctly. Check the WS log file and web interface
for errors and to monitor progress.
Announce the project on Slack
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Once you've tested your project yourself, create a slack channel for your
project series. E.g. if your projects are in the range 1234xx then create the
channel ``#123400``. Announce your project on slack.com and tell users how
to get the WUs by sharing the project key in the current channel topic. This
will allow many more users to test out your new project. Work with the users to
resolve any problems.
Project Beta Testing
^^^^^^^^^^^^^^^^^^^^
After successful internal testing, move your project to beta release. This is
achieved by relaxing the project key constraint and adding the beta client
constraint. Like this:
::
(ProjectKey=12334 ProjectKey=0) Client=BETA
This allows clients with your project key to continue to get WUs but also opens
it up to beta testers with out a project key.
Public Release
^^^^^^^^^^^^^^
After successful beta testing normal procedure is to move your project to
full public release. This is achieved by removing the client constraint.
Your constraints should look like this:
::
(ProjectKey=12334 ProjectKey=0)
No ``Client`` constraint is needed.
Example ``config.xml``
----------------------
A complete ``config.xml`` might look something like this:
.. code-block:: xml
zzz.zzz.zzz.0/24
.. _CIDR: https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing
.. _GROMACS: https://www.gromacs.org/
.. _OpenMM: https://openmm.org/
.. _project numbers:
https://github.com/FoldingAtHome/fah-docs/wiki/Project-numbers