Configuration

This documention provides instructions on how to configure a Folding@home Work Server and the projects on it.

Global Configuration

During install a default configuration file will be created at /etc/fah-work/config.xml. The contents should look something like this:

<config>
  <contact v="change-me@example.com"/>
  <server-ip v="0.0.0.0"/>
  <reported-ip v="xxx.xxx.xxx.xxx"/>
  <domains>ws.example.com</domains>
  <cs/> <!-- Act as a global CS -->
</config>

xxx.xxx.xxx.xxx should match your WS’ public IP address and ws.example.com its domain name. Edit these if they are wrong.

Replace change-me@example.com with your email address.

If you installed the GROMACS tools you may also have a path entry. This replaces the executable PATH environment variable for project commands run by the WS.

Start the WS

Once the global configuration is correct you can start the WS with:

sudo service fah-work start

Verify that it is really running by searching for it in the list of running processes:

ps aux | grep fah-work

If the above command yields no response then the WS is not running.

Accessing the Log

The WS writes a log file which details it’s activities, error and warnings. You can follow this log with:

sudo tail -n 1000 -F /var/log/fah-work/log.txt

Accessing the Web Interface

The WS web interface provides usefull information about the status of the WS and the projects running on it. To access the web interface the IP address of your computer must be listed in the WS’ web-allow configuration option.

Add your IP address:

<web-allow>xxx.xxx.xxx.xxx</web-allow>

or add a CIDR IP address range eg:

<web-allow>123.456.798.0/24</web-allow>

The above would allow any IP address in the range 123.456.798.0 to 123.456.798.255.

Visit the web interface at https://ws.example.com/, where ws.example.com is the domain name of your WS.

Accept WS certificate

When the WS first starts up it will automatically contact the Assignment Servers (AS) a thttps://assign1.foldingathome.org/ andhttps://assign2.foldingathome.org/ and submit a Certificate Signing Request (CSR) to the AS. You must login to the AS and approve your WS to give it access to Folding@home.

With your WS running logon to the AS by going to https://assign1.foldingathome.org/ and clicking on the Folding@home icon. Once you are logged in find your WS in the list on the WS page and click the Accept button. After a few moments your WS should communicate with the WS and acquire a certificate which allows it to participate in the Folding@home network.

You should be able to see the WS connect to the AS or discover any errors by looking at the WS’ log. You can also see the AS connection status on the web interface.

Global CS

All WS should have at least one Collection Server (CS). A CS provides a backup in case your WS goes down but you must configure it ahead of time and keep it running. A CS uses the same software but may have no projects. One CS may serve many WS. A WS can act as both a WS and a CS at the same time.

In the default setup, WS are configured as “global CS” with this option:

<cs/>

Normally, this is all you need to do. Your WS will act as a CS to other WS on Folding@home home and vice versa.

For global CS the collect-allow option is not needed. WS are authenticated using the Folding@home certificate system and any WS in the network is allowed to use any global CS.

Manual CS

To manually configure a CS, install the WS software as normal. Then add the following to its config.xml:

<collect/>
<collect-allow>
  xxx.xxx.xxx.xxx
</collect-allow>

Where xxx.xxx.xxx.xxx is the IP address or IP address ranges (in CIDR format) for the WS that are allowed to use the CS. Restart the CS to reload the configuration.

Next, on each of the WS which will use the CS add the following to their config.xml:

<collection-servers>
  xxx.xxx.xxx.xxx
</collection-servers>

Where xxx.xxx.xxx.xxx is one or more IP addresses, IP address ranges or hostnames. Restart the WS for the changes to take effect. Check the WS log file for errors and the WS web interface to make sure it is successful talking to the CS.

Project Configuration

A WS may run one or more folding projects. These are added to the WS by creating project configuration files named project.xml in subdirectories under /var/lib/fah-work/projects. This directory may be a symbolic link to somewhere else on the file system. For example, you may have:

/var/lib/fah-work/projects/p12345/project.xml

There are currently two main types of simulation project which can be run on Folding@home. These use either the GROMACS or OpenMM molecular dynamics simulation software.

If you are using GROMACS you will need to install the appropriate tools. See GROMACS install.

Acquire a Project Number

Each project needs a unique project number. These numbers are tracked in a document which can be found on the project numbers page on GitHub. If you have not already been allocated a range please contact projectmanager@foldingathome.org.

In the following sections we will refer to the project number as 1234

Configure the Project

In the project directory create a file called project.xml with the following contents:

<project type="0xa8" id="1234">
  <contact v="john.doe@example.com"/>
</project>

Use 0xa8 for GROMACS projects and 0x22 for OpenMM. Enter your email address for the contact.

Next set the number of runs and clones. This tells the WS how many jobs to create. For each run there will be a number of clones. Generally, clones are identical but start with a different random seed whereas runs start with different initial positions and velocities.

<runs v="930"/>
<clones v="15"/>

gens determine the maximum number of Work Units or WUs in a trajectory. A job trajectory is made up of multiple consecutive WUs or generations.

<gens v="50000"/>

The number of atoms must be specified but is for display purposes only:

<atoms v="33805"/>

credit is the base credit per WU.

<credit v="1000"/>

timeout determines the amount of time in days a client has to return the WU and still be eligible to receive bonus points. A related option is timeout-factor which determines when the WS will give up on waiting for the WU results and potentially reissue the WU. It is not normally necessary to set the timeout-factor. The default should be sufficient.

deadline should be longer than timeout and is the maximum number of days in which a WU can be returned for credit.

<timeout v="3.84"/>
<deadline v="8.32"/>

send is a list of file glob patterns which tell the WS which files to send to the client with a WU. return similarly tells the core which files to send back from the client. Variables such as $gen may be used in file names. See the Help System system for which variables are defined where and when.

<send>frame$gen.tpr</send>
<return>frame*.trr md.log</return>

core-args can be used to pass additional arguments to the core. These arguments are core dependent. In the case of Gromacs, core-args are arguments you would normally pass to mdrun on the command line.

<core-args>
  -s frame$gen.tpr
  -o frame$gen.trr
</core-args>

create-command and next-gen-command are shell commands run by the WS to create the first and next generation WUs. See the WS online Help System for more information about the variables defined for these commands.

<create-command>
  gmx grompp \
    -f $home/mdp/nav.mdp \
    -c $home/runs/$run.gro \
    -p $home/runs/$run.top \
    -o $jobdir/frame0.tpr \
    -po $jobdir/mdout.mdp \
    -maxwarn 2
</create-command>

<next-gen-command>
  gmx convert-tpr \
    -s $jobdir/frame0.tpr \
    -f $results/frame$prev-gen.trr \
    -o $jobdir/frame$gen.tpr \
    -extend $(mul $gen 160)
</next-gen-command>

Other commands include cleanup-command and test-command. Which may be used to cleanup old files and run further tests on WUs. The following can be used to automatically delete old .tpr files:

<cleanup-command>
  $(if $(neq $prev-gen 0) "rm -f $jobdir/frame$prev-gen.tpr")
</cleanup-command>

The above command removes the previous frame###.tpr if it is not frame0.tpr.

Configuration Variables

Many configuration options reference variables or call configuration functions. The variables defined for an option vary. Consult the WS Help System system to determine which variables are defined where and when.

Variables may be referred to in one of two ways:

$name

or

$(name)

Both are equivalent but the later form is useful for separating variable names from content. Note that variable names always start with a letter but may contain numbers, dashes or underscores. Variable and function names are generally all lower case but they are case sensitive.

Configuration Functions

Functions are called in much the same way but functions have arguments which are separated by white-space. Arguments containing white-space may be escaped using double or single quotes or parenthesis.

Sometimes it is desirable to compute a value from other variables in an option. This may be achieved using math functions like mul, div, sub, add. For example:

$(mul 4 $(add 3 3))

The above would compute the value 24.

The conditional function if takes 2 or 3 arguments. If the first argument evaluates as true then the result is the value of the second argument otherwise the result is the value of the third argument or empty. Note that any non-empty string other than false is considered true. For example:

$(if $(eq $run 0) first_run.xml other_runs.xml)

The above would result in first_run.xml when $run is 0 otherwise other_runs.xml.

If the dollar sign is needed in an option it can be escaped with \$. Likewise, the backslash is escaped with \\.

Example Gromacs Configuration

This is for 0xa8 and newer GROMACS. Using GROMACS core 0xa7 is not recommended for new projects. Note that 0xa8 used Gromacs 2020.

<project type="0xa8" id="1234">
  <contact v="john.doe@example.com"/>

  <runs v="930"/>
  <clones v="15"/>
  <gens v="50000"/>
  <atoms v="33805"/>

  <credit v="1000"/>
  <timeout v="3.84"/>
  <deadline v="8.32"/>

  <send>frame$gen.tpr</send>
  <return>frame*.trr md.log</return>

  <core-args>
    -s frame$gen.tpr
    -o frame$gen.trr
  </core-args>

  <create-command>
    gmx grompp \
      -f $home/mdp/nav.mdp \
      -c $home/runs/$run.gro \
      -p $home/runs/$run.top \
      -o $jobdir/frame0.tpr \
      -po $jobdir/mdout.mdp \
      -maxwarn 2
  </create-command>

  <next-gen-command>
    gmx convert-tpr \
      -s $jobdir/frame0.tpr \
      -f $results/frame$prev-gen.trr \
      -o $jobdir/frame$gen.tpr \
      -extend $(mul $gen 160)
  </next-gen-command>

  <cleanup-command>
    $(if $(neq $prev-gen 0) "rm -f $jobdir/frame$prev-gen.tpr")
  </cleanup-command>
</project>

Example OpenMM Configuration

<project type="0x22" id="1234">
  <contact v="john.doe@example.com"/>

  <runs v="8"/>
  <clones v="100"/>
  <gens v="100"/>
  <atoms v="117196"/>

  <credit v="2700"/>
  <timeout v="4"/>
  <deadline v="5"/>

  <send>
    $home/core.xml
    $home/RUN$run/system.xml
    $home/RUN$run/integrator.xml
    state.xml
  </send>

  <create-command>
    ln -s -f $home/RUN$run/state$clone.xml $jobdir/state.xml
  </create-command>

  <next-gen-command>
    mv -f $results/checkpointState.xml $jobdir/state.xml
  </next-gen-command>
</project>

Project description

Each project has a public description which is shown to F@H users. This description should be as non-technical as possible and should give the user some idea of the goals of your simulation project. To edit your project’s description navigate to https://apps.foldingathome.org/projects/ You will need to login using credentials acquired above.

Benchmark the Project

In order to set a fair amount of points for your project’s WUs you need to benchmark. Benchmark your project against a comparable project already running on FAH. For example:

• Run a general FAH WU (Work Unit)

• Note the PPD (Points Per Day)of that project

• Run your project

• Configure base-credit, timeout, and deadline to match your project’s PPD to that of the production WU.

A typical GPU project runs for about half a day and will give around 175k points per day (PPD) on the Nvidia GTX 780 and 55k PPD on the AMD Radeon HD 7970.

The PPD formula is:

PPD = [Base Credit] / 1000 * sqrt(0.75 * [Deadline] / [Average TTP]^3)

where:

• Base Credit is the minimum points awarded for completing the WU

• Deadline is the time in days before the WU expires

• Average TPP, or time-per-percent, is the mean amount of time in days needed to complete 1% of a WU.

Please note that this is a slight overestimate of PPD, and that various factors including file download times, etc. may limit actual WU performance.

Setup Internal Testing

Next logon to the AS and configure the project’s assign constraints and weight. Use the AS’ online documentation by clicking on the book icons. Initially, you should assign a project key to your project. A project key is a 31-bit random integer. You can run gen-project-key to generate a valid random project key.

Note, the project key must not start with zero as this will make the AS think it is in octal format. Make sure you set a non-zero weight for the WS as well as the project. Then double check your setup by viewing the project’s Full Constraints on the AS.

Set the Project Cause

Decide which of the cause categories, if any, your project falls in to then add a constraint to your project on the AS. For example:

Cause=Cancer

This will not restrict your project to only users who specify a matching cause but it will give priority to your project when assigning to users who have specified a matching preference. If your project does not fit in to one of the available causes, do not add this constraint. For the current list of causes see the Constraint Variables section in the AS online help.

Note that the Disease category in the project description is for display purposes only and is independent of the cause constraint.

Self Test the Project

Next test your project using the F@H client software. Specify the project key you choose above. Make sure you have either a CPU (GROMACS) or GPU (OpenMM) slot configured on your client. Then you run the client as follows:

./FAHClient --project-key=########

You can find more information about running the F@H client at https://foldingathome.org/

Make sure you can download a WU, upload the results, create the next generation and then download and upload the next generation WU to make sure all of the project commands are working correctly. Check the WS log file and web interface for errors and to monitor progress.

Announce the project on Slack

Once you’ve tested your project yourself, create a slack channel for your project series. E.g. if your projects are in the range 1234xx then create the channel #123400. Announce your project on slack.com and tell users how to get the WUs by sharing the project key in the current channel topic. This will allow many more users to test out your new project. Work with the users to resolve any problems.

Project Beta Testing

After successful internal testing, move your project to beta release. This is achieved by relaxing the project key constraint and adding the beta client constraint. Like this:

(ProjectKey=12334 ProjectKey=0) Client=BETA

This allows clients with your project key to continue to get WUs but also opens it up to beta testers with out a project key.

Public Release

After successful beta testing normal procedure is to move your project to full public release. This is achieved by removing the client constraint. Your constraints should look like this:

(ProjectKey=12334 ProjectKey=0)

No Client constraint is needed.

Example config.xml

A complete config.xml might look something like this:

<config>
  <contact v="me@example.com"/>
  <server-ip v="xxx.xxx.xxx.xxx"/>
  <collection-servers v="yyy.yyy.yyy.yyy"/>
  <web-allow>zzz.zzz.zzz.0/24</web-allow>
</config>