Batch running of multiple optimizations in parallel

In the previous tutorials we fit homogeneous and heterogeneous models to the pooled empirical functional data of a single group.

However, we often want to perform multiple optimizations, especially if we are fitting subject-specific (individualized) BNMs. Furthermore, even when fitting the model to data of a single individual or group, we should perform multiple optimization runs using different random seeds to reduce the chance of convergence to local optima.

At the same time, the CMA-ES runs we performed earlier used a relatively small number of simulations in parallel (128 particles). Depending on the GPU hardware, this may underutilize the available GPU resources.

To address this, the cuBNM toolbox includes functionality for batch-running multiple optimizations in parallel: across subjects, across runs for the same subject, or both. This helps fully leverage the GPU and improves efficiency compared to running the optimizations independently and serially.

In this tutorial, we’ll use this batch optimization functionality to run CMA-ES optimizations of node-based heterogeneous models (based on canonical resting state networks) in parallel for two subjects from the HCP-YA dataset, performing two optimizations (with different sampling seeds) per subject (i.e., a total of four parallel optimization runs).

Loading the data

We will use the data of the first two subjects from the HCP-YA dataset. The data of these two subjects (and only these two) is included in the toolbox and can be loaded as follows:

from cubnm import datasets, utils

subjects = ['100206', '100307']

scs = {}
emp_bolds = {}

for sub in subjects:
    scs[sub] = datasets.load_sc('strength', 'schaefer-100', sub)
    emp_bolds[sub] = datasets.load_bold('schaefer-100', sub, ses='REST1_LR')

Let’s first visualize the SC matrices and BOLD signals:

from cubnm import datasets
import seaborn as sns
import matplotlib.pyplot as plt
import scipy.stats


fig, axes = plt.subplots(1, 2, figsize=(8, 4))
for i_sub, sub in enumerate(subjects):
    sns.heatmap(scs[sub], cmap='Blues', cbar=False, ax=axes[i_sub])
    axes[i_sub].set_title(f"SC sub-{sub}")

fig, axes = plt.subplots(1, 2, figsize=(15, 2))
for i_sub, sub in enumerate(subjects):
    ax = axes[i_sub]
    # plot z-scored BOLD signal
    sns.heatmap(scipy.stats.zscore(emp_bolds[sub], axis=1), cbar=False, ax=ax)
    ax.set_ylabel("Nodes")
    ax.set_xlabel("Time")
    ax.set_xticks([])
    ax.set_yticks([])
    ax.set_title(f"BOLD sub-{sub}")

Running batch optimization

Now we’ll create separate BNMProblem objects for each subject. These will define heterogeneous rWW models with FIC, similar to the model used in the node-based heterogeneous CMA-ES tutorial. Note that each subject has a different sc, emp_bold, and out_dir, so we pass sc directly to the BNMProblem instead of including it in sim_options dictionary.

import os
from cubnm import optimize

# define simulation options
sim_options = dict(
    duration=900,
    bold_remove_s=30,
    TR=0.72,
    sc_dist=None,
    dt='0.1',
    bw_dt='1.0',
    states_ts=False,
    states_sampling=None,
    noise_out=False,
    sim_seed=0,
    noise_segment_length=30,
    gof_terms=['+fc_corr', '-fcd_ks'],
    do_fc=True,
    do_fcd=True,
    window_size=30,
    window_step=5,
    fcd_drop_edges=True,
    exc_interhemispheric=True,
    bw_params='heinzle2016-3T',
    sim_verbose=False,
    do_fic=True,
    max_fic_trials=0,
    fic_penalty_scale=0.5,
)

# load the Yeo networks map
yeo_map = datasets.load_maps('yeo7', parc='schaefer-100').astype(int)[0]

# define a map-based heterogeneous problem for each subject
problems = {}
for sub in subjects:
    problems[sub] = optimize.BNMProblem(
        model = 'rWW',
        params = {
            'G': (0.001, 10.0),
            'w_p': (0, 2.0),
            'J_N': (0.001, 0.5)
        },
        emp_bold = emp_bolds[sub], # subject-specific bold signal
        sc = scs[sub], # subject-specific SC
        het_params = ['w_p', 'J_N'],
        node_grouping = yeo_map,
        out_dir = os.path.join('./batch_cmaes', sub), # subject-specific output directories
        **sim_options
    )

problems

{'100206': <cubnm.optimize.BNMProblem at 0x1493f6bfef80>,
 '100307': <cubnm.optimize.BNMProblem at 0x1493f67fb6d0>}

Next, we define two CMA-ES optimizers with different seeds. These will be reused across subjects, and subject-specific copies of the optimizers will be created internally.

optimizers = {}
for seed in [1, 2]:
    optimizers[seed] = optimize.CMAESOptimizer(
        popsize=128,
        n_iter=120,
        seed=seed,
        algorithm_kws=dict(tolfun=5e-3),
        print_history=False
    )

optimizers

{1: <cubnm.optimize.CMAESOptimizer at 0x1493f67fbaf0>,
 2: <cubnm.optimize.CMAESOptimizer at 0x1493f67fba30>}

Note that unlike single-run optimizations, we do not need to call setup_problem() as it is handled internally during batch optimization.

We can now run all four optimization runs (2 subjects × 2 seeds) in parallel using the optimize.batch_optimize() function. This function takes lists of problems and optimizers corresponding to each run. After running the optimizations, it returns fitted copies of the provided Optimizer (e.g. CMAESOptimizer) instances.

Technical Note: Internally, batch_optimize() creates a MultiSimGroup by merging the SimGroup objects of all provided BNMProblems. This allows the simulations to be run in parallel on GPU. After the optimization is complete, it re-runs the optimal simulations (by default, when save=True) and saves all results to disk, just like in individual runs.

%%time
# define the corresponding lists of problems and optimizers for all 4 runs
batch_problems   = [problems['100206'], problems['100206'], problems['100307'], problems['100307']]
batch_optimizers = [optimizers[1],      optimizers[2],      optimizers[1],      optimizers[2]]

# run batch optimization
fitted_optimizers = optimize.batch_optimize(batch_optimizers, batch_problems)

WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Initializing GPU session...
took 3.196610 s
Running 512 simulations...
took 104.247814 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.427767 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.434260 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.806808 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.961200 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.429795 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.196031 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.423851 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.470495 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.853771 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 105.861008 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.401897 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.692648 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.615236 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.707995 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.546199 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.823764 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.022061 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.489138 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.879365 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.452240 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.006391 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.719477 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.477268 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.717529 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.893181 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.804019 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.098787 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.753617 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.428441 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.519028 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.771769 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.485212 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.393589 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.676835 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.739667 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.929206 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.715193 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.563624 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.917694 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.855024 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.801503 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.928029 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.692434 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.617595 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.491056 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.491522 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.118466 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.557321 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.682339 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.817942 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.804082 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.070424 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.692349 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.833109 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.415471 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.713832 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.494252 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.782064 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 105.045267 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.687975 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.043442 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.408423 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.505676 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.464610 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.845578 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.206459 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.361718 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.033232 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.936801 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.657149 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.500300 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.980349 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.936565 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.886983 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.171692 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 105.077429 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.659070 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.343379 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.772933 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.981197 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.705898 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.875170 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.651267 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.340327 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.630459 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.819425 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.359943 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.335158 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.714790 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.428742 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.944278 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.479762 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.095374 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.881291 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.939473 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 105.009510 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.686908 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.483659 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.478323 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.830899 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.474134 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.596413 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.383148 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.854548 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.673229 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.232972 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.884908 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.831166 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 103.197690 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.514613 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.988629 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.960211 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.883751 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 105.048376 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.829343 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.596789 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.672542 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.267304 s
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Current session is already initialized
Running 512 simulations...
took 104.807795 s
Rerunning and saving the optimal simulation(s)
WARNING:cubnm.sim.base:Progress cannot be shown in real time when using Jupyter, but simulations are running in background. Please wait...
Initializing GPU session...
took 2.929051 s
Running 4 simulations...
took 29.417482 s
CPU times: user 3h 41min 44s, sys: 1min 24s, total: 3h 43min 9s
Wall time: 3h 53min 48s

We can observe that, in each iteration (before any optimizer converges), a total of 128 × 4 = 512 simulations are run in parallel. On an NVIDIA A100 (40 GB) GPU, this takes approximately 125 seconds per generation. For comparison, in the single-run CMA-ES tutorial with 128 particles, running 128 simulations per generation took around 40 seconds on average. So, by running all 4 optimizations as a batch (taking around 125 seconds per generation total), instead of running them individually (which would take 40 × 4 = 160 seconds per generation), we have made the process approximately ~20% more efficient.

Efficiency gains of batch optimization will depend on the GPU hardware, but it typically becomes more efficient as more optimizations are included, up to the hardware’s parallelization limits.

Visualization of the results

Just like the individually-run optimizers from previous tutorials, the batch-fitted optimizers provide:

• optimizer.history: The optimization history (free parameters and scores per particle).

• optimizer.opt: The set of free parameters with the lowest cost in addition to its associated scores.

• optimizer.problem.sim_group: A SimGroup (N = 1) object which contains the simulation data (BOLD, FC/D, states (time series), regional parameters) of the optimal simulation.

We will now look at the results of these optimization runs. To make this easier, we first convert the list of returned optimizers into a nested dictionary indexed by subject and seed:

fitted_optimizers_dict = {}
i_run = 0
for sub in subjects:
    fitted_optimizers_dict[sub] = {}
    for seed in [1, 2]:
        fitted_optimizers_dict[sub][seed] = fitted_optimizers[i_run]
        i_run += 1

Negative cost trajectories

We can visualize how negative cost evolved over the course of the 4 CMA-ES optimization runs. Here, to indicate better simulations with higher values, rather than the cost function we will visualize negative cost function.

fig, axes = plt.subplots(2, 2, figsize=(12, 8))

for i_sub, sub in enumerate(subjects):
    for i_seed, seed in enumerate([1, 2]):
        ax = axes[i_sub, i_seed]
        fitted_optimizers_dict[sub][seed].plot_history('-cost', ax=ax)
        ax.set_title(f'sub-{sub} (seed = {seed})')

fig.tight_layout(pad=0.5)

Each subplot shows one optimization run. We can use these plots to evaluate convergence behavior across seeds and subjects. As we can see in all runs the optimization is close to convergence, but not yet converged.

Goodness-of-fit in the best runs across subjects

For each subject, we select the best optimization run, i.e., the run with the lowest cost across the two CMA-ES runs of that subject:

import numpy as np

best_optimizers = {}

for sub in subjects:
    best_cost = np.inf
    for seed in [1, 2]:
        if fitted_optimizers_dict[sub][seed].opt['cost'] < best_cost:
            best_cost = fitted_optimizers_dict[sub][seed].opt['cost']
            best_seed = seed
            best_optimizers[sub] = fitted_optimizers_dict[sub][seed]
    print(f'{sub} best run has seed = {best_seed}')
    print(fitted_optimizers_dict[sub][best_seed].opt)

100206 best run has seed = 2
index            87.000000
G                 1.803556
w_p0              0.004612
w_p1              1.010187
w_p2              1.215477
w_p3              0.417861
w_p4              0.396428
w_p5              0.000006
w_p6              1.891427
J_N0              0.087728
J_N1              0.220735
J_N2              0.262890
J_N3              0.473064
J_N4              0.005650
J_N5              0.090618
J_N6              0.020657
cost             -0.333063
+fc_corr          0.517663
-fcd_ks          -0.166325
+gof              0.351338
-fic_penalty     -0.018275
gen             119.000000
Name: 15191, dtype: float64
100307 best run has seed = 2
index            38.000000
G                 1.510128
w_p0              0.024995
w_p1              1.101752
w_p2              0.246972
w_p3              0.078933
w_p4              0.743568
w_p5              1.362386
w_p6              0.000649
J_N0              0.243930
J_N1              0.295312
J_N2              0.499666
J_N3              0.411785
J_N4              0.001317
J_N5              0.096313
J_N6              0.051360
cost             -0.235814
+fc_corr          0.364477
-fcd_ks          -0.115171
+gof              0.249306
-fic_penalty     -0.013492
gen             119.000000
Name: 15142, dtype: float64

For the best run of each subject, we can then visualize:

• The regression plot comparing simulated and empirical functional connectivity (FC)

• The distribution comparison between simulated and empirical functional connectivity dynamics (FCD)

fig, axes = plt.subplots(2, 2, figsize=(9, 6))
axes = axes.flatten()
for i_sub, sub in enumerate(subjects):
    # regression plot
    ax = axes[i_sub*2]
    sns.regplot(
        x=best_optimizers[sub].problem.sim_group.sim_fc_trils[0],
        y=best_optimizers[sub].problem.emp_fc_tril,
        scatter_kws=dict(s=5, alpha=0.2, color="grey"),
        line_kws=dict(color="red"),
        ax=ax
    )
    ax.set_xlabel("Simulated FC")
    ax.set_ylabel("Empirical FC")
    # add fc_corr statistic
    ax.set_title(f'{sub} (r = {best_optimizers[sub].opt["+fc_corr"]:.3f})')
    # histograms
    ax = axes[i_sub*2+1]
    sns.histplot(best_optimizers[sub].problem.sim_group.sim_fcd_trils[0], element='step', alpha=0.5, label='Simulated', stat='density', ax=ax)
    sns.histplot(best_optimizers[sub].problem.emp_fcd_tril, element='step', alpha=0.5, label='Empirical', stat='density', ax=ax)
    ax.legend()
    # add fcd_ks statistic
    # -fcd_ks in .opt needs to be negated to have positive values
    ax.set_title(f'{sub} (KS = {-best_optimizers[sub].opt["-fcd_ks"]:.3f})')
fig.tight_layout()

Parameters and state variables of optimal simulations

We now visualize the global coupling parameter \(G\) and the regional values of \(w^{p}\) and \(J^{N}\) for the optimal simulations:

# create the axes
fig, axd = plt.subplot_mosaic(
    [["G"] + ["space0"] + ["w_p"] * 100 + ["space1"] + ["J_N"] * 100],
    gridspec_kw=dict(width_ratios=[10] + [1] * 202),
    figsize=(12, 1.5)
)
# plot heatmaps of parameters
sns.heatmap(
    np.array([o.problem.sim_group.param_lists['G'][0] for o in best_optimizers.values()])[:, None],
    cbar=False, cmap='viridis', ax=axd['G']
)
axd['G'].set_title(r'$G$')
sns.heatmap(
    np.array([o.problem.sim_group.param_lists['w_p'][0] for o in best_optimizers.values()]),
    cbar=False, cmap='viridis', ax=axd['w_p']
)
axd['w_p'].set_title(r'$w^{p}$')
sns.heatmap(
    np.array([o.problem.sim_group.param_lists['J_N'][0] for o in best_optimizers.values()]),
    cbar=False, cmap='viridis', ax=axd['J_N']
)
axd['J_N'].set_title(r'$J^{N}$')
# aesthetics and labeling
for ax in list(axd.values())[1:]:
    ax.axis('off')
axd['G'].set_xticks([])
labels = []
for sub in subjects:
    labels.append(f'sub-{sub}')
axd['G'].set_yticklabels(labels, rotation=0)

[Text(0, 0.5, 'sub-100206'), Text(0, 1.5, 'sub-100307')]

Similarly, we can also visualize node-wise time-averaged values of a model state, such as the input to excitatory neuronal ensembles \(I^E\), for each optimal simulation.

# take time-averaged node-wise I_E in optimal simulations
I_E_avgs = np.array([
    o.problem.sim_group.sim_states['I_E'][0, :]
    for o in best_optimizers.values()
])
# plot
fig, ax = plt.subplots(figsize=(7, 1.5))
sns.heatmap(I_E_avgs, cmap='viridis', ax=ax)
ax.set_title(r'$\langle I^E \rangle$')
labels = []
for sub in subjects:
    labels.append(f'sub-{sub}')
ax.set_yticklabels(labels, rotation=0)
ax.set_xticks([]);

Limitations of batch optimization

Currently, batch optimization supports varying only the following attributes across the batched BNMProblem instances:

• sc: Structural connectivity strength matrix

• emp_bold: Empirical BOLD signal

• out_dir: Output directory

Note: When multiple optimizers are run on the same problem, their outputs are saved in the same directory and automatically numbered, e.g., out_dir/cmaes_run-0, out_dir/cmaes_run-1, etc.

All other model and simulation parameters must be identical across batched problems. This includes, for example:

• sc_dist: Structural connectivity lengths (used in simulations with conduction delays)

• sim_seed: Random seed for generating simulation noise

In addition, all optimizers that are batched together must be:

• Subclasses of PymooOptimizer (e.g., CMAESOptimizer). Therefore, GridOptimizer for example is not supported.

• Of the same optimizer type across all runs, therefore it is not possible to mix e.g., CMA-ES with NSGA-II.

However, optimizer-specific properties, including seed, popsize and n_iter can differ between runs.

The toolbox may support varying additional features across batched problems and optimizers in the future versions.