meta_trainer (faber_neuroevolution v1.2.4)

Gradient-based training for the meta-controller.

This module implements gradient-based optimization for updating the meta-controller's LTC network weights. It uses policy gradient methods adapted for continuous action spaces.

Training Algorithm

We use a simplified REINFORCE-style policy gradient:

1. Collect experience: (state, action, reward) tuples 2. Compute returns: G_t = sum of future discounted rewards 3. Estimate gradients: nabla_theta = E[G_t * nabla_theta log pi(a|s)] 4. Update weights: theta = theta + alpha * gradient

LTC-Specific Considerations

LTC neurons have temporal state that affects gradient flow: - Backpropagation through time (BPTT) for temporal dependencies - Truncated gradients for computational efficiency - Momentum to smooth updates across generations

Summary

Types

meta_config/0

meta_param/0

meta_training_event/0

reward_component/0

Functions

apply_gradients(Weights, Gradients, LearningRate)

Apply gradients to weights.

compute_gradients(Experience, Weights, Config)

Compute gradients from experience.

estimate_advantage(Reward, History)

Estimate advantage for a reward.

update_weights(Weights, Experience, Config, LearningRate)

Update meta-controller weights based on collected experience.

Types

meta_config/0

-type meta_config() ::
          #meta_config{network_topology :: {pos_integer(), [pos_integer()], pos_integer()},
                       neuron_type :: ltc | cfc,
                       time_constant :: float(),
                       state_bound :: float(),
                       reward_weights :: #{reward_component() => float()},
                       learning_rate :: float(),
                       param_bounds :: #{meta_param() => {float(), float()}},
                       control_population_size :: boolean(),
                       control_topology :: boolean(),
                       history_window :: pos_integer(),
                       momentum :: float()}.

meta_param/0

-type meta_param() ::
          mutation_rate | mutation_strength | selection_ratio | evaluations_per_individual |
          max_concurrent_evaluations | population_size | add_node_rate | add_connection_rate |
          complexity_penalty.

meta_training_event/0

-type meta_training_event() ::
          #meta_training_event{generation :: pos_integer(),
                               inputs :: [float()],
                               outputs :: [float()],
                               reward :: float(),
                               gradients :: #{term() => float()}}.

reward_component/0

-type reward_component() ::
          convergence_speed | final_fitness | efficiency_ratio | diversity_aware | normative_structure.

Functions

apply_gradients(Weights, Gradients, LearningRate)

-spec apply_gradients(map(), map(), float()) -> map().

Apply gradients to weights.

Uses gradient descent with optional momentum and gradient clipping.

compute_gradients(Experience, Weights, Config)

-spec compute_gradients([meta_training_event()], map(), meta_config()) -> map().

Compute gradients from experience.

Uses REINFORCE-style policy gradient estimation.

estimate_advantage(Reward, History)

-spec estimate_advantage(float(), [float()]) -> float().

Estimate advantage for a reward.

Advantage = reward - baseline, where baseline is a moving average.

update_weights(Weights, Experience, Config, LearningRate)

-spec update_weights(map(), [meta_training_event()], meta_config(), float()) -> map().

Update meta-controller weights based on collected experience.