HYPERBAND BANDIT-BASED CONFIGURATION EVALUATION FOR HYPERPARAMETER OPTIMIZATION
1 minute read ∼ Filed in : A paper noteHyperBand
The successive halving method suffers from a trade-off between selecting the number configurations and allocating the budget. To solve this problem, HyperBand proposed to frequently perform the successive halving method with different budgets to find the best configurations.
Algorithm
R: the maximum amount of resource that can be allocated to a single configuration
- the training set size for dataset downsampling;
- limitations based on memory constraints for feature downsampling;
- rule of thumb regarding the number of epochs when iteratively training neural networks
1/η: how many configurations to retain in each round. if η=3. each round retains 1/3 configurations。
- A large value corresponds to fewer rounds of elimination, and an aggressive elimination schedule.
Example
