Introduction

The paper compares 31 nas algorithms in 4 search spaces and 4 datasets.

• The algorithm ranges from zero-cost, model-based, learning curve extrapolation, and weight sharing.
• Configs:
  • 101+cifar10, darts+cifar10
  • 201 + (c10, c100, imageNet),
  • nas-bench-NLP + Penn TreeBank.

After those experiments, the paper tries to

1. Find which predictors have consistent performance across search space from three dimensions comparison
2. Analysis insights.
3. Find complementary predictors and invesigate how to combine them

Experiments

Effectiveness

Measure SRCC, Pearson, Kendall Tau.

For prediction model, they train the model with 1k archs and then test on 200 archs.

Conclusion

1. low query time + low initialization time, Jacob and Synflow preform well.
2. Zero-cost do not perform well on large space like DARTS
3. High init time + low quer time, performance predictors are best.

Efficiency

The papre combine 3 metrics from 3 different families.

1. SoTL-E from learning curve methods
2. Jacob from zero-cost method
3. Both of above are used as input feature for a model-based predictor. (SemiNAS and NGBoost. )

After this, the papre measure how to use those metrics to speed up the NAS. It mainly use two methids.

1. Predictor-guided evolution framework.
2. Bayesian optimization.

This suggests that using zero-cost methods in conjunction with model-based methods is a promising direction for future study.

---

• ← Previous Post
• Next Post →