Introduction

Problems

1. It’s hard to design task-specific search spaces.
2. The search algorithm is tightly bound to search space, and it’s hard to implement every search algorithm once switching to another search space.

Current works

1. Train each sampled arch independently from search, => computationally prohibitive when facing large datasets

2. Weight sharing / network morphism / performance prediction => case-specific, relying on architecture-specific code

   to work. It requires lots of engineering work for adapting the search algorithm to a new search space.

Contribution

1. Integrate many popular NAS search algorithms.
2. Decoupling the search spaces, optimization algorithms, network transformations, and evaluation strategies, each of them is a reusable component.
3. Automatically generate search space by replacing modules of a given network with integrated components.
4. Experiments show the system can be efficiently utilized to carry out architecture searches for various DL applications. (Image classification, speech recognition, recommender system.)

Framework design

Different algorithm requires different ways of initializing the weight.

Usage cases

Subnet based search

Transform will init the new architecture’s weight from scratch.

SuperNet based search

Transform function selects the corresponding paths specified by alpha.

Network morphism

For OFA training, the transform just alters the computational graph topology ( decrease or increase the sizes. )

Additional features

Hyperparameter tuning: Reuse the origina frameworks,

Process pipeline: chaining search, training, and hyperparameter tuning processes to form an pipeline. There is a scheduler to schedule the tasks.

Distributed search support: For discrete NAS methods,

Evaluation

Setting

Tasks

Image classification task: Cifar10, Cifar100, ImageNet

Voice recogination task: CTR prediction challenge on Kaggle.

NAS benchmarks: Use the datasets from NASBench101

Search Space

Image classification:

• cell-based search space,
• ResNet50: search for convolutions with kernel size (3,5,7)
• MobileNetV2: kernel size, expansion ratio of convolutions in each residual block.

voice recognition task: ResNet-34 is used as the macro architecture, and variants of ResNet basic blocks are chosen as candidates.

Multi-branch network for the recommendation task. and search for feature interaction layers, including inner product, outer product, and MLPs.

Accuracy Result

Mainyl measure the top architecture’s accuracy across different datasets/algorithms/spaces..

Usability Result

Evaluate the LOC in implementing NAS, and compare it with other frameworks.

Use voice recognition task,

Pareto Optimal Search Result

Apart from searching for one architecture with the best accuracy, It also explored Pareto optimal architecture search with multi-objective discrete search algorithms, such as genetic algorithms or even random search.

Details

The algorithm details are in attachement.

---

• ← Previous Post
• Next Post →