layout: post

Introduction

Contributions

1. Evaluate the main models.
2. It shows that the ResNet-like model is effective
3. Introduce FT-Transformer, a simple adaptation of transformer for tabular data
4. No universally superior solution among GBDT and deep models.

Existing work

1. Tree-based models
   1. XGBoost, LightGBM, CatBoost.
2. Deep learning models
   1. Differentiable trees (traditional tree is not differentiable).
   2. Attention-based models ().
   3. Explicit modeling of multiplicative interactions between features.
      1. MLP is unsuitable for modeling the multiplicative interactions between features.

FT-Transformer

Feature Tokenizer

It transforms the input features x to embedding T.

• Numerical Features: multiple by a W in element-wise.
• Categorical Features: Lookup Table.

Transformer

asdf

---

• ← Previous Post
• Next Post →