training RF-DETR from scratch

[pabloprz6]

Hello, I would like to know if I can training a RF-DETR from scratch.

I have done a benchmark between several models whose trainings have been from scratch so I want to train a RF-DETR model from scratch to include this model in the benchmark and have a fair comparison

[isaacrob-roboflow]

You can if you want, don't think we've built a nice interface to do that though. Performance should be disproportionately bad as the ViT backbone compensates for its lack of inductive bias via pretraining. Happy to link some papers on that if you want.

[pabloprz6]

You can if you want, don't think we've built a nice interface to do that though. Performance should be disproportionately bad as the ViT backbone compensates for its lack of inductive bias via pretraining. Happy to link some papers on that if you want.

I'm interested about some papers on that, thank you.

[isaacrob-roboflow]

the big theme basically is that ViT models scale super well with a huge amount of data and then do well at transfer time, but training on the same small to medium scale datasets need extra tricks to match performance of convolutional networks.

the original ViT paper discuss that ViT models do best when pretrained on lots of data and transferred

DeiT does an in-depth analysis of how to get good performance with a ViT without significant pretraining, highlighting that with extra tricks you can still do well

how to train your ViT does lots of ablations on what factors impact the training dynamics of a ViT, showing that significant regularization such as aggressive augmentation are very helpful when training on smaller datasets

a personal favorite of mine is Hiera, which removes structural implicit biases from ViT hybrid models and replaces them with large pretraining, showing that the pretraining explicitly compensates for the lack of inductive bias in the model. while not directly a ViT, it shows the idea of pretraining vs structural inductive bias and it's likely the same thing is happening in vanilla ViT models. the consequence here is that removing those structural things can make the model run significantly faster while maintaining accuracy. they do claim speedup over plain ViT, but we find the opposite, please refer to the RF-DETR paper for more details on that ablation

TLDR is that RF-DETR uses massive pretraining to both compensate for a lack of inductive bias and gain world knowledge to better transfer to downstream datasets

[rlewkowicz]

(I'm just a hobbyist, so take my input with a grain of salt), the TLDR on this, is more or less, with the transformer based models, vs convolutions, you essentially will never train a model from scratch. Dinov2 and v3 are god mode models. They are the pinnacle foundation models trained on more data than you can ever acquire naturally. In fact, I think it was trained on more than they could acquire naturally. I think they have large synthetic data sets.

In vision you have, "is it?", "what is it?", "where is it?". Out of the box, un-tuned, The foundation models excel at "is it?" better than anything else on the planet.

[isaacrob-roboflow]

People use ViT models because they scale well. When you throw large amounts of data at them, they work extremely well. When training on small datasets, it is almost always helpful to use a pretrained model, and because transformers scale so well, they are often the best available pretrained model, so it is often a good choice to use a ViT to transfer to new small datasets.

Foundation models excel at answering all three of those questions, as long as you know how to ask. By that, I mean using a frozen foundation model you can train additional layers on top of it that can answer any of those questions, which implies that the answers do live somewhere in the foundation model. For information on training with frozen models, the DINOv2 and v3 papers are I think excellent sources of information.