Aurelien said the DistilBERT’s bias might be developed during pretraining.