Pooler type of pretrained models?

[juliusc]

I'm trying to use the pretrained mSimCSE models, and I see that in the evaluate_*.py files, the default pooler is cls_before_pooler. I also see that when loading xlm-roberta-large-cross_all with AutoModel, the weights in the model's pooler are uninitialized (they are randomized every time I load). All of this suggests that pooler layer is unused.

However, in the train_*.sh files, pooler_type is set to cls, suggesting that these models were trained with a pooling layer. Can you clear up whether the downloadable pretrained models were trained with a pooling layer and confirm whether the provided evaluation scripts are correct?

Thanks!