Does training new tokenizer effect the pre-trained model?

[dinhngoc267]

Hi,

It's nice to see your repository. I can see that you train a new tokenizer based on your new corpus, but I wonder does it change the token id of the original model base? If it does then it might effect the weight of original pre-train model, as you continue to pre-train the model from the pre-trained model right?

[w11wo]

Hi @dinhngoc267, very sorry for the late reply. I might have missed the notification for some reason.

To clarify, we did not continue training from a pre-trained model. IndoT5 was trained completely from scratch, with a new vocabulary/tokenizer. It is only when we fine-tuned to a downstream task like QA, summarization, etc., where we fine-tuned our model; but we kept the same vocabulary for this step.

Hope that clears it up.