Loading model weights more efficiently

[kerthcet]

What would you like to be added:

Right now we can download model weights from model hub directly, but each time we start/restart a pod, it will downloading the model weights again. Without the loading accelerators like fluid or dragonfly, we should think of a way to tackle this more efficiently, let's focus on three things:

• download the models the first time should be as quick as possible
• don't need to download the model weights again when pod restarted
• handle the model cache efficiently

Why is this needed:

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[kerthcet]

/milestone v0.1.0

[kerthcet]

/kind feature

[kerthcet]

/assign

[kerthcet]

We may implement a simplified p2p network for efficient model distributing. See https://github.com/InftyAI/Manta

[kerthcet]

How transformer handles large models: https://huggingface.co/docs/transformers/big_models