Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move generation functionality to base classes #1861

Open
mattdangerw opened this issue Sep 22, 2024 · 1 comment
Open

Move generation functionality to base classes #1861

mattdangerw opened this issue Sep 22, 2024 · 1 comment
Assignees
Labels
type:feature New feature or request

Comments

@mattdangerw
Copy link
Member

We want it to be easy to add more generative models to the repository, and we want it to be easy to add new features related to generation.

Currently quite a bit of common functionality for generation live in every XXCausalLM subclass, which makes it hard to add new models and build new generation features. I think we can pull a lot of this functionality down into the base class. I also as part of this work we should move the compiled while loop out of the sampler class and into the CausalLM base class, which will make it much easier to write custom generative loops (e.g. would help integrating with JetStream).

@mattdangerw
Copy link
Member Author

This is mostly implemented, but still needs a little work. I'll push code shortly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:feature New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants