Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Consider a suppressed_tokens arg in generate() #975

Closed
abheesht17 opened this issue Apr 8, 2023 · 2 comments · May be fixed by #978
Closed

Consider a suppressed_tokens arg in generate() #975

abheesht17 opened this issue Apr 8, 2023 · 2 comments · May be fixed by #978
Labels
type:feature New feature or request

Comments

@abheesht17
Copy link
Collaborator

The user can optionally pass a suppressed_tokens arg to GPT-2/BART's generate() function. We will set the probability of these tokens to 0 (i.e., logits to -infinity), so that they aren't generated. This is super-useful in order to avoid generating a special token (like bos token) during generation.
@abheesht17 abheesht17 changed the title Consider a "suppressed_tokens" args in generate() Consider a suppressed_tokens args in generate() Apr 8, 2023
@abheesht17 abheesht17 changed the title Consider a suppressed_tokens args in generate() Consider a suppressed_tokens arg in generate() Apr 8, 2023
@abheesht17 abheesht17 mentioned this issue Apr 10, 2023
Draft
@mattdangerw
Copy link
Member

mattdangerw commented Apr 14, 2023 •
edited
Loading

I think the general problem we need to think about is how we transform logit outputs. We need to be able to ship default transformations (e.g. for whisper logit supression), and potentially allow users a hook to provide their own.

One option is adding more and more sampler config #978. A few other options to consider...

  • Users can pass in an arbitrary logit transformation somewhere. logit_transform_fn=None.
  • Users need to subclass a language model and override a method to do custom logit transformation.

@jbischof
Copy link
Contributor

I do like keeping this general instead of making framework-y promises that we can fix the "bad words"

@sachinprasadhs sachinprasadhs added the type:feature New feature or request label Apr 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type:feature New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add functionality to suppress tokens during generation abheesht17/keras-nlp
4 participants
@jbischof @mattdangerw @abheesht17 @sachinprasadhs