New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix checkpoint #66

Open

gameofdimension wants to merge 1 commit into ankanbhunia:main from gameofdimension:gameofdimension-patch-1

gameofdimension commented Jun 11, 2024 •

edited

Loading

make use_reentrant=True explicit, because it will default to true if it is not assigned
fix gradient checkpoint when it used with dropout turned on. if preserve_rng_state=False, the dropout will definitely not work, because gradient flows into wrong input cells

it can be shown with a failed 300 epoch training, with preserve_rng_state=False and use_checkpoint=True.

some samples at the end of one failed training:


          fix checkpoint

c49855c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet