Model development timeline

1. Transfer initialize, padding only the first conv layer to 5x5 and leaving all residuals.
2. If it goes well I can convert the first residual block, then the second, etc. until it's too slow or it stops helping.
3. Then just add as many 3x3 residuals as possible at the back, right before the output heads. Initialize to 0 (the identity for res blocks).