Performance Benchmarks

[avik-pal]

When using regularization, the training doesn't scale very well with the increase in batch size:

All the benchmarks use the models from here and use Tracker.jl

Batch Size	Vanilla NODE forward	Vanilla NODE backward	Reg NODE forward	Reg NODE backward
1	0.285 ms	3.193 ms	0.315 ms	5.920 ms
64	1.067 ms	5.475 ms	1.147 ms	37.768 ms
256	3.360 ms	11.872 ms	3.671 ms	140.085 ms

[ChrisRackauckas]

How much is the overhead compared to other regularization methods, and how fast is the trained forward methods afterwards? Those are the two factors to understand. Let's get @jessebet on here to start discussing as well.

[avik-pal]

Updated the pdf with the FFJORD results on the gaussian dataset:

• At a tolerance of 6f-8 (both rtol and atol) the training times for 100 epochs are 35.23 hrs for vanilla FFJORD and 29.22 hrs for regularized one.
• The final inference timings are 53.08s and 52.17s respectively (batch size of 32).

The MNIST classification shows similar trend with a reduced training time when the tolerance is low. At lower tolerance, the training timing is slower. The overall inference time is still lower than unregularized one.

[avik-pal]

Regarding performance comparisons with other methods. The papers report GPU training times, but currently I am not being able to get the regularization to work on GPUs #42.

Once jacobjinkelly/easy-neural-ode#2 gets fixed, I can run these on CPU and measure the performance.

[avik-pal]

Proper Benchmarks available in the paper