subsampling LOO estimates with diff-est-srs-wor start #496
Conversation
This reverts commit 781e331.
|
I wanted to use R2, and as I had rewrote summary stats anyway, added R2 and made all mse, rmse, and R2 to use only normal approximation with as much shared computation as possible With the added R2 support, this PR will close also #483 |
|
I can take a look |
|
I have added some comments, but I'm not done with the review yet. Besides, I think documentation needs to be updated (at least re-roxygenized, but also progressr should be mentioned), the vignettes perhaps as well, and I haven't run |
| value.se <- weighted.sd((mu - y)^2 - (mu.bs - y)^2, wcv, | ||
| na.rm = TRUE) / | ||
| sqrt(n_notna) | ||
| # Use normal approximation for mse and delta method for rmse and R2 |
There was a problem hiding this comment.
Is this supposed to refer to the standard error estimation method or the CI method? The first part ("normal approximation") refers to a CI method, but the second part ("delta method") to a standard error estimation method.
| # Compute mean and variance in log scale by matching the variance of a | ||
| # log-normal approximation |
There was a problem hiding this comment.
Is it common to assume a log-normal distribution as the sampling distribution of the MSE and RMSE estimators? I haven't seen that yet (I think), but it might be perfectly fine (motivated by the central limit theorem, I guess).
| # store for later calculations | ||
| mse_e <- value | ||
| if (!is.null(summaries_baseline)) { | ||
| # delta=TRUE, variance of difference of two normally distributed |
There was a problem hiding this comment.
There is something missing at the end; perhaps "random variables"?
| ((mu_baseline - y)^2 - mse_b))[loo_inds], | ||
| y_idx = loo_inds, | ||
| w = wobs) | ||
| cov_mse_e_b <- srs_diffe$y_hat / n_full^2 |
There was a problem hiding this comment.
Just for my understanding: This procedure for estimating the covariance between mse_e and mse_b assumes that the summands within mse_e and mse_b coming from different observations are uncorrelated, right? At first, I thought this was violated here because mu, mu_baseline, and summaries_fast$mu are model-based and hence there is potential for cross-observations dependencies, but then I realized that mu, mu_baseline, and summaries_fast$mu all are based on the leave-one-out principle, so is this the reason why we can assume a cross-observation correlation of zero here?
|
When I changed several bootstraps to analytic approximations and improved other approximations, I thought the math I was writing in the code was so trivial that I didn't write all the derivations and assumptions separately. Now I see I should have done that, as it takes also me a bit of time to re-check any of these when you ask a question, so they are not as obvious as I thought them to be. If you like, I can some day write the equations and assumptions for easier checking. Before that, at least every approximation I wrote is based on the tests at least as accurate as the earlier bootstrap, but much faster. |
R/summary_funs.R
Outdated
| mse_y <- mean(wobs * (mean(y) - y)^2) | ||
| value <- 1 - mse_e / mse_y - ifelse(is.null(summaries_baseline), 0, 1 - mse_b / mse_y) | ||
| # the first-order Taylor approximation of the variance | ||
| var_mse_y <- .weighted_sd((mean(y) - y)^2, wobs)^2 / n_full |
There was a problem hiding this comment.
For mean(y), don't we need to take wobs into account? I think this is similar to line var_mse_b <- .weighted_sd((mu_baseline - y)^2, wobs)^2 / n_full where the parameter estimates from which mu_baseline is computed also take wobs into account.
There was a problem hiding this comment.
This also concerns several other occurrences of mean(y) here in get_stat().
R/summary_funs.R
Outdated
| if (!is.null(summaries_baseline)) { | ||
| # delta=TRUE | ||
| mse_e <- mse_e - mse_b | ||
| } | ||
| value_se <- sqrt((value_se^2 - | ||
| 2 * mse_e / mse_y * cov_mse_e_y + | ||
| (mse_e / mse_y)^2 * var_mse_y) / mse_y^2) |
There was a problem hiding this comment.
If my understanding from stan-dev/loo#205 (comment) is correct, then I think we would need a trivariate delta method in the !is.null(summaries_baseline) case (because mse_b comes in, too). I haven't checked whether such a trivariate delta method would give the same formula as used here. Have you checked this?
For me, does not work (runs into some error, similarly as works for me. Does it work for you as well (on |
|
With |
That sounds good. The warnings probably arise from the first creation of the snapshots. If you are running the tests via Since this solution seems to be working for you, I will push a commit to |
…of `cv_varsel()`
This fixes commit 0d73c8e. However, before commit 0d73c8e, `is.null(mu_baseline)` should have never occurred because if `summaries_baseline` was `NULL`, then `mu_baseline` was set to `0` (and if `summaries_baseline` was not `NULL`, then `mu_baseline` was set to `summaries_baseline$mu` which should not be `NULL` either). Hence, this fixup here does not only fix commit 0d73c8e, but also the incorrect behavior which existed before it.
| # log-normal approximation | ||
| # https://en.wikipedia.org/wiki/Log-normal_distribution#Arithmetic_moments | ||
| mul <- log(value^2 / sqrt(value_se^2 + value^2)) | ||
| varl <- log(1 + value_se^2 / value^2) |
There was a problem hiding this comment.
Would it make sense to use log1p() here (for numerical stability)?
fix `validate_baseline()`'s check
always needs to match `summaries_fast`)
(`mu_fast`, `loo_inds`, and `n_loo` were not properly updated when de-aggregating the aggregated dataset)
applied to the subsampled LOO results)
also need to be applied to the subsampled LOO results)
|
While working on the tests, I discovered a few bugs and fixed them. However (apart from what is still open in the comments above), we still need to think about the following:
For the tests, I think it would make sense to add subsampled-LOO Finally, while experimenting with the tests, I discovered some strange projpred/tests/testthat/test_varsel.R Lines 1452 to 1456 in 7469aea excl_nonargs(args_cvvs_i, nms_excl_add = "validate_search")) on tstsetup <- "rstanarm.glm.cumul.stdformul.without_wobs.with_offs.latent.default_meth.default_cvmeth.default_search_trms". This needs to be investigated.
|
`validate_search = FALSE`, the search is not run again when creating `summaries_fast`. Only the performance evaluation (including the re-projections required for it) is run again. Hence, it would be inconsistent to treat `summaries_fast` like the search-related arguments of `cv_varsel.refmodel()` when calling `cv_varsel.refmodel()` from within `cv_varsel.vsel()`. Thus, a lot of code related to `summaries_fast` can be removed, which is done here.
I think I have forgotten the context to understand this question
Fine for me. NA handling was there before I started this PR, and I left most of that there. |
No problem, I think I figured this out and pushed commit 0dcfcf2 which resolves this.
Ok, I will check this in detail and will commit changes (if possible). |
why some test snapshots changed unexpectedly).
|
I just pushed commit 6151b0c which reverts changes that are unrelated to subsampled LOO-CV because I observed some unexpected changes in the test snapshots. The state before reverting these subsampling-unrelated changes is now in branch Lines 797 to 891 in 6151b0c Lines 819 to 902 in 0dcfcf2 misc_from_fix-subsampling) after this PR has been merged. And, of course, there is still branch mixed_deltas_plot which needs to be worked on and eventually merged as well.
|
Work in progress.
Tested with
Notes
cv_varselif nloo<n and fast PSIS-LOO result is not yet available, fast PSIS-LOO result is computedcv_varselif nloo<n, fast PSIS-LOO result is stored in slot $summaries_fastNext
tagging @n-kall