MMM Calibration

[PrathameshKachkure]

While training a model I encountered an issue during model calibration. We initially built a model using 2 years of data and then added synthetic spend data for a media channel over the recent 4 weeks. This channel had 0 contribution earlier, and after adding significant spend data for the last 4 weeks, it still showed 0 contribution.
To address this, we calibrated the model using incremental revenue and spends for the same 4-week period, but the channel continued to show 0 contribution. However, when we increased the training size, we started to see contribution for this media channel.
Could anyone share best practices or insights on how to effectively calibrate the model for recent periods?

@gufengzhou @laresbernardo

[laresbernardo]

If all the spending period was excluded from the training dataset then there's no spending data being used for that new channel, thus the coef will be zero. If you use all the available data, and also add the calibration input (calibration input is part of the training data), then it should have a larger impact than 0.
Notice the test periods will always be the last periods of the available data.

[PrathameshKachkure]

thanks @laresbernardo for your response,

I have a few follow-up questions regarding calibration:

What should we typically expect from calibration in terms of impact on the model’s outputs?
For the period that we've calibrated, should we expect the ROI to align closely with the experiment's ROI?
How does calibration impact the overall contribution distribution between paid media, baseline, and other variables?
After calibration, should we generally expect the contribution of paid media variables to increase, especially if the synthetic spend data was added for recent periods?

[amanrai2508]

Hi @laresbernardo
Let’s assume our MMM model (up to July 2024) indicated a 0% contribution for a particular channel. We then ran an experiment for August for this channel and recalibrated the model, setting ts_validation to false.
In these cases we will be able to get the contribution, right ?

Currently we are not getting it ?