Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to a minimal archive save method #16

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

lnksz
Copy link
Contributor

@lnksz lnksz commented Jun 30, 2017

Storing the whole concrete model can take a lot of time, furthermore, for plotting or analysis of the results, it is an overkill.

@ojdo
Copy link
Collaborator

ojdo commented Jul 2, 2017

May I submit to you an alternative route? For urbs, I alsow wanted to speed up storing the result, while not having the user to maintain pre-knowledge of what they wanted to know from the result later. For that, I collected thoughts in issue tum-ens/urbs#47 and implemented a generic result cache generator in PR tum-ens/urbs#117.

Could you have a look at that and decide whether you like that idea, too? If not, I would be fine with merging this. I just thought that you might find the other design (almost no user-facing changes in "contract" with what the model object contains, except for a missing model after a reload from a HDF5 file) applicable here, too.

@lnksz
Copy link
Contributor Author

lnksz commented Jul 3, 2017

Thank you for the suggestion. I will have a look on the urbs solution.

HDF5

If I remember correctly MatLab uses HDF5, so I suppose that is a method worth considering.
(I suppose, the added "cache" is not that big to have a big effect during run-time, right?)
Have you made some benchmarks, to see the difference between pickle-gzip-all versus the HDF5 version? (Time and size difference would be interesting especially)

Modelless pickle

For transparency, you altered get_constants() to handle the _result "cache".
I could see that happening with the proposed minimal save too.
Then hdf5 conversion is not needed, and the function calls remain the same.
Of course, this implicates that the user is somehow expecting the minimal data.
For your concerns regarding the user:

  • What about a simple warning/info during load? (At save the user already has to know what he is doing to achieve minimal export)
  • Or simply duck-typing (as advocated in your other code-review) could reveal possible incompatibilities if someone tries to use a data-prob as a pyomo-prob.
  • Anyhow, I see that this is handling a sub problem, as it does not solve the case if someone does want to store the whole model
    • I already thought about this, and wanted to ask, whether you already profiled the create_model() function?
      EDIT: Speeding up model creation #26 happened since this comment. Model creation is not so slow as used to be. Makes the following question more legit in my opinion.
    • Or what is the sense of storing the whole model? Why would somebody load and then re-run it instead of recreating the model and run it, if he already has the input data + results and does not want to modify the input parameters?

@lnksz lnksz mentioned this pull request Aug 27, 2017
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants