build RGL synopsis nightly

[odanoburu]

it'd be nice if Travis built http://www.grammaticalframework.org/lib/doc/synopsis.html from scratch for every commit, and published all the tagged versions and the latest commit, right?

[johnjcamilleri]

It sure would! We are eagerly awaiting your PR :)

The synopsis is hosted on a, um, normal server. How would we get the docs built on the Travis server onto the GF server? Can it do it via SCP or something like that?

Another thing it would be nice to get from Travis is nightly builds of gf-core, but that belongs in a separate issue...

[johnjcamilleri]

By "nightly" I mean "for each passing commit", and by "builds" I mean binary distributions.

[odanoburu]

Another thing it would be nice to get from Travis is nightly builds of gf-core

that would be nice to have indeed! but I don't think we'd be able to have windows builds unless trans-compilation from linux is possible.. do you know if the instructions here are up to date?

I'll try working on docs first, but, erm, how do I build them...? its just make all using the Makefile in the doc directory?

[johnjcamilleri]

I'll try working on docs first, but, erm, how do I build them...? its just make all using the Makefile in the doc directory?

Your guess is as good as mine! Honestly this is a part of the RGL I know little about. But a quick look at the Makefile seems to indicate that you are right. A welcome side-effect of this effort would be clean up the scripts which are no longer used

[odanoburu]

ok, I have a dockerfile with the build requirements (gfdoc, txt2tags, dot), and I can built the synopsis running (from the rgl root directory):

docker run --mount src="$(pwd)",target=/home,type=bind odanoburu/haskell-gf-doc /bin/bash -c "cd /home/; mkdir rgl ; export GF_LIB_PATH=/home/rgl ; runghc Make.hs install prelude all --verbose ; cd doc ; make all"

do we want to build any other documentation..? if not, what would be the best way to deploy the docs to the GF server? travis has this documentation for custom deployment. also, it would be nice if there was a page explaining why to use gfdoc and what's its syntax (of course one can pick it up from the examples, but...) or does that exist already?

[johnjcamilleri]

do we want to build any other documentation..?

Not that I know of.

what would be the best way to deploy the docs to the GF server?

My idea was simply using ssh/scp, although there's a security problem. The docker file is in a public repository, so how can we it access to the GF server without anyone else being able to do so?

An alternative is that the doc generation runs directly on the server via a git post-pull hook. I guess this won't require docker at all, just a cron job. The HTML pages from gf-core are already generated in this way on the server from txt2tags files. In fact I'm not sure why the RGL synopsis isn't generated automatically.

it would be nice if there was a page explaining why to use gfdoc and what's its syntax

I don't know of any, off the top of my head. It's possible that it's buried somewhere in all the various doc files but I cannot quickly find it.

As I mentioned before I would really like to clean up the RGL docs in general. There's a lot of ancient stuff there which we can get rid of. The synopsis document is good, actually a central part of the RGL, but could also do with some updating. One day I will find the time to go through it all but sadly I cannot do it at the moment.

[odanoburu]

My idea was simply using ssh/scp, although there's a security problem. The docker file is in a public repository, so how can we it access to the GF server without anyone else being able to do so?

it is usually done by accessing the credentials from an environment variable, which you can safely set on travis for the GF repository!

An alternative is that the doc generation runs directly on the server via a git post-pull hook. I guess this won't require docker at all, just a cron job. The HTML pages from gf-core are already generated in this way on the server from txt2tags files. In fact I'm not sure why the RGL synopsis isn't generated automatically.

but I like this solution better!

[Thomas-H]

The cron job that keeps www.grammaticalframework.org updated runs make html after pulling from github, so if you want to run more stuff, all you need to do is add it to the rule for html in Makefile…

[odanoburu]

@Thomas-H does the server also pull gf-rgl? if so, how is the directory structure? does it have gfdoc installed? (I imagine so)

the problem with using hooks and cron jobs is that we might build faulty documentation, isn't it? or does gfdoc never fail? @johnjcamilleri

[Thomas-H]

@odanoburu yes, it pulls gf-rgl too. It resides under lib, where it was before the repository split (but this could be changed). gfdoc wasn't installed, but it is now (assuming you are referring to src/tools/GFDoc.hs).

Another consideration is perhaps which version of the RGL it makes sense to document on the web site: the latest released version, the bleeding edge development version, or both?

[johnjcamilleri]

the problem with using hooks and cron jobs is that we might build faulty documentation, isn't it?

Yes, although it shouldn't be too difficult to modify the doc-building job to only run if building the RGL completes successfully, using the return value from make install.

Another consideration is perhaps which version of the RGL it makes sense to document on the web site: the latest released version, the bleeding edge development version, or both?

And this is related to the issue of the RGL release cycle (issue #3). Regardless of what we decide there, I think having both is best. In fact maybe we should build and store the docs for every single committed version of the RGL. This sounds like overkill and maybe creates some storage problems on the server, but it is arguably the simplest to manage.

[johnjcamilleri]

It resides under lib, where it was before the repository split (but this could be changed).

I think that if we make any significant changes to the RGL documentation (which I would like to see) then we should migrate it to a subdomain such as http://lib.grammaticalframework.org. The old content under http://www.grammaticalframework.org/lib can remain there with a warning that it's outdated.

[odanoburu]

I was thinking of keeping the stable versions and latest commit (whose page would include a link to the stable versions). I don't think we need to keep all versions if it's going to be some trouble, we can generate them with a git checkout them after all!

[odanoburu]

in #48 and in GrammaticalFramework/gf-core#11 I have added Portuguese to the synopsis, changed the makefile at gf-core, and added a link in the current synopsis to the stable version. how does that seem to you?

@Thomas-H does the server have graphviz/dot installed? I think the documentation needs it.

[Thomas-H]

@odanoburu yes.

[johnjcamilleri]

(Sorry for cross-posting)
I disagree with GrammaticalFramework/gf-core#11 - it's messy to have gf-core build the RGL documentation. We need a separate post-pull hook for gf-rgl on the server, not hijack gf-core's.

[odanoburu]

@johnjcamilleri I agree with your disagreement! I did get the feeling that that wasn't good at all -- it wouldn't be clear to anyone but us because it created a hidden link between the repositories.

@Thomas-H can we just add to the cron job a call of make -f lib/doc/Makefile all? it makes sense for it to be a call of make html, but because of the repository split it's better to separate things!

[Thomas-H]

@odanoburu In principle yes, but it looks like something more is needed to make it work. The Makefile assumes that GF_LIB_PATH is defined and that a directory $(GF_LIB_PATH)/alltenses exists. I assume that means that you have to build the RGL first?

[odanoburu]

well, we just import -retain the modules so unless the haskell scripts depend on the RGL being built, we could get around it by setting the variable from the command line (make all -e GF_alltenses=../src/api) and changing the .gfos to .gfs.

but on my computer the kernel killed the process (too much memory spent maybe?) and it is definitely much slower than using the .gfos, so if it's not too much trouble building the RGL I'd do it.

[Thomas-H]

The compiled RGL should be reused obviously, but I think this dependency should be made explicit in the Makefile(s), and all the commands needed to build the docs should also be there of course, and not hidden away in a cron job on the server.

I also think building the docs needs to be more incremental. Now it seems to take ~80 seconds, even if nothing has changed... (Rebuilding the RGL with make build takes ~6 seconds when nothing has changed. That's fine.).

[odanoburu]

The compiled RGL should be reused obviously, but I think this dependency should be made explicit in the Makefile(s)

do you think we should merge the Makefiles? or at least call the one in doc/ from the one in the directory root? I have little experience with Make so I apologize if I say something that doesn't make sense.

and all the commands needed to build the docs should also be there of course, and not hidden away in a cron job on the server.

yes, I agree.

[Thomas-H]

I have pushed a patch that adds a doc target in the main Makefile. It calls on doc/Makefile to build the documentation, after building the RGL.

I am also looking into making doc/Makefile more incremental.

[odanoburu]

@Thomas-H thanks, that looks great! is it live already? are we keeping #48 with the idea of having the stable versions of the synopsis + the current version? can you add Arabic, Basque, and Portuguese to the languages built? (I can add it in #48 if we're keeping it) I think other languages can also be added, maybe all of the ones that have a Try*.gf file?

[Thomas-H]

@odanoburu It is not live yet, because it would overwrite the existing synopsis.html. Does #48 fix that? What I think #48 should do is git mv synopsis.html synopisis-v3.9.html to make room for the automatically generated synopsis.html (which should not be added to the repository). (This assumes that the current synopsis.html corresponds to the gf-3.9 release. I am not sure it does, but maybe it is close enough?)

There is also a conflict in #48 doc/Makefile now that needs to be fixed. The addition of Portuguese and the automatic generation of synopsis.html are two unrelated things, so even better if these things were two separate pull requests!

Also, in the past @aarneranta used to update the synopsis manually, so it would be good to know that he approves of this plan (i.e. keeping synopsis-v3.9.html around and automatically updating synopsis.html from a cron job on the server). A separate issue is perhaps which version the GF home page should link to...

[odanoburu]

It is not live yet, because it would overwrite the existing synopsis.html. Does #48 fix that?

yes!

This assumes that the current synopsis.html corresponds to the gf-3.9 release.

we can use the GF 3.9 binary to generate it, but I think using the current file might be best!

The addition of Portuguese and the automatic generation of synopsis.html are two unrelated things, so even better if these things were two separate pull requests!

indeed, will do!

[Thomas-H]

OK, I have applied the pull requests and added make doc to the cron job, to update the synopsis. Have a look and see if it works as expected.

[odanoburu]

looks good! can we close this @johnjcamilleri ?

[Thomas-H]

I just noticed that some details in synopsis-intro.txt need to be updated manually: it says that there are 32 languages and enumerates them, but I guess there are 34 languages now. There are also some links under "The main contents are:" that now point to the wrong section...

There probably some more details to fix in the Makefile, e.g. maybe categories.png should also exist in two versions and be updated automatically...

[odanoburu]

we should probably think of a way of updating these automatically.. and also not having several lists of languages. maybe we could use languages.csv for that?

[johnjcamilleri]

maybe we could use languages.csv for that?

I definitely think that's the way go to. One list of languages to rule them all.
We should be able to add columns to language.csv without needing any changes to the build scripts. (Side note: I would rather use some nicer config format like YAML, but this will make the build scripts more complicated by either adding dependencies or having to implement YAML parsing. So I think we should stick to CSV).

[Thomas-H]

I just had a chat with Aarne, and he said (paraphrasing) that languages should be included in the synopsis when they are considered complete enough to be recommended to users of the RGL. Maybe there should be a column that reflects this human judgement in languages.csv?

[johnjcamilleri]

I'm not against adding this column, but I wonder if this information is already implied by one of the existing columns? (e.g. All && Try?)

[johnjcamilleri]

Maybe there should be a column that reflects this human judgement in languages.csv?

Done! Check out ec9f74d

[odanoburu]

I'm closing this because I believe this issue has been solved; please reopen it if my understanding is wrong!

[johnjcamilleri]

Not quite. Currently the synopsis is always built for the latest version of the RGL. It would be good to have a fixed version for release 3.10 too. The previous solution was ok but feels a bit too manual —someone has to remember to build the synopsis for the release, give it a certain name, hardcode the link in the live documentation etc.

Keeping a history of the synopsis for every commit would be better because it's automatic, and would allow developers to view the docs of any RGL version. The main issue is server space. Each index.html would be around 1.6 MB. This is compressible to ~180 kb, but "browsing" through tgz files is not exactly user friendly...