Wayback Machine & website metadata

[thomas-zahner]

Hi there, as part of our bachelor thesis we started working on a link checking tool and after some research came across lychee.
lychee fits our use-cases very well and we are trying to integrate lychee with our CLI application (which is why we've noticed the stderr bug) and we are also thinking about extending lychee itself. For the moment we have two ideas in mind how we could potentially extend the functionality of lychee.

One of our main use-cases would be to replace broken links with links from the Wayback Machine/Internet Archive. We saw that a GitHub action exists already for that use case. Would you think it made sense to add a Wayback Machine integration into lychee directly, so that one can receive suggestions by using the CLI directly?

Another use-case, our second idea regarding lychee, involves the incorporation of website metadata (for example the website title) for link checking. By extending the config file or creating a new "snapshot" file, lychee could throw a warning or error when a website's metadata changes, so that the content of a website doesn't change drastically without noticing. Do you think such a feature would be suitable for lychee?

[mre]

Hey Thomas,

exciting ideas and thanks for considering lychee as part of your thesis work.

Considering Wayback Machine, this already was suggested in #283. I'd be interested to add that feature.

I would be vary to add metadata checking into lychee's core, however we could think of a simple plugin system, which takes a URI and returns a status. Then we could do whatever we want and you could keep the snapshot of the metadata in a cahce file, Redis, or in a MySQL database for example.

Happy to further discuss these ideas. I'll send you an email to have a chat soon; hope that's fine.