Decouple recipes from main distribution

[balajiv113]

Putting out the idea that was started in discord chat for better tracking

Problem
Currently the ferdium recipes are bundled together within the application distribution itself. Due to this we have a issue that we need to release a new version of ferdium whenever there is a fix/update done to the recipe itself.

Idea
One way is to decouple recipes from ferdium distribution. But to do this, we need the following,

• recipes library (host all recipes, manage different versions)
• service that support reading / fetching these info's and ultimately download recipes
• recipes should work with both no login and hosted service as well

Solution 1 - Treating recipes as a npm packages
The major work in the above idea is to create a service that manages the recipes library.

The steps will be more as follows,

• All the recipes should be deployed to a npm registry under a scope say (ferdium-recipes)
• ferdium should use API's exposed from NPM registry to fetch metadata's like available packages, versions
• When a recipe is needed, ferdium can do a similar step to npm install package

Note

• NPM registry should not have any rate-limits. As this will be a blocker in future when ferdium user base increases / during frequent check for updates call
• Need to implement required caches for avoiding unwanted registry calls

[balajiv113]

cc: @vraravam @SpecialAro
Feel free to add if i missed something from the conversation in discord

[vraravam]

Yes, a catalog with filter capability is required. If npm has APIs for this, great. If not, we will have to create and host a standalone service for the same. Could npmjs work for this? Or GH packages?

The act of installing a recipe is just to unzip and create/overwrite to the current user's Ferdium profile. I'm not sure whether we need a "heavy" tool like npm for installing on local filesystem. We already have that in place btw and so some minor tweaking should allow us to reuse most of this.

I have some more questions:

1. Also - do we need to allow recipes hosted from different (self-hosted) registries?
2. How do we tackle breaking changes? For eg, when we introduced Ferdium.safeParseInt or any of the other methods exposed by the Ferdium object, since we were in control of what gets packaged into the installable package, we were able to disregard backward compatibility by human logic. If we go ahead with this idea, we will need a mechanism to denote whether a recipe version is suitable for the version of Ferdium that's requesting the catalog/download.
3. Finally, currently, the ferdium-recipes repo is used as a git submodule for both the app and the hosted server. So, this change should be done consistently in both places.

[balajiv113]

npmjs or GH

On this, we need to study more related to rate-limits. If there are no rate-limits in npmjs i would prefer to use this in that case.

a catalog with filter capability is required

On this, my idea was to query all recipes under a scope and hold a cache locally. We can do cache-invalidation based on our usecase. Like once per day or manual trigger. But npm also provides text match as well

Example
To fetch all package under scope nest

https://registry.npmjs.org/-/v1/search?text=scope:nest

"heavy" tool like npm for installing on local filesystem

We will not use npm cli at all. We can use npm-registry-fetch for getting package info and do downloads manually and manage ourself. Even npm lock will not be needed

Also - do we need to allow recipes hosted from different (self-hosted) registries?

On this, my take would be to allow only our own scoped packages. Later we can surely support others as well. Only issue is, larger the search more delay/heavy the npm api calls will become

How do we tackle breaking changes?

Haven't given much thought on this, one quick idea i could think of is, use engine version (like {engines: {ferdium: ^6.2.0}} within our recipes package.json. This will provide a means for us to enforce which recipes are allowed under which ferdium application

ferdium-recipes repo is used as a git submodule for both the app and the hosted server

Am not sure why we use recipes in hosted server. With this modal i believe all should be managed by users within the app. The installed recipes info of user alone will go to hosted server db. We will not bundle recipes in both hosted server and self app as well

[vraravam]

Lets decouple the caching as a strategy to the point about removing the recipes from the bundled installer file - imo, that's more of an optimization. wdyt?

[vraravam]

On this, my take would be to allow only our own scoped packages. Later we can surely support others as well. Only issue is, larger the search more delay/heavy the npm api calls will become

Since there are individuals/companies who want to host the Ferdium server, they would need a mechanism to get the recipes. I guess this might be the reason for the recipes being hosted as a submodule and packaged inside the current server as well. Someone who knows the server code can confirm if this is exposed via an endpoint. If so, then we can simply replace both codebases to use whatever API (your suggestion of npm-registry-fetch could be an option) to expose the packaged recipes and the catalog.

[vraravam]

Haven't given much thought on this, one quick idea i could think of is, use engine version (like {engines: {ferdium: ^6.2.0}} within our recipes package.json. This will provide a means for us to enforce which recipes are allowed under which ferdium application

Yes - this could be done. Good idea.

[balajiv113]

Lets decouple the caching as a strategy to the point about removing the recipes from the bundled installer file - imo, that's more of an optimization. wdyt?

Ya ya i mentioned caching as further optimisation only (Just to avoid too many calls to npm registry). But we can stick to removing recipes :)

[jwmh]

Someone closed my ticket on a DIFFERENT aspect of the recipes distribution which hasn't yet been brought up(???) that I've seen:
#1168
#1168