Restructure modules (#1572)

* Move portbase into monorepo

* Add new simple module mgr

* [WIP] Switch to new simple module mgr

* Add StateMgr and more worker variants

* [WIP] Switch more modules

* [WIP] Switch more modules

* [WIP] swtich more modules

* [WIP] switch all SPN modules

* [WIP] switch all service modules

* [WIP] Convert all workers to the new module system

* [WIP] add new task system to module manager

* [WIP] Add second take for scheduling workers

* [WIP] Add FIXME for bugs in new scheduler

* [WIP] Add minor improvements to scheduler

* [WIP] Add new worker scheduler

* [WIP] Fix more bug related to new module system

* [WIP] Fix start handing of the new module system

* [WIP] Improve startup process

* [WIP] Fix minor issues

* [WIP] Fix missing subsystem in settings

* [WIP] Initialize managers in constructor

* [WIP] Move module event initialization to constrictors

* [WIP] Fix setting for enabling and disabling the SPN module

* [WIP] Move API registeration into module construction

* [WIP] Update states mgr for all modules

* [WIP] Add CmdLine operation support

* Add state helper methods to module group and instance

* Add notification and module status handling to status package

* Fix starting issues

* Remove pilot widget and update security lock to new status data

* Remove debug logs

* Improve http server shutdown

* Add workaround for cleanly shutting down firewall+netquery

* Improve logging

* Add syncing states with notifications for new module system

* Improve starting, stopping, shutdown; resolve FIXMEs/TODOs

* [WIP] Fix most unit tests

* Review new module system and fix minor issues

* Push shutdown and restart events again via API

* Set sleep mode via interface

* Update example/template module

* [WIP] Fix spn/cabin unit test

* Remove deprecated UI elements

* Make log output more similar for the logging transition phase

* Switch spn hub and observer cmds to new module system

* Fix log sources

* Make worker mgr less error prone

* Fix tests and minor issues

* Fix observation hub

* Improve shutdown and restart handling

* Split up big connection.go source file

* Move varint and dsd packages to structures repo

* Improve expansion test

* Fix linter warnings

* Fix interception module on windows

* Fix linter errors

---------

Co-authored-by: Vladimir Stoilov <[email protected]>

Earthfile

@@ -202,7 +202,7 @@ go-build:
             ENV EXTRA_LD_FLAGS = ""
         END
 
-        RUN --no-cache go build -ldflags="-X github.com/safing/portbase/info.version=${VERSION} -X github.com/safing/portbase/info.buildSource=${SOURCE} -X github.com/safing/portbase/info.buildTime=${BUILD_TIME} ${EXTRA_LD_FLAGS}" -o "/tmp/build/" ./cmds/${bin}
+        RUN --no-cache go build  -ldflags="-X github.com/safing/portmaster/base/info.version=${VERSION} -X github.com/safing/portmaster/base/info.buildSource=${SOURCE} -X github.com/safing/portmaster/base/info.buildTime=${BUILD_TIME}" -o "/tmp/build/" ./cmds/${bin}
     END
 
     DO +GO_ARCH_STRING --goos="${GOOS}" --goarch="${GOARCH}" --goarm="${GOARM}"

assets/icons_default.go

@@ -11,7 +11,7 @@ import (
 
 	"golang.org/x/image/draw"
 
-	"github.com/safing/portbase/log"
+	"github.com/safing/portmaster/base/log"
 )
 
 // Colored Icon IDs.

base/.gitignore

@@ -0,0 +1,8 @@
+portbase
+apitest
+misc
+
+go.mod.*
+vendor
+go.work
+go.work.sum

base/README.md

@@ -0,0 +1,157 @@
+> **Check out our main project at [safing/portmaster](https://github.com/safing/portmaster)**
+
+# Portbase
+
+Portbase helps you quickly take off with your project. It gives you all the basic needs you would have for a service (_not_ tool!).
+Here is what is included:
+
+- `log`: really fast and beautiful logging
+- `modules`: a multi stage, dependency aware boot process for your software, also manages tasks
+- `config`: simple, live updating and extremely fast configuration storage
+- `info`: easily tag your builds with versions, commit hashes, and so on
+- `formats`: some handy data encoding libs
+- `rng`: a feedable CSPRNG for great randomness
+- `database`: intelligent and syncable database with hooks and easy integration with structs, uses buckets with different backends
+- `api`: a websocket interface to the database, can be extended with custom http handlers
+
+Before you continue, a word about this project. It was created to hold the base code for both Portmaster and Gate17. This is also what it will be developed for. If you have a great idea on how to improve portbase, please, by all means, raise an issue and tell us about it, but please also don't be surprised or offended if we ask you to create your own fork to do what you need. Portbase isn't for everyone, it's quite specific to our needs, but we decided to make it easily available to others.
+
+Portbase is actively maintained, please raise issues.
+
+## log
+
+The main goal of this logging package is to be as fast as possible. Logs are sent to a channel only with minimal processing beforehand, so that the service can continue with the important work and write the logs later.
+
+Second, is beauty, both in form what information is provided and how.
+
+You can use flags to change the log level on a source file basis.
+
+## modules <small>requires `log`</small>
+
+packages may register themselves as modules, to take part in the multi stage boot and coordinated shutdown.
+
+Registering only requires a name/key and the `prep()`, `start()` and `stop()` functions.
+
+This is how modules are booted:
+
+- `init()` available: ~~flags~~, ~~config~~, ~~logging~~, ~~dependencies~~
+  - register flags (with the stdlib `flag` library)
+  - register module
+- `module.prep()` available: flags, ~~config~~, ~~logging~~, ~~dependencies~~
+  - react to flags
+  - register config variables
+  - if an error occurs, return it
+  - return ErrCleanExit for a clean, successful exit. (eg. you only printed a version)
+- `module.start()` available: flags, config, logging, dependencies
+  - start tasks and workers
+  - do not log errors while starting, but return them
+- `module.stop()` available: flags, config, logging, dependencies
+  - stop all work (ie. goroutines)
+  - do not log errors while stopping, but return them
+
+You can start tasks and workers from your module that are then integrated into the module system and will allow for insights and better control of them in the future.
+
+## config <small>requires `log`</small>
+
+The config package stores the configuration in json strings. This may sound a bit weird, but it's very practical.
+
+There are three layers of configuration - in order of priority: user configuration, default configuration and the fallback values supplied when registering a config variable.
+
+When using config variables, you get a function that checks if your config variable is still up to date every time. If it did not change, it's _extremely_ fast. But if it, it will fetch the current value, which takes a short while, but does not happen often.
+
+    // This is how you would get a string config variable function.
+    myVar := GetAsString("my_config_var", "default")
+    // You then use myVar() directly every time, except when you must guarantee the same value between two calls
+    if myVar() != "default" {
+      log.Infof("my_config_var is set to %s", myVar())
+    }
+    // no error handling needed! :)
+
+WARNING: While these config variable functions are _extremely_ fast, they are _NOT_ thread/goroutine safe! (Use the `Concurrent` wrapper for that!)
+
+## info
+
+Info provides a easy way to store your version and build information within the binary. If you use the `build` script to build the program, it will automatically set build information so that you can easily find out when and from which commit a binary was built.
+
+The `build` script extracts information from the host and the git repo and then calls `go build` with some additional arguments.
+
+## formats/varint
+
+This is just a convenience wrapper around `encoding/binary`, because we use varints a lot.
+
+## formats/dsd <small>requires `formats/varint`</small>
+
+DSD stands for dynamically structured data. In short, this a generic packer that reacts to the supplied data type.
+
+- structs are usually json encoded
+- []bytes and strings stay the same
+
+This makes it easier / more efficient to store different data types in a k/v data storage.
+
+## rng <small>requires `log`, `config`</small>
+
+This package provides a CSPRNG based on the [Fortuna](https://en.wikipedia.org/wiki/Fortuna_(PRNG)) CSPRNG, devised by Bruce Schneier and Niels Ferguson. Implemented by Jochen Voss, published [on Github](https://github.com/seehuhn/fortuna).
+
+Only the Generator is used from the `fortuna` package. The feeding system implemented here is configurable and is focused with efficiency in mind.
+
+While you can feed the RNG yourself, it has two feeders by default:
+- It starts with a seed from `crypto/rand` and periodically reseeds from there
+- A really simple tickfeeder which extracts entropy from the internal go scheduler using goroutines and is meant to be used under load.
+
+## database <small>requires `log`</small>
+_introduction to be written_
+
+## api <small>requires `log`, `database`, `config`</small>
+_introduction to be written_
+
+## The main program
+
+If you build everything with modules, your main program should be similar to this - just use an empty import for the modules you need:
+
+    import (
+      "os"
+      "os/signal"
+      "syscall"
+
+      "github.com/safing/portmaster/base/info"
+      "github.com/safing/portmaster/base/log"
+      "github.com/safing/portmaster/base/modules"
+
+      // include packages here
+      _ "path/to/my/custom/module"
+    )
+
+    func main() {
+
+    	// Set Info
+    	info.Set("MySoftware", "1.0.0")
+
+    	// Start
+    	err := modules.Start()
+    	if err != nil {
+    		if err == modules.ErrCleanExit {
+    			os.Exit(0)
+    		} else {
+    			os.Exit(1)
+    		}
+    	}
+
+    	// Shutdown
+    	// catch interrupt for clean shutdown
+    	signalCh := make(chan os.Signal)
+    	signal.Notify(
+    		signalCh,
+    		os.Interrupt,
+    		syscall.SIGHUP,
+    		syscall.SIGINT,
+    		syscall.SIGTERM,
+    		syscall.SIGQUIT,
+    	)
+    	select {
+    	case <-signalCh:
+    		log.Warning("main: program was interrupted")
+    		modules.Shutdown()
+    	case <-modules.ShuttingDown():
+    	}
+
+    }

base/api/api_bridge.go

@@ -0,0 +1,173 @@
+package api
+
+import (
+	"bytes"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"path"
+	"strings"
+	"sync"
+
+	"github.com/safing/portmaster/base/database"
+	"github.com/safing/portmaster/base/database/record"
+	"github.com/safing/portmaster/base/database/storage"
+)
+
+const (
+	endpointBridgeRemoteAddress = "websocket-bridge"
+	apiDatabaseName             = "api"
+)
+
+func registerEndpointBridgeDB() error {
+	if _, err := database.Register(&database.Database{
+		Name:        apiDatabaseName,
+		Description: "API Bridge",
+		StorageType: "injected",
+	}); err != nil {
+		return err
+	}
+
+	_, err := database.InjectDatabase("api", &endpointBridgeStorage{})
+	return err
+}
+
+type endpointBridgeStorage struct {
+	storage.InjectBase
+}
+
+// EndpointBridgeRequest holds a bridged request API request.
+type EndpointBridgeRequest struct {
+	record.Base
+	sync.Mutex
+
+	Method   string
+	Path     string
+	Query    map[string]string
+	Data     []byte
+	MimeType string
+}
+
+// EndpointBridgeResponse holds a bridged request API response.
+type EndpointBridgeResponse struct {
+	record.Base
+	sync.Mutex
+
+	MimeType string
+	Body     string
+}
+
+// Get returns a database record.
+func (ebs *endpointBridgeStorage) Get(key string) (record.Record, error) {
+	if key == "" {
+		return nil, database.ErrNotFound
+	}
+
+	return callAPI(&EndpointBridgeRequest{
+		Method: http.MethodGet,
+		Path:   key,
+	})
+}
+
+// Get returns the metadata of a database record.
+func (ebs *endpointBridgeStorage) GetMeta(key string) (*record.Meta, error) {
+	// This interface is an API, always return a fresh copy.
+	m := &record.Meta{}
+	m.Update()
+	return m, nil
+}
+
+// Put stores a record in the database.
+func (ebs *endpointBridgeStorage) Put(r record.Record) (record.Record, error) {
+	if r.DatabaseKey() == "" {
+		return nil, database.ErrNotFound
+	}
+
+	// Prepare data.
+	var ebr *EndpointBridgeRequest
+	if r.IsWrapped() {
+		// Only allocate a new struct, if we need it.
+		ebr = &EndpointBridgeRequest{}
+		err := record.Unwrap(r, ebr)
+		if err != nil {
+			return nil, err
+		}
+	} else {
+		var ok bool
+		ebr, ok = r.(*EndpointBridgeRequest)
+		if !ok {
+			return nil, fmt.Errorf("record not of type *EndpointBridgeRequest, but %T", r)
+		}
+	}
+
+	// Override path with key to mitigate sneaky stuff.
+	ebr.Path = r.DatabaseKey()
+	return callAPI(ebr)
+}
+
+// ReadOnly returns whether the database is read only.
+func (ebs *endpointBridgeStorage) ReadOnly() bool {
+	return false
+}
+
+func callAPI(ebr *EndpointBridgeRequest) (record.Record, error) {
+	// Add API prefix to path.
+	requestURL := path.Join(apiV1Path, ebr.Path)
+	// Check if path is correct. (Defense in depth)
+	if !strings.HasPrefix(requestURL, apiV1Path) {
+		return nil, fmt.Errorf("bridged request for %q violates scope", ebr.Path)
+	}
+
+	// Apply default Method.
+	if ebr.Method == "" {
+		if len(ebr.Data) > 0 {
+			ebr.Method = http.MethodPost
+		} else {
+			ebr.Method = http.MethodGet
+		}
+	}
+
+	// Build URL.
+	u, err := url.ParseRequestURI(requestURL)
+	if err != nil {
+		return nil, fmt.Errorf("failed to build bridged request url: %w", err)
+	}
+	// Build query values.
+	if ebr.Query != nil && len(ebr.Query) > 0 {
+		query := url.Values{}
+		for k, v := range ebr.Query {
+			query.Set(k, v)
+		}
+		u.RawQuery = query.Encode()
+	}
+
+	// Create request and response objects.
+	r := httptest.NewRequest(ebr.Method, u.String(), bytes.NewBuffer(ebr.Data))
+	r.RemoteAddr = endpointBridgeRemoteAddress
+	if ebr.MimeType != "" {
+		r.Header.Set("Content-Type", ebr.MimeType)
+	}
+	w := httptest.NewRecorder()
+	// Let the API handle the request.
+	server.Handler.ServeHTTP(w, r)
+	switch w.Code {
+	case 200:
+		// Everything okay, continue.
+	case 500:
+		// A Go error was returned internally.
+		// We can safely return this as an error.
+		return nil, fmt.Errorf("bridged api call failed: %s", w.Body.String())
+	default:
+		return nil, fmt.Errorf("bridged api call returned unexpected error code %d", w.Code)
+	}
+
+	response := &EndpointBridgeResponse{
+		MimeType: w.Header().Get("Content-Type"),
+		Body:     w.Body.String(),
+	}
+	response.SetKey(apiDatabaseName + ":" + ebr.Path)
+	response.UpdateMeta()
+
+	return response, nil
+}

base/api/auth_wrapper.go

@@ -0,0 +1,30 @@
+package api
+
+import "net/http"
+
+// WrapInAuthHandler wraps a simple http.HandlerFunc into a handler that
+// exposes the required API permissions for this handler.
+func WrapInAuthHandler(fn http.HandlerFunc, read, write Permission) http.Handler {
+	return &wrappedAuthenticatedHandler{
+		HandlerFunc: fn,
+		read:        read,
+		write:       write,
+	}
+}
+
+type wrappedAuthenticatedHandler struct {
+	http.HandlerFunc
+
+	read  Permission
+	write Permission
+}
+
+// ReadPermission returns the read permission for the handler.
+func (wah *wrappedAuthenticatedHandler) ReadPermission(r *http.Request) Permission {
+	return wah.read
+}
+
+// WritePermission returns the write permission for the handler.
+func (wah *wrappedAuthenticatedHandler) WritePermission(r *http.Request) Permission {
+	return wah.write
+}