Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructure modules #1572

Merged
merged 58 commits into from
Aug 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
a3df8e2
Move portbase into monorepo
dhaavi Jun 5, 2024
3533f81
Add new simple module mgr
dhaavi Jun 5, 2024
4f90afe
[WIP] Switch to new simple module mgr
dhaavi Jun 5, 2024
311536c
Add StateMgr and more worker variants
dhaavi Jun 6, 2024
469aef8
[WIP] Switch more modules
dhaavi Jun 6, 2024
2831227
[WIP] Switch more modules
vlabo Jun 7, 2024
b75ef77
[WIP] swtich more modules
vlabo Jun 11, 2024
14991ae
[WIP] switch all SPN modules
vlabo Jun 13, 2024
4b4ff4a
[WIP] switch all service modules
vlabo Jun 17, 2024
49e98fe
[WIP] Convert all workers to the new module system
vlabo Jun 20, 2024
89e4399
[WIP] add new task system to module manager
vlabo Jun 24, 2024
7880b13
[WIP] Add second take for scheduling workers
dhaavi Jun 24, 2024
f9eeae0
[WIP] Add FIXME for bugs in new scheduler
dhaavi Jun 24, 2024
91f3c70
[WIP] Add minor improvements to scheduler
vlabo Jun 24, 2024
f6cb6b4
[WIP] Add new worker scheduler
vlabo Jun 25, 2024
3c9b636
[WIP] Fix more bug related to new module system
vlabo Jun 25, 2024
4d9b908
[WIP] Fix start handing of the new module system
vlabo Jun 27, 2024
b9edb7e
[WIP] Improve startup process
vlabo Jul 2, 2024
594e53d
[WIP] Fix minor issues
vlabo Jul 3, 2024
8c3109a
[WIP] Fix missing subsystem in settings
vlabo Jul 3, 2024
286a0d3
[WIP] Initialize managers in constructor
dhaavi Jul 5, 2024
6d83596
[WIP] Move module event initialization to constrictors
vlabo Jul 8, 2024
e9ae583
[WIP] Fix setting for enabling and disabling the SPN module
vlabo Jul 9, 2024
bb9c645
[WIP] Move API registeration into module construction
vlabo Jul 9, 2024
0bcdd5c
[WIP] Update states mgr for all modules
vlabo Jul 11, 2024
b60c736
[WIP] Add CmdLine operation support
vlabo Jul 11, 2024
482f2ff
Add state helper methods to module group and instance
dhaavi Jul 16, 2024
d142536
Add notification and module status handling to status package
dhaavi Jul 16, 2024
51ec5cd
Fix starting issues
dhaavi Jul 16, 2024
8d8ce0b
Remove pilot widget and update security lock to new status data
dhaavi Jul 16, 2024
e55965a
Remove debug logs
dhaavi Jul 16, 2024
9b54f88
Improve http server shutdown
dhaavi Jul 16, 2024
84936ed
Add workaround for cleanly shutting down firewall+netquery
dhaavi Jul 16, 2024
b93ed09
Improve logging
dhaavi Jul 16, 2024
d2c6ab5
Add syncing states with notifications for new module system
dhaavi Jul 18, 2024
5f3147e
Improve starting, stopping, shutdown; resolve FIXMEs/TODOs
dhaavi Jul 18, 2024
f51c559
[WIP] Fix most unit tests
vlabo Jul 25, 2024
ad64e04
Review new module system and fix minor issues
dhaavi Jul 26, 2024
2831d97
Merge concurrent changes
dhaavi Jul 26, 2024
4980f33
Push shutdown and restart events again via API
dhaavi Jul 26, 2024
2598455
Set sleep mode via interface
dhaavi Jul 26, 2024
a3703c2
Update example/template module
dhaavi Jul 26, 2024
ecd2186
[WIP] Fix spn/cabin unit test
vlabo Jul 30, 2024
018e640
Remove deprecated UI elements
dhaavi Jul 30, 2024
491d9ea
Make log output more similar for the logging transition phase
dhaavi Jul 30, 2024
a286c2c
Switch spn hub and observer cmds to new module system
dhaavi Jul 30, 2024
696481a
Fix log sources
dhaavi Jul 30, 2024
b1db2e9
Make worker mgr less error prone
dhaavi Jul 30, 2024
4f76f43
Fix tests and minor issues
dhaavi Aug 2, 2024
33f6030
Fix observation hub
dhaavi Aug 2, 2024
b55c986
Improve shutdown and restart handling
dhaavi Aug 2, 2024
52ea77e
Split up big connection.go source file
dhaavi Aug 2, 2024
0911bd2
Move varint and dsd packages to structures repo
dhaavi Aug 9, 2024
a5cf25f
Improve expansion test
dhaavi Aug 9, 2024
d45e0b9
Merge branch 'develop' into feature/simplified-modules
dhaavi Aug 9, 2024
06862b7
Fix linter warnings
dhaavi Aug 9, 2024
45b50fe
Fix interception module on windows
dhaavi Aug 9, 2024
b2b51ff
Fix linter errors
vlabo Aug 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion Earthfile
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ go-build:
ENV EXTRA_LD_FLAGS = ""
END

RUN --no-cache go build -ldflags="-X github.com/safing/portbase/info.version=${VERSION} -X github.com/safing/portbase/info.buildSource=${SOURCE} -X github.com/safing/portbase/info.buildTime=${BUILD_TIME} ${EXTRA_LD_FLAGS}" -o "/tmp/build/" ./cmds/${bin}
RUN --no-cache go build -ldflags="-X github.com/safing/portmaster/base/info.version=${VERSION} -X github.com/safing/portmaster/base/info.buildSource=${SOURCE} -X github.com/safing/portmaster/base/info.buildTime=${BUILD_TIME}" -o "/tmp/build/" ./cmds/${bin}
END

DO +GO_ARCH_STRING --goos="${GOOS}" --goarch="${GOARCH}" --goarm="${GOARM}"
Expand Down
2 changes: 1 addition & 1 deletion assets/icons_default.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import (

"golang.org/x/image/draw"

"github.com/safing/portbase/log"
"github.com/safing/portmaster/base/log"
)

// Colored Icon IDs.
Expand Down
8 changes: 8 additions & 0 deletions base/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
portbase
apitest
misc

go.mod.*
vendor
go.work
go.work.sum
157 changes: 157 additions & 0 deletions base/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
> **Check out our main project at [safing/portmaster](https://github.com/safing/portmaster)**

# Portbase

Portbase helps you quickly take off with your project. It gives you all the basic needs you would have for a service (_not_ tool!).
Here is what is included:

- `log`: really fast and beautiful logging
- `modules`: a multi stage, dependency aware boot process for your software, also manages tasks
- `config`: simple, live updating and extremely fast configuration storage
- `info`: easily tag your builds with versions, commit hashes, and so on
- `formats`: some handy data encoding libs
- `rng`: a feedable CSPRNG for great randomness
- `database`: intelligent and syncable database with hooks and easy integration with structs, uses buckets with different backends
- `api`: a websocket interface to the database, can be extended with custom http handlers

Before you continue, a word about this project. It was created to hold the base code for both Portmaster and Gate17. This is also what it will be developed for. If you have a great idea on how to improve portbase, please, by all means, raise an issue and tell us about it, but please also don't be surprised or offended if we ask you to create your own fork to do what you need. Portbase isn't for everyone, it's quite specific to our needs, but we decided to make it easily available to others.

Portbase is actively maintained, please raise issues.

## log

The main goal of this logging package is to be as fast as possible. Logs are sent to a channel only with minimal processing beforehand, so that the service can continue with the important work and write the logs later.

Second, is beauty, both in form what information is provided and how.

You can use flags to change the log level on a source file basis.

## modules <small>requires `log`</small>

packages may register themselves as modules, to take part in the multi stage boot and coordinated shutdown.

Registering only requires a name/key and the `prep()`, `start()` and `stop()` functions.

This is how modules are booted:

- `init()` available: ~~flags~~, ~~config~~, ~~logging~~, ~~dependencies~~
- register flags (with the stdlib `flag` library)
- register module
- `module.prep()` available: flags, ~~config~~, ~~logging~~, ~~dependencies~~
- react to flags
- register config variables
- if an error occurs, return it
- return ErrCleanExit for a clean, successful exit. (eg. you only printed a version)
- `module.start()` available: flags, config, logging, dependencies
- start tasks and workers
- do not log errors while starting, but return them
- `module.stop()` available: flags, config, logging, dependencies
- stop all work (ie. goroutines)
- do not log errors while stopping, but return them

You can start tasks and workers from your module that are then integrated into the module system and will allow for insights and better control of them in the future.

## config <small>requires `log`</small>

The config package stores the configuration in json strings. This may sound a bit weird, but it's very practical.

There are three layers of configuration - in order of priority: user configuration, default configuration and the fallback values supplied when registering a config variable.

When using config variables, you get a function that checks if your config variable is still up to date every time. If it did not change, it's _extremely_ fast. But if it, it will fetch the current value, which takes a short while, but does not happen often.

// This is how you would get a string config variable function.
myVar := GetAsString("my_config_var", "default")
// You then use myVar() directly every time, except when you must guarantee the same value between two calls
if myVar() != "default" {
log.Infof("my_config_var is set to %s", myVar())
}
// no error handling needed! :)

WARNING: While these config variable functions are _extremely_ fast, they are _NOT_ thread/goroutine safe! (Use the `Concurrent` wrapper for that!)

## info

Info provides a easy way to store your version and build information within the binary. If you use the `build` script to build the program, it will automatically set build information so that you can easily find out when and from which commit a binary was built.

The `build` script extracts information from the host and the git repo and then calls `go build` with some additional arguments.

## formats/varint

This is just a convenience wrapper around `encoding/binary`, because we use varints a lot.

## formats/dsd <small>requires `formats/varint`</small>

DSD stands for dynamically structured data. In short, this a generic packer that reacts to the supplied data type.

- structs are usually json encoded
- []bytes and strings stay the same

This makes it easier / more efficient to store different data types in a k/v data storage.

## rng <small>requires `log`, `config`</small>

This package provides a CSPRNG based on the [Fortuna](https://en.wikipedia.org/wiki/Fortuna_(PRNG)) CSPRNG, devised by Bruce Schneier and Niels Ferguson. Implemented by Jochen Voss, published [on Github](https://github.com/seehuhn/fortuna).

Only the Generator is used from the `fortuna` package. The feeding system implemented here is configurable and is focused with efficiency in mind.

While you can feed the RNG yourself, it has two feeders by default:
- It starts with a seed from `crypto/rand` and periodically reseeds from there
- A really simple tickfeeder which extracts entropy from the internal go scheduler using goroutines and is meant to be used under load.

## database <small>requires `log`</small>
_introduction to be written_

## api <small>requires `log`, `database`, `config`</small>
_introduction to be written_

## The main program

If you build everything with modules, your main program should be similar to this - just use an empty import for the modules you need:

import (
"os"
"os/signal"
"syscall"

"github.com/safing/portmaster/base/info"
"github.com/safing/portmaster/base/log"
"github.com/safing/portmaster/base/modules"

// include packages here
_ "path/to/my/custom/module"
)

func main() {

// Set Info
info.Set("MySoftware", "1.0.0")

// Start
err := modules.Start()
if err != nil {
if err == modules.ErrCleanExit {
os.Exit(0)
} else {
os.Exit(1)
}
}

// Shutdown
// catch interrupt for clean shutdown
signalCh := make(chan os.Signal)
signal.Notify(
signalCh,
os.Interrupt,
syscall.SIGHUP,
syscall.SIGINT,
syscall.SIGTERM,
syscall.SIGQUIT,
)
select {
case <-signalCh:
log.Warning("main: program was interrupted")
modules.Shutdown()
case <-modules.ShuttingDown():
}

}
173 changes: 173 additions & 0 deletions base/api/api_bridge.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
package api

import (
"bytes"
"fmt"
"net/http"
"net/http/httptest"
"net/url"
"path"
"strings"
"sync"

"github.com/safing/portmaster/base/database"
"github.com/safing/portmaster/base/database/record"
"github.com/safing/portmaster/base/database/storage"
)

const (
endpointBridgeRemoteAddress = "websocket-bridge"
apiDatabaseName = "api"
)

func registerEndpointBridgeDB() error {
if _, err := database.Register(&database.Database{
Name: apiDatabaseName,
Description: "API Bridge",
StorageType: "injected",
}); err != nil {
return err
}

_, err := database.InjectDatabase("api", &endpointBridgeStorage{})
return err
}

type endpointBridgeStorage struct {
storage.InjectBase
}

// EndpointBridgeRequest holds a bridged request API request.
type EndpointBridgeRequest struct {
record.Base
sync.Mutex

Method string
Path string
Query map[string]string
Data []byte
MimeType string
}

// EndpointBridgeResponse holds a bridged request API response.
type EndpointBridgeResponse struct {
record.Base
sync.Mutex

MimeType string
Body string
}

// Get returns a database record.
func (ebs *endpointBridgeStorage) Get(key string) (record.Record, error) {
if key == "" {
return nil, database.ErrNotFound
}

return callAPI(&EndpointBridgeRequest{
Method: http.MethodGet,
Path: key,
})
}

// Get returns the metadata of a database record.
func (ebs *endpointBridgeStorage) GetMeta(key string) (*record.Meta, error) {
// This interface is an API, always return a fresh copy.
m := &record.Meta{}
m.Update()
return m, nil
}

// Put stores a record in the database.
func (ebs *endpointBridgeStorage) Put(r record.Record) (record.Record, error) {
if r.DatabaseKey() == "" {
return nil, database.ErrNotFound
}

// Prepare data.
var ebr *EndpointBridgeRequest
if r.IsWrapped() {
// Only allocate a new struct, if we need it.
ebr = &EndpointBridgeRequest{}
err := record.Unwrap(r, ebr)
if err != nil {
return nil, err
}
} else {
var ok bool
ebr, ok = r.(*EndpointBridgeRequest)
if !ok {
return nil, fmt.Errorf("record not of type *EndpointBridgeRequest, but %T", r)
}
}

// Override path with key to mitigate sneaky stuff.
ebr.Path = r.DatabaseKey()
return callAPI(ebr)
}

// ReadOnly returns whether the database is read only.
func (ebs *endpointBridgeStorage) ReadOnly() bool {
return false
}

func callAPI(ebr *EndpointBridgeRequest) (record.Record, error) {
// Add API prefix to path.
requestURL := path.Join(apiV1Path, ebr.Path)
// Check if path is correct. (Defense in depth)
if !strings.HasPrefix(requestURL, apiV1Path) {
return nil, fmt.Errorf("bridged request for %q violates scope", ebr.Path)
}

// Apply default Method.
if ebr.Method == "" {
if len(ebr.Data) > 0 {
ebr.Method = http.MethodPost
} else {
ebr.Method = http.MethodGet
}
}

// Build URL.
u, err := url.ParseRequestURI(requestURL)
if err != nil {
return nil, fmt.Errorf("failed to build bridged request url: %w", err)
}
// Build query values.
if ebr.Query != nil && len(ebr.Query) > 0 {
query := url.Values{}
for k, v := range ebr.Query {
query.Set(k, v)
}
u.RawQuery = query.Encode()
}

// Create request and response objects.
r := httptest.NewRequest(ebr.Method, u.String(), bytes.NewBuffer(ebr.Data))
r.RemoteAddr = endpointBridgeRemoteAddress
if ebr.MimeType != "" {
r.Header.Set("Content-Type", ebr.MimeType)
}
w := httptest.NewRecorder()
// Let the API handle the request.
server.Handler.ServeHTTP(w, r)
switch w.Code {
case 200:
// Everything okay, continue.
case 500:
// A Go error was returned internally.
// We can safely return this as an error.
return nil, fmt.Errorf("bridged api call failed: %s", w.Body.String())
default:
return nil, fmt.Errorf("bridged api call returned unexpected error code %d", w.Code)
}

response := &EndpointBridgeResponse{
MimeType: w.Header().Get("Content-Type"),
Body: w.Body.String(),
}
response.SetKey(apiDatabaseName + ":" + ebr.Path)
response.UpdateMeta()

return response, nil
}
30 changes: 30 additions & 0 deletions base/api/auth_wrapper.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
package api

import "net/http"

// WrapInAuthHandler wraps a simple http.HandlerFunc into a handler that
// exposes the required API permissions for this handler.
func WrapInAuthHandler(fn http.HandlerFunc, read, write Permission) http.Handler {
return &wrappedAuthenticatedHandler{
HandlerFunc: fn,
read: read,
write: write,
}
}

type wrappedAuthenticatedHandler struct {
http.HandlerFunc

read Permission
write Permission
}

// ReadPermission returns the read permission for the handler.
func (wah *wrappedAuthenticatedHandler) ReadPermission(r *http.Request) Permission {
return wah.read
}

// WritePermission returns the write permission for the handler.
func (wah *wrappedAuthenticatedHandler) WritePermission(r *http.Request) Permission {
return wah.write
}
Loading
Loading