Redesign Gorums to use Client Interceptors

[meling]

This would be a fairly large redesign, but if my initial thoughts are correct, we could possibly get rid of the whole code generator. I may be wrong of course.

Here is the godoc:
https://godoc.org/google.golang.org/grpc#UnaryClientInterceptor

Here is a simple example of using a Client Interceptor to measure time:
https://about.sourcegraph.com/go/grpc-in-production/#client-interceptor
Same for a Server Interceptor:
https://about.sourcegraph.com/go/grpc-in-production/#server-interceptor
Here is a Client Interceptor for retry logic:
https://about.sourcegraph.com/go/grpc-in-production/#networks-fail

Here is a youtube video with more details (not only on inceptors):
https://www.youtube.com/watch?v=7FZ6ZyzGex0

Another source of reference examples can be found here:
https://github.com/grpc-ecosystem/go-grpc-middleware

[meling]

After a more thorough examination, I have concluded that interceptors are not appropriate for our use case. What we should have is something more akin to a client-side proxy, that is disconnected from a specific client connection.

[meling]

Here are some more thoughts on the issue of interceptors. gRPC does not support other Interceptor/Invoker types than these:

type StreamClientInterceptor
type StreamServerInterceptor
type UnaryClientInterceptor
type UnaryInvoker
type UnaryServerInterceptor

However, Gorums could add support for new interceptor/invoker types, e.g.:

// QuorumInvoker is called by QuorumClientInterceptor to complete quorum calls.
type QuorumInvoker func(ctx context.Context, method string, req, reply interface{}, cc *ConfigConn, opts ...grpc.CallOption) error

// QuorumClientInterceptor intercepts the execution of a quorum call on the client.
// invoker is the handler to complete the quorum call
// and it is the responsibility of the interceptor to call it.
// This is an EXPERIMENTAL API.
type QuorumClientInterceptor func(ctx context.Context, method string, req, reply interface{}, cc *ConfigConn, invoker QuorumInvoker, opts ...grpc.CallOption) error

type ConfigConn []*grpc.ClientConn

This would allow more flexibility in that the basic quorum call can call out to the user-supplied interceptor for customized quorum calls. Note that this would leave it to the user to call the invoker on each entry in ConfigConn and the quorum function to determine when to return; so we should probably just pass in the Configuration object instead of the ConfigConn, since the interceptor would need access to the QuorumSpec.

PS: The above API is just a first draft idea, and needs to be revised.