The core Trillian service provides a Merkle tree store (and primitives to interact with it), but a full transparency application requires additional functionality on top of that base, which needs to be provided by a separate personality.
The following sections describe the conceptual responsibilities of a Trillian personality, followed by the practical responsibilities that are likely to be needed for real-world use. Overall design considerations for transparent Log applications are also discussed in a separate document.
The primary purpose of a personality is to implement admission criteria for the store, so that only particular types of data are added to the store. For example, a Certificate Transparency Log only accepts data items that are valid certificates; a "CT Log" personality would police this, so that the Trillian service can process all incoming data blindly.
It is common for these admission criteria to be broken into two distinct but related areas:
- Structure: Submissions to the personality have to be in a well known format, thus limiting the transparent data to a particular field of interest. (For example, for CT the structure must be a DER-encoded X.509 certificate.)
- Authentication: Submissions to the personality may have to be signed by a known authority, in which case the personality must be configured with a set of acceptable authorities. (For CT, each submission must chain to one of a set of accepted web PKI root certificates.)
A personality may also perform canonicalization on incoming data, to convert equivalent formulations of the same underlying data to a single canonical format, avoiding needless duplication. For example, keys in JSON dictionaries could be sorted, or Unicode string data could be normalised.
One particularly important form of canonicalization is de-duplication of timestamped data: if the Merkle leaf contents includes a timestamp indicating when the leaf was added, subsequent submissions of the same data with a new timestamp should not be treated as a new leaf.
This use case is supported by the Trillian API, which includes the concept of an identity hash associated with a data item: items with the same identity hash are assumed equal, even if the full Merkle leaf structure differs (e.g. because it has a different timestamp attached).
This means that a personality that wants to prevent duplicate entries can
do so by setting the LeafIdentityHash value on new Merkle tree leaves
appropriately. The personality code also needs to cope with duplicates: the
Merkle tree leaf in a successful QueueLeaves response may not be the same
as the tree leaf in the corresponding QueueLeaves request (it will have the
same LeafIdentityHash but may differ in fields that are not covered by this
hash, e.g. it may have an earlier timestamp).
In many environments the personality provides an externally-visible API. The design of such an API depends on the application, but key points to consider include:
-
syntax – the mechanism used to access the API, where possibilities include:
- HTTPS + JSON: A web-based interface has the advantage of being
accessible using standard tools, ranging from
standard
libraries to
command-line tools like
curlandwget. (The Certificate Transparency API described in RFC 6962 takes this approach.) - gRPC: An RPC-based interface allows for additional tooling (such as mechanical checks of schema adherence) and may allow features that are not easily achieved in a RESTful web API (such as continuous streaming of data).
- HTTPS + JSON: A web-based interface has the advantage of being
accessible using standard tools, ranging from
standard
libraries to
command-line tools like
-
semantics – the set of primitives and data structures exposed by the API, for example:
- What submission formats are accepted?
- What cryptographic promises/guarantees does the personality generate?
- Which Merkle tree primitives are exposed to external users?
A Trillian personality may need an additional storage mechanism over and above the Merkle tree storage provided by Trillian. For example, if Trillian is run by a separate operator, then audit verification data may need to be stored.
However, note that the Trillian APIs allow a personality to store additional
data that is associated with a leaf in the underlying Merkle tree, in the
ExtraData field. This extra data is immutable but is not covered by the
Merkle tree hashing algorithms, and so cannot be used for
verifiable/cryptographic purposes (as it could be modified without affecting
tree hash values).
The core Trillian service can be configured to provide traffic control limits using its quota system; out of the box, this can be governed by:
- global rate limits (for both read and write operations)
- per-Merkle tree rate limits (for read and write).
A particular application may also want to provide other kinds of limits. This
is supported by the per-user quota limits, where the user is entirely
specified by the Trillian personality, using the ChargeTo field in each
Trillian API request.
A typical use case for this is to add a ChargeTo.User string that identifies
an external requestor, so traffic from a single source can be limited without
affecting other users.
However, the user quota system can also be used for more flexible limits – for example, by applying limits to particular authentication keys.
A personality that is in real-world use should probably be monitored, for performance, alerting and debugging. Trillian uses a metrics interface to support this, which can also be re-used in an associated personality.
This section describes two Trillian personalities, together with their choices for the various design decisions described above.
Certificate Transparency (CT) is the original transparency application, and has been successful in improving the security of the web PKI ecosystem. It is also the most mature and widely deployed Trillian use case, built on the CTFE personality.
The conceptual responsibilities of the CTFE personality are given by:
- Admission Control: Submissions have to take the form of a chain of valid X.509 certificates, each certificate signed by the next certificate in the chain, until a acceptable root certificate is reached. The set of accepted roots is configured in the personality at startup.
- Canonicalization: Submissions have a
LeafIdentityHashwhich covers the leaf certificate, but also have a timestamp generated by the CTFE. A re-submission of a previously-seen certificate will return the original leaf with its earlier timestamp. - Auditability: The CTFE personality assumes that the Trillian service is run by the same operator, so does not store personality-to-Trillian audit data.
In terms of practical responsibilities:
- The external API to the CTFE personality is the HTTP(S)+JSON interface specified by RFC 6962, in both syntax and semantics. The CTFE personality is configured with private key material for each CT Log instance, and uses this to generate those API data structures that are signed – the Signed Tree Head (STH) and Signed Certificate Timestamp (SCT).
- The CTFE personality does not require storage independent of the core Merkle
tree. The chain of certificates used to validate the original submission of
a leaf certificate are stored as the
ExtraDataassociated with a Merkle tree leaf. - The CTFE personality can be configured to charge API requests to per-user
quota for:
- The IP address of the original requestor.
- The set of intermediate and root certificates used to validate a submission.
- Metrics are exposed using Trillian's metrics library, and sample Prometheus consoles and scripts are provided.
Users of individual transparent Logs are theoretically susceptible to a split-view attack, where a Log prevents a different view of its Merkle tree to different users – each user's view of the tree is internally consistent, but differs between users.
A gossip protocol is a key mechanism to prevent this: different observers of a transparent Log share their views of the Log, allowing cross-comparison and eventual detection of split views.
The Gossip Hub is an experimental implementation of a repository for information about a Log's Merkle tree, in the form of (another) transparent Log. This allows cross-comparison of the source Log's Merkle tree: different observers can submit their view of the source Log, and retrieve other observers' views of the same source Log.
(Aside: in practice, split-view attacks are less of a concern for Certificate Transparency and the web PKI because browser policies typically require individual certificates to be logged in multiple independent CT Logs, limiting the scope of any individual Log's split-view attack.)
The conceptual responsibilities of the Gossip Hub personality are given by:
- Admission Control: Submissions take the form of signed blobs of data, and the data may be required to take the form of a valid RFC 6962 signed tree head structure. The set of accepted public keys from the source Logs, and the expected submission structure, is configured in the personality at start-of-day.
- Canonicalization: Submissions have a
LeafIdentityHashwhich covers the submitted data and signature, but also have a timestamp generated by the Hub. A re-submission of a previously-seen entry will return the original leaf with its earlier timestamp. - Auditability: The Hub personality assumes that the Trillian service is run by the same operator, so does not store personality-to-Trillian audit data.
In terms of practical responsibilities:
- The external API to the Gossip Hub personality is an HTTP(S)+JSON interface analogous to that specified by RFC 6962, in both syntax and semantics. The Hub personality is configured with private key material for each Hub instance, and uses this to generate signed API data structures – the Signed Tree Head (STH) and Signed Gossip Timestamp (SGT).
- The Hub personality does not require storage independent of the core Merkle
tree, and the
ExtraDatafield is not (currently) used. - The Hub personality can be configured to charge API requests to per-user
quota for:
- The IP address of the original requestor.
- The public key used to sign a submission.
- Metrics are exposed using Trillian's metrics library.