Define base encoding in x509.serial_number #2383

haesbaert · 2024-09-17T20:19:20Z

This proposes enforcing the base encoding of x509.serial_number to be 16.

The current definition is too loose and leaves room for interpretation, base 16 is also more common and it will help users correlate the value with existing tools. This change was prompted by elastic/sdh-beats#5089 (internal link) where a user expected to see it in the same format as other tools.

This narrows the definition of x509.serial_number to be encoded in hexadecimal, otherwise we end up with integrations choosing their own encoding, as noted below, Zeek uses base 16 while the rest of beats is using base 10.

Particularly relevant comment from @andrewkroh (on an internal issue):

I think tls.client.x509.serial_number and tls.server.x509.serial_number would better serve our users if they were hex-encoded. It would provide better interoperability with other tools. I found one example in the Zeek integration that is already using hex (there may be other integrations).

The ECS description is not as "strong" as I would prefer, I think its original intent was for uppercase hex when it said "if this value is alphanumeric, it should be formatted without colons and uppercase characters." So I'm of the opinion we should treat this as a bug.

The source of these values should be in the packetbeat/protos/tls Go code.

The tls.detailed.* description explicitly says it is base-10 so I don't see a reason to change those.

Information Gathering / Research

Field Definitions

name type external source description

tls.client.x509.serial_number keyword ecs data_stream/tls/fields/ecs.yml:137:3 Unique serial number issued by the certificate authority. For consistency, if this value is alphanumeric, it should be formatted without colons and uppercase characters.

tls.detailed.client_certificate_chain.issuer.serial_number keyword data_stream/tls/fields/protocol.yml:188:19

tls.detailed.client_certificate_chain.serial_number keyword data_stream/tls/fields/protocol.yml:210:15 Base 10 representation of the certificate serial number.

tls.detailed.client_certificate_chain.subject.serial_number keyword data_stream/tls/fields/protocol.yml:188:19

tls.detailed.server_certificate_chain.issuer.serial_number keyword data_stream/tls/fields/protocol.yml:188:19

tls.detailed.server_certificate_chain.serial_number keyword data_stream/tls/fields/protocol.yml:210:15 Base 10 representation of the certificate serial number.

tls.detailed.server_certificate_chain.subject.serial_number keyword data_stream/tls/fields/protocol.yml:188:19

tls.server.x509.serial_number keyword ecs data_stream/tls/fields/ecs.yml:213:3 Unique serial number issued by the certificate authority. For consistency, if this value is alphanumeric, it should be formatted without colons and uppercase characters.

x509 (RFC 5280) tells us that serial number must be integers and may contain as many as 20 bytes.

openssl emits hex
$ openssl x509 -in googlecert.pem -noout -serial
serial=244E52D96B551F960A00000000F2BAF4
wireshark shows hex
serialNumber: 0x045d7ad2b40288f1d046fff688c14e0b0bb5
References

https://www.rfc-editor.org/rfc/rfc5280#page-19

zeek sample_event.json showing hex

Narrow the definition of x509.serial_number to be encoded in hexadecimal, otherwise we end up with integrations choosing their own encoding, as noted below, Zeek uses base 16 while the rest of beats is using base 10. Related to elastic/sdh-beats#5089. Reasoning in: elastic/sdh-beats#5089 (comment)

github-actions · 2024-09-17T20:19:31Z

Documentation changes preview: https://ecs_bk_2383.docs-preview.app.elstc.co/diff

mjwolf · 2024-09-17T21:25:16Z

schemas/x509.yml

@@ -52,8 +52,8 @@
      type: keyword
      short: Unique serial number issued by the certificate authority.
      description: >
-        Unique serial number issued by the certificate authority. For consistency, if this value is alphanumeric, it should be
-        formatted without colons and uppercase characters.
+        Unique serial number issued by the certificate authority. For consistency, this should be


Given this is should, it implies this encoding is still optional, which might not really completely resolve the original problem. Is that what you intend? But if the wording is changed to something stronger, it could be considered a breaking change.

If you want it to be enforced, what do you think of having should in ECS v8, and changing to must in the next major release?

I agree, that making it a MUST would be breaking for v8 so I think it needs to remain a SHOULD.

About changing it in the next major, sounds like a good idea.

I'm fine with that, I'm not familiar with the process at all.
For the next release, shouldn't we restrict it even more? I'd prefer to have something that is strict on case, semi-colons and base.

andrewkroh · 2024-09-17T21:34:42Z

schemas/x509.yml

+        Unique serial number issued by the certificate authority. For consistency, this should be
+        encoded in base 16 and formatted without colons and uppercase characters.


"without colons and uppercase characters" is ambiguous. Does it mean "without colons and without uppercase"? Or does it mean "without colons and with uppercase"? 😄

Suggested change

Unique serial number issued by the certificate authority. For consistency, this should be

encoded in base 16 and formatted without colons and uppercase characters.

Unique serial number issued by the certificate authority. For consistency, this should be

encoded in base 16 and formatted as uppercase characters without colons.

(edited to leave "base 16" as is)

Ha! Good point, my brain didn't spot that.
I think it should still be base 16, not base-16, as it's consistent with the other cases that use space, as in:
https://github.com/elastic/ecs/blob/main/schemas/dns.yml#L98
grep -r 'base-' doesn't return any other uses.

If all agree, I'd bring thid PR to 8.x with the wording changes from @andrewkroh, and then open a second PR to do the breaking/must changes.

SGTM. If we haven't already created an 8.x branch then we should do that before you merge the second PR.

Agree, I'm just a bit confused, we have branches 8.{11,12,13,14}, when you say 8.x does this mean 8.{12,13,14} ?
Do we change ECS definitions (even if should), for current and older releases (<= 8.11).
In other words, where should I branch from?

This is what other repos like beats/kibana/elasticsearch and doing... We should create an 8.x from main. And then future 8.{16,17,18} branches will come from 8.x. We should backport to 8.x unless the change is a something specifically for 9.0 (like a breaking change).

Got it, in the meantime @mjwolf volunteered to take care of this.

I've created an 8.x branch now. I think the best way to handle this is to merge this PR to main first (and I'll handle the backport to the 8.x branch), and then afterwards create another PR with the changes intended for 9.x only that will stay in main only.

@haesbaert with this PR merged, and backported to 8.x, you can now create another PR with the changes intended for 9.x only. You can target it to main, and a backport-skip label

andrewkroh · 2024-09-17T21:36:58Z

schemas/x509.yml

@@ -52,8 +52,8 @@
      type: keyword
      short: Unique serial number issued by the certificate authority.
      description: >
-        Unique serial number issued by the certificate authority. For consistency, if this value is alphanumeric, it should be
-        formatted without colons and uppercase characters.
+        Unique serial number issued by the certificate authority. For consistency, this should be


I agree, that making it a MUST would be breaking for v8 so I think it needs to remain a SHOULD.

About changing it in the next major, sounds like a good idea.

Narrow the definition of x509.serial_number to be encoded in hexadecimal, otherwise we end up with integrations choosing their own encoding, as noted below, Zeek uses base 16 while the rest of beats is using base 10.

We made 8.x a `should` for the same field in 4fa0abd. As discussed in #2383 (comment) we are making this a `must` for 9.x.

haesbaert requested a review from a team as a code owner September 17, 2024 20:19

CHANGELOG.next.md entry

89f9df5

mjwolf reviewed Sep 17, 2024

View reviewed changes

andrewkroh reviewed Sep 17, 2024

View reviewed changes

Merge branch 'main' into x509.serial_number

8034054

mjwolf added the backport-8.x label Sep 26, 2024

Merge branch 'main' into x509.serial_number

f1f876d

mjwolf approved these changes Oct 15, 2024

View reviewed changes

mjwolf merged commit 4fa0abd into main Oct 15, 2024
4 checks passed

mjwolf deleted the x509.serial_number branch October 15, 2024 20:51

haesbaert added a commit that referenced this pull request Nov 6, 2024

Restrict x509.serial_number to base 16 for 9.x

80a2461

We made 8.x a `should` for the same field in 4fa0abd. As discussed in #2383 (comment) we are making this a `must` for 9.x.

haesbaert mentioned this pull request Nov 6, 2024

Restrict x509.serial_number to base 16 for 9.x #2398

Open

fearful-symmetry mentioned this pull request Nov 6, 2024

Switch to base 16 for TLS serial number in packetbeat in line with ECS changes elastic/beats#41542

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Define base encoding in x509.serial_number #2383

Define base encoding in x509.serial_number #2383

haesbaert commented Sep 17, 2024 •

edited by andrewkroh

Loading

github-actions bot commented Sep 17, 2024

mjwolf Sep 17, 2024

andrewkroh Sep 17, 2024

haesbaert Sep 18, 2024

andrewkroh Sep 17, 2024 •

edited

Loading

haesbaert Sep 18, 2024

haesbaert Sep 18, 2024

andrewkroh Sep 19, 2024

haesbaert Sep 23, 2024

andrewkroh Sep 23, 2024

haesbaert Sep 25, 2024

mjwolf Sep 26, 2024

mjwolf Oct 15, 2024

andrewkroh Sep 17, 2024

name	type	external	source	description
tls.client.x509.serial_number	keyword	ecs	data_stream/tls/fields/ecs.yml:137:3	Unique serial number issued by the certificate authority. For consistency, if this value is alphanumeric, it should be formatted without colons and uppercase characters.
tls.detailed.client_certificate_chain.issuer.serial_number	keyword		data_stream/tls/fields/protocol.yml:188:19
tls.detailed.client_certificate_chain.serial_number	keyword		data_stream/tls/fields/protocol.yml:210:15	Base 10 representation of the certificate serial number.
tls.detailed.client_certificate_chain.subject.serial_number	keyword		data_stream/tls/fields/protocol.yml:188:19
tls.detailed.server_certificate_chain.issuer.serial_number	keyword		data_stream/tls/fields/protocol.yml:188:19
tls.detailed.server_certificate_chain.serial_number	keyword		data_stream/tls/fields/protocol.yml:210:15	Base 10 representation of the certificate serial number.
tls.detailed.server_certificate_chain.subject.serial_number	keyword		data_stream/tls/fields/protocol.yml:188:19
tls.server.x509.serial_number	keyword	ecs	data_stream/tls/fields/ecs.yml:213:3	Unique serial number issued by the certificate authority. For consistency, if this value is alphanumeric, it should be formatted without colons and uppercase characters.

		Unique serial number issued by the certificate authority. For consistency, this should be
		encoded in base 16 and formatted without colons and uppercase characters.

Define base encoding in x509.serial_number #2383

Define base encoding in x509.serial_number #2383

Conversation

haesbaert commented Sep 17, 2024 • edited by andrewkroh Loading

Information Gathering / Research

github-actions bot commented Sep 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh Sep 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haesbaert commented Sep 17, 2024 •

edited by andrewkroh

Loading

andrewkroh Sep 17, 2024 •

edited

Loading