Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add example value composite type checking #966

Merged
merged 8 commits into from
Sep 23, 2020
Merged

Add example value composite type checking #966

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
0d7a9e3
add example field value checking
ebeahan Sep 9, 2020
ba5075b
add unit testing
ebeahan Sep 9, 2020
821319f
update docs
ebeahan Sep 9, 2020
752b6f6
quote all arrays in the schema spec
ebeahan Sep 9, 2020
5bdd8a9
generate artifacts
ebeahan Sep 9, 2020
6083258
changelog
ebeahan Sep 9, 2020
f3d7b51
double quote each array element
ebeahan Sep 23, 2020
95c22ab
generate artifacts
ebeahan Sep 23, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions USAGE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,7 @@ $ python/generator.py --strict
Strict mode requires the following conditions, else the script exits on an exception:

* Short descriptions must be less than or equal to 120 characters.
* Example values containing arrays or objects must be quoted to avoid unexpected YAML interpretation when the schema files or artifacts are relied on downstream.

The current artifacts generated and published in the ECS repo will always be created using strict mode. However, older ECS versions (pre `v1.5.0`) will cause
an exception if attempting to generate them using `--strict`. This is due to schema validation checks introduced after that version was released.
Expand Down
14 changes: 7 additions & 7 deletions docs/field-details.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1211,7 +1211,7 @@ Note: this field should contain an array of values.



example: `['RD', 'RA']`
example: `["RD", "RA"]`

| extended

Expand Down Expand Up @@ -1343,7 +1343,7 @@ Note: this field should contain an array of values.



example: `['10.10.10.10', '10.10.10.11']`
example: `[10.10.10.10, 10.10.10.11]`

| extended

Expand Down Expand Up @@ -4205,7 +4205,7 @@ Note: this field should contain an array of values.



example: `['/usr/bin/ssh', '-l', 'user', '10.0.0.16']`
example: `["/usr/bin/ssh", "-l", "user", "10.0.0.16"]`

| extended

Expand Down Expand Up @@ -4718,7 +4718,7 @@ Note: this field should contain an array of values.



example: `['Star-Lord']`
example: `["Star-Lord"]`

| extended

Expand Down Expand Up @@ -5624,7 +5624,7 @@ Note: this field should contain an array of values.



example: `['MII...', 'MII...']`
example: `["MII...", "MII..."]`

| extended

Expand Down Expand Up @@ -5757,7 +5757,7 @@ Note: this field should contain an array of values.



example: `['TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384', 'TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384', '...']`
example: `["TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "..."]`

| extended

Expand Down Expand Up @@ -5838,7 +5838,7 @@ Note: this field should contain an array of values.



example: `['MII...', 'MII...']`
example: `["MII...", "MII..."]`

| extended

Expand Down
37 changes: 9 additions & 28 deletions generated/beats/fields.ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1007,9 +1007,7 @@
description: 'Array of 2 letter DNS header flags.

Expected values are: AA, TC, RD, RA, AD, CD, DO.'
example:
- RD
- RA
example: '["RD", "RA"]'
- name: id
level: extended
type: keyword
Expand Down Expand Up @@ -1090,9 +1088,7 @@
formats it can contain. Extracting all IP addresses seen in there to `dns.resolved_ip`
makes it possible to index them as IP addresses, and makes them easier to
visualize and query for.'
example:
- 10.10.10.10
- 10.10.10.11
example: '[10.10.10.10, 10.10.10.11]'
- name: response_code
level: extended
type: keyword
Expand Down Expand Up @@ -3223,11 +3219,7 @@
the executable.

May be filtered to protect sensitive information.'
example:
- /usr/bin/ssh
- -l
- user
- 10.0.0.16
example: '["/usr/bin/ssh", "-l", "user", "10.0.0.16"]'
- name: args_count
level: extended
type: long
Expand Down Expand Up @@ -3370,11 +3362,7 @@
the executable.

May be filtered to protect sensitive information.'
example:
- /usr/bin/ssh
- -l
- user
- 10.0.0.16
example: '["/usr/bin/ssh", "-l", "user", "10.0.0.16"]'
default_field: false
- name: parent.args_count
level: extended
Expand Down Expand Up @@ -3878,8 +3866,7 @@
ignore_above: 1024
description: Name, organization, or pseudonym of the author or authors who created
the rule used to generate this event.
example:
- Star-Lord
example: '["Star-Lord"]'
default_field: false
- name: category
level: extended
Expand Down Expand Up @@ -4640,9 +4627,7 @@
description: Array of PEM-encoded certificates that make up the certificate
chain offered by the client. This is usually mutually-exclusive of `client.certificate`
since that value should be the first certificate in the chain.
example:
- MII...
- MII...
example: '["MII...", "MII..."]'
default_field: false
- name: client.hash.md5
level: extended
Expand Down Expand Up @@ -4723,10 +4708,8 @@
type: keyword
ignore_above: 1024
description: Array of ciphers offered by the client during the client hello.
example:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- '...'
example: '["TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
"..."]'
default_field: false
- name: client.x509.alternative_names
level: extended
Expand Down Expand Up @@ -4943,9 +4926,7 @@
description: Array of PEM-encoded certificates that make up the certificate
chain offered by the server. This is usually mutually-exclusive of `server.certificate`
since that value should be the first certificate in the chain.
example:
- MII...
- MII...
example: '["MII...", "MII..."]'
default_field: false
- name: server.hash.md5
level: extended
Expand Down
16 changes: 8 additions & 8 deletions generated/csv/fields.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,dns,dns.answers.name,keyword,extended,,www.example.com,The domain name to which this resource record pertains.
2.0.0-dev,true,dns,dns.answers.ttl,long,extended,,180,The time interval in seconds that this resource record may be cached before it should be discarded.
2.0.0-dev,true,dns,dns.answers.type,keyword,extended,,CNAME,The type of data contained in this resource record.
2.0.0-dev,true,dns,dns.header_flags,keyword,extended,array,"['RD', 'RA']",Array of DNS header flags.
2.0.0-dev,true,dns,dns.header_flags,keyword,extended,array,"[""RD"", ""RA""]",Array of DNS header flags.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noting some behavior here - the CSV generator is double quoting string values when the schema specification already contains a quoted value.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ended up being expected CSV behavior for the csv.writer() object 😄 .

2.0.0-dev,true,dns,dns.id,keyword,extended,,62111,The DNS packet identifier assigned by the program that generated the query. The identifier is copied to the response.
2.0.0-dev,true,dns,dns.op_code,keyword,extended,,QUERY,The DNS operation code that specifies the kind of query in the message.
2.0.0-dev,true,dns,dns.question.class,keyword,extended,,IN,The class of records being queried.
Expand All @@ -126,7 +126,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,dns,dns.question.subdomain,keyword,extended,,www,The subdomain of the domain.
2.0.0-dev,true,dns,dns.question.top_level_domain,keyword,extended,,co.uk,"The effective top level domain (com, org, net, co.uk)."
2.0.0-dev,true,dns,dns.question.type,keyword,extended,,AAAA,The type of record being queried.
2.0.0-dev,true,dns,dns.resolved_ip,ip,extended,array,"['10.10.10.10', '10.10.10.11']",Array containing all IPs seen in answers.data
2.0.0-dev,true,dns,dns.resolved_ip,ip,extended,array,"[10.10.10.10, 10.10.10.11]",Array containing all IPs seen in answers.data
2.0.0-dev,true,dns,dns.response_code,keyword,extended,,NOERROR,The DNS response code.
2.0.0-dev,true,dns,dns.type,keyword,extended,,answer,"The type of DNS event captured, query or answer."
2.0.0-dev,true,ecs,ecs.version,keyword,core,,1.0.0,ECS version this event conforms to.
Expand Down Expand Up @@ -362,7 +362,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,package,package.size,long,extended,,62231,Package size in bytes.
2.0.0-dev,true,package,package.type,keyword,extended,,rpm,Package type
2.0.0-dev,true,package,package.version,keyword,extended,,1.12.9,Package version
2.0.0-dev,true,process,process.args,keyword,extended,array,"['/usr/bin/ssh', '-l', 'user', '10.0.0.16']",Array of process arguments.
2.0.0-dev,true,process,process.args,keyword,extended,array,"[""/usr/bin/ssh"", ""-l"", ""user"", ""10.0.0.16""]",Array of process arguments.
2.0.0-dev,true,process,process.args_count,long,extended,,4,Length of the process.args array.
2.0.0-dev,true,process,process.code_signature.exists,boolean,core,,true,Boolean to capture if a signature is present.
2.0.0-dev,true,process,process.code_signature.status,keyword,extended,,ERROR_UNTRUSTED_ROOT,Additional information about the certificate status.
Expand All @@ -381,7 +381,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,process,process.hash.sha512,keyword,extended,,,SHA512 hash.
2.0.0-dev,true,process,process.name,keyword,extended,,ssh,Process name.
2.0.0-dev,true,process,process.name.text,text,extended,,ssh,Process name.
2.0.0-dev,true,process,process.parent.args,keyword,extended,array,"['/usr/bin/ssh', '-l', 'user', '10.0.0.16']",Array of process arguments.
2.0.0-dev,true,process,process.parent.args,keyword,extended,array,"[""/usr/bin/ssh"", ""-l"", ""user"", ""10.0.0.16""]",Array of process arguments.
2.0.0-dev,true,process,process.parent.args_count,long,extended,,4,Length of the process.args array.
2.0.0-dev,true,process,process.parent.code_signature.exists,boolean,core,,true,Boolean to capture if a signature is present.
2.0.0-dev,true,process,process.parent.code_signature.status,keyword,extended,,ERROR_UNTRUSTED_ROOT,Additional information about the certificate status.
Expand Down Expand Up @@ -447,7 +447,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,related,related.hosts,keyword,extended,array,,All the host identifiers seen on your event.
2.0.0-dev,true,related,related.ip,ip,extended,array,,All of the IPs seen on your event.
2.0.0-dev,true,related,related.user,keyword,extended,array,,All the user names seen on your event.
2.0.0-dev,true,rule,rule.author,keyword,extended,array,['Star-Lord'],Rule author
2.0.0-dev,true,rule,rule.author,keyword,extended,array,"[""Star-Lord""]",Rule author
2.0.0-dev,true,rule,rule.category,keyword,extended,,Attempted Information Leak,Rule category
2.0.0-dev,true,rule,rule.description,keyword,extended,,Block requests to public DNS over HTTPS / TLS protocols,Rule description
2.0.0-dev,true,rule,rule.id,keyword,extended,,101,Rule ID
Expand Down Expand Up @@ -547,7 +547,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,threat,threat.technique.subtechnique.reference,keyword,extended,array,https://attack.mitre.org/techniques/T1059/001/,Threat subtechnique URL reference.
2.0.0-dev,true,tls,tls.cipher,keyword,extended,,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,String indicating the cipher used during the current connection.
2.0.0-dev,true,tls,tls.client.certificate,keyword,extended,,MII...,PEM-encoded stand-alone certificate offered by the client.
2.0.0-dev,true,tls,tls.client.certificate_chain,keyword,extended,array,"['MII...', 'MII...']",Array of PEM-encoded certificates that make up the certificate chain offered by the client.
2.0.0-dev,true,tls,tls.client.certificate_chain,keyword,extended,array,"[""MII..."", ""MII...""]",Array of PEM-encoded certificates that make up the certificate chain offered by the client.
2.0.0-dev,true,tls,tls.client.hash.md5,keyword,extended,,0F76C7F2C55BFD7D8E8B8F4BFBF0C9EC,Certificate fingerprint using the MD5 digest of DER-encoded version of certificate offered by the client.
2.0.0-dev,true,tls,tls.client.hash.sha1,keyword,extended,,9E393D93138888D288266C2D915214D1D1CCEB2A,Certificate fingerprint using the SHA1 digest of DER-encoded version of certificate offered by the client.
2.0.0-dev,true,tls,tls.client.hash.sha256,keyword,extended,,0687F666A054EF17A08E2F2162EAB4CBC0D265E1D7875BE74BF3C712CA92DAF0,Certificate fingerprint using the SHA256 digest of DER-encoded version of certificate offered by the client.
Expand All @@ -557,7 +557,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,tls,tls.client.not_before,date,extended,,1970-01-01T00:00:00.000Z,Date/Time indicating when client certificate is first considered valid.
2.0.0-dev,true,tls,tls.client.server_name,keyword,extended,,www.elastic.co,Hostname the client is trying to connect to. Also called the SNI.
2.0.0-dev,true,tls,tls.client.subject,keyword,extended,,"CN=myclient, OU=Documentation Team, DC=example, DC=com",Distinguished name of subject of the x.509 certificate presented by the client.
2.0.0-dev,true,tls,tls.client.supported_ciphers,keyword,extended,array,"['TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384', 'TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384', '...']",Array of ciphers offered by the client during the client hello.
2.0.0-dev,true,tls,tls.client.supported_ciphers,keyword,extended,array,"[""TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"", ""TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"", ""...""]",Array of ciphers offered by the client during the client hello.
2.0.0-dev,true,tls,tls.client.x509.alternative_names,keyword,extended,array,*.elastic.co,List of subject alternative names (SAN).
2.0.0-dev,true,tls,tls.client.x509.issuer.common_name,keyword,extended,array,Example SHA2 High Assurance Server CA,List of common name (CN) of issuing certificate authority.
2.0.0-dev,true,tls,tls.client.x509.issuer.country,keyword,extended,array,US,List of country (C) codes
Expand Down Expand Up @@ -587,7 +587,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
2.0.0-dev,true,tls,tls.next_protocol,keyword,extended,,http/1.1,String indicating the protocol being tunneled.
2.0.0-dev,true,tls,tls.resumed,boolean,extended,,,Boolean flag indicating if this TLS connection was resumed from an existing TLS negotiation.
2.0.0-dev,true,tls,tls.server.certificate,keyword,extended,,MII...,PEM-encoded stand-alone certificate offered by the server.
2.0.0-dev,true,tls,tls.server.certificate_chain,keyword,extended,array,"['MII...', 'MII...']",Array of PEM-encoded certificates that make up the certificate chain offered by the server.
2.0.0-dev,true,tls,tls.server.certificate_chain,keyword,extended,array,"[""MII..."", ""MII...""]",Array of PEM-encoded certificates that make up the certificate chain offered by the server.
2.0.0-dev,true,tls,tls.server.hash.md5,keyword,extended,,0F76C7F2C55BFD7D8E8B8F4BFBF0C9EC,Certificate fingerprint using the MD5 digest of DER-encoded version of certificate offered by the server.
2.0.0-dev,true,tls,tls.server.hash.sha1,keyword,extended,,9E393D93138888D288266C2D915214D1D1CCEB2A,Certificate fingerprint using the SHA1 digest of DER-encoded version of certificate offered by the server.
2.0.0-dev,true,tls,tls.server.hash.sha256,keyword,extended,,0687F666A054EF17A08E2F2162EAB4CBC0D265E1D7875BE74BF3C712CA92DAF0,Certificate fingerprint using the SHA256 digest of DER-encoded version of certificate offered by the server.
Expand Down
37 changes: 9 additions & 28 deletions generated/ecs/ecs_flat.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1384,9 +1384,7 @@ dns.header_flags:
description: 'Array of 2 letter DNS header flags.

Expected values are: AA, TC, RD, RA, AD, CD, DO.'
example:
- RD
- RA
example: '["RD", "RA"]'
flat_name: dns.header_flags
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -1514,9 +1512,7 @@ dns.resolved_ip:
it can contain. Extracting all IP addresses seen in there to `dns.resolved_ip`
makes it possible to index them as IP addresses, and makes them easier to visualize
and query for.'
example:
- 10.10.10.10
- 10.10.10.11
example: '[10.10.10.10, 10.10.10.11]'
flat_name: dns.resolved_ip
level: extended
name: resolved_ip
Expand Down Expand Up @@ -4777,11 +4773,7 @@ process.args:
executable.

May be filtered to protect sensitive information.'
example:
- /usr/bin/ssh
- -l
- user
- 10.0.0.16
example: '["/usr/bin/ssh", "-l", "user", "10.0.0.16"]'
flat_name: process.args
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -5007,11 +4999,7 @@ process.parent.args:
executable.

May be filtered to protect sensitive information.'
example:
- /usr/bin/ssh
- -l
- user
- 10.0.0.16
example: '["/usr/bin/ssh", "-l", "user", "10.0.0.16"]'
flat_name: process.parent.args
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -5778,8 +5766,7 @@ rule.author:
dashed_name: rule-author
description: Name, organization, or pseudonym of the author or authors who created
the rule used to generate this event.
example:
- Star-Lord
example: '["Star-Lord"]'
flat_name: rule.author
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -6998,9 +6985,7 @@ tls.client.certificate_chain:
description: Array of PEM-encoded certificates that make up the certificate chain
offered by the client. This is usually mutually-exclusive of `client.certificate`
since that value should be the first certificate in the chain.
example:
- MII...
- MII...
example: '["MII...", "MII..."]'
flat_name: tls.client.certificate_chain
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -7126,10 +7111,8 @@ tls.client.subject:
tls.client.supported_ciphers:
dashed_name: tls-client-supported-ciphers
description: Array of ciphers offered by the client during the client hello.
example:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- '...'
example: '["TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
"..."]'
flat_name: tls.client.supported_ciphers
ignore_above: 1024
level: extended
Expand Down Expand Up @@ -7508,9 +7491,7 @@ tls.server.certificate_chain:
description: Array of PEM-encoded certificates that make up the certificate chain
offered by the server. This is usually mutually-exclusive of `server.certificate`
since that value should be the first certificate in the chain.
example:
- MII...
- MII...
example: '["MII...", "MII..."]'
flat_name: tls.server.certificate_chain
ignore_above: 1024
level: extended
Expand Down
Loading