[Enhancement] ECS Generator - Ability to Selectively Extend Reusable Mappings

[BenB196]

Summary

I would like to be able to have a way with the ECS generator.py to be able to "selectively" extend reusable mappings.

Motivation:

I recently encountered an issue where I add my own schema, and have it be reusable for user:

  reusable:
    order: 1
    top_level: true
    expected:
      - user

While I intend for these mappings to be added to the user field under the generated schemas client, destination, server, source, and user. They also get added to process and this is where the issue arises and where my enhancement comes in. The process schema reuses user 33 times, most of these don't have any need for my additional mappings that I'm adding, and I'd like to exclude my mappings from the process generation, as the additional mappings put the generated process over 1000 fields.

I looked through the docs and wasn't able to find a way to do this today, so opening this enhancement request.

[BenB196]

A note, my current workaround is to run the generator, and then before committing the newly generated mappings to my local git repo, I discard the changes made to the generated process file. But, ideally, it would be nice to have a way in the config file to prevent this from happening in the first place.

[kgeller]

Hi @BenB196 !

We currently do this by utilizing the --subset filter as a part of the generator.

Check out #1847 to see how we implemented it with the process fields.

[BenB196]

Hi @kgeller, thanks! This is exactly what I was looking for, just didn't think to search by that name, or consider that it could be in a different spot other than the main schema definitions.

A quick follow question (and I can open as a new issue or take to discussion forums if wanted), is there a way to force the generator to fail if it generates a component template over 1000 fields (or just X number of fields)? My OP was initially because of that issue, and I don't see anything in the usage docs about limits outside of legacy templates. I'd assume that something like the --strict arg would do this, but that doesn't seem to be the case.

[kgeller]

is there a way to force the generator to fail if it generates a component template over 1000 fields (or just X number of fields)?

I don't think we currently have any functionality to force a failure like that.

I'd assume that something like the --strict arg would do this, but that doesn't seem to be the case.

Correct, strict is defined here, and the number of fields isn't something it looks at today.

[BenB196]

Thanks, will open a new enhancement request for it, as I think component templates are hard capped at 1000 fields, and I think it would be useful to catch issue at the generation side, not the ES upload side.