agent.environment and service.environment

[ruflin]

An agent and a service are often run in a certain environment like testing, staging, production. To group services and agents together by environment I'm proposing the following fields to be added to ECS:

• service.environment
• agent.environment

[ruflin]

@graphaelli Would this be helpful for APM?

[graphaelli]

Certainly, we actually already use service.environment for this purpose. We capture environment variables in env (not under service though).

[ruflin]

I wonder if one event is always from max 1 environment. So it can be that the service we monitor is in production but the agent is in testing? If yes, we could unify the two fields in event.environment.

[webmat]

I would rather add environment once, in a general enough place, though.

I feel like needing two instances of environments (e.g. in the example of pre-release agent you mention) is an edge case. I'd rather track a beta release of an "off the shelf" tool like a third party monitoring agent via its version.

I've always tracked which environment my stuff was in with free form labels in previous lives. Be it in Chef, some competing monitoring systems, or in my Elastic Stack logging pipeline. So I like the idea of adding it explicitly to ECS.

[ruflin]

Sounds like we should go with event.environment then?

To prevent any confusion: I expect this field NOT to be for environment variables but to be set to the environment the service / agent / device / host is running. Values could be testing, staging etc.

[graphaelli]

I see environment as part of the name/version/environment triple that uniquely identifies a service so service.environment makes more sense to me. Also, The event fields are used for context information about the data itself - I that supports the idea that environment is about the service, not the event itself.

APM has to move this either way we go (service is nested under context right now).

[ruflin]

@graphaelli I think you make a good point here: +1 on having both then.

[webmat]

In ECS I would much rather have one definition of environment, at the top level, so environment.

When thinking about services, service.environment makes sense, of course. But my problem with this is when processing logs and metrics across the board (a lot of which won't be using service. specifically), then setting service.environment will seem out of place.

Conversely, I do think it will make sense to people if their APM events have environment, service.name and service.version (), even if environment is not nested under `service.

[graphaelli]

Top level environment seems fine, wasn't sure it was worthy of that level. Given nesting under service vs event, I prefer service or both.

[ruflin]

I was more thinking to have also device.environment, meaning adding it to all objects where it makes sense.

[webmat]

But why have multiple places to store environment? Do you see instances of having one event with a different value for environment for a device, host, service etc. at the same time?

This sounds like an edge case. I rather think we should start simple here, and expand only when needed.

[ruflin]

The way I look at it is more to what the environment belongs to. It is the environment of the service or the environment of the device and not necessarily the environment of the event itself.

I don't think environment is something we need to resolve now.

[webmat]

I'm coming at it from a different point of view. The goal of ECS is to help define where each bit of information belongs, to facilitate correlation and match user's expectations.

It's totally fine to reuse some fields or sets of fields, when the information is expected to be present more than one time on a given event. Examples: ip, os.*, etc. are nested in a bunch of places because it's necessary.

For environment, however, I've always used that as a blanket setting, to flag everything about a certain part of my infra as being "prod", "staging" and the like. I've never had situations where a given event stream would actually have different values for environment on multiple fields, like:

{ "device": {
    "ip": "192.168.10.10",
    "environment": "production" ... },
  "agent": {
     "environment": "staging" ... }
}

If there's no expectation of environment being seen multiple times on a given event, I'd much rather define only one field for it, for predictability. My dumb brain is a big driver for me to push in this direction ;-)

@MikePaquette WDYT? Should we nest environment in multiple places, or should we define it once, at the top level for now?

[MikePaquette]

@webmat I think we should have just one environment field, for the reasons you stated. I would be happy with it as a base field environment or under event as event.environment, with a slight preference for environment.

As a side note, I think this is logically similar to version, which we currently have in event.version but I would prefer it as a base field version too.

[iorfix]

service.environment has been adopted by apmserver, but it is still not in ECS. Are there planning to formally add it?

[ebeahan]

Expecting RFC proposal from @cyrille-leclerc to propose adding serivce.environment field

[ebeahan]

Stage 0 RFC for serivce.environment merged in #891