[New Analyzer] Elasticsearch Analyzer

[nmprokop]

Query any ElasticSearch cluster or multiple clusters for any field and any index and return common fields that use the ElasticSearch Common Schema (ECS)

#841

[LaZyDK]

I love this pull request. I modified only the fields in the code, but everything is exactly the same as in the initial pull request. Working brilliant!

[azgaviperr]

I love this pull request. I modified only the fields in the code, but everything is exactly the same as in the initial pull request. Working brilliant!

Have you got an example how you changed the field in the code ? I got issue to get a proper view, filed must not be right on my change but don't know why.

[nicpenning]

Hey @azgaviperr, what code did you change? What fields did you want to add to the table?

Basically you need to define new fields in the python analyzer file and then adjust the template code to display the new field.

If you are just looking to search on something specific you should just be able to add whatever fields you want to search on and the indices you want to search in the cortex configuration in the UI.

What exactly are you looking to tweak?

[azgaviperr]

Hey @azgaviperr, what code did you change? What fields did you want to add to the table?

Basically you need to define new fields in the python analyzer file and then adjust the template code to display the new field.

If you are just looking to search on something specific you should just be able to add whatever fields you want to search on and the indices you want to search in the cortex configuration in the UI.

What exactly are you looking to tweak?

Hello @nicpenning May main trouble is that the report table is mostly empty as show in my screenshot. I suppose this is due to the field used there are not right with the one used on my ELK. My source ip field is "src_ip" butn ot sure where in the code I should change it.

[LaZyDK]

I will recommend that you start converting your fields to follow the ecs format, so you won't have to model everything around your custom fields :)

[nicpenning]

I would push what @LaZyDK said, it will pay dividends if you can use Logstash or Elastic ingest pipelines to convert your src_ip to source.ip to match the Elastic Common Schema (ECS).

But, if you did want to use custom fields then find the source ip fields in the analyzer and duplicate the code and change it to src_ip.

I can give a more detailed way to do this when I get to a computer.

[nicpenning]

You would need to add this at 274:

                        if 'src_ip' in hit['_source']:
                            source_ip = hit['_source']['src_ip']

I didn't test but that should be close.

But again, you are way better off to convert that that field on ingest from src_ip to source.ip ( [source][ip] ).

Because I have the feeling you will need to do the same for destination ip, domain, etc.. ECS is worth every minute of understanding and using.

If one does make those changes above then they should show up in the analyzer report under the Source IP column since we mapped that field to the source_ip variable.

If you wanted to create it's own column for src_ip you could do that by adding a table header and body on lines 124 and 169 in the html template as such:
<th>Custom Column</th>

<td style="overflow:auto"> {{hit.custom_field_that_needs_to_be_baked_in_the_analyzer}} </td>

I am sure the creator of this analyzer will be able to validate these claims :D

[azgaviperr]

You would need to add this at 274:

                        if 'src_ip' in hit['_source']:
                            source_ip = hit['_source']['src_ip']

I didn't test but that should be close.

But again, you are way better off to convert that that field on ingest from src_ip to source.ip ( [source][ip] ).

Because I have the feeling you will need to do the same for destination ip, domain, etc.. ECS is worth every minute of understanding and using.

If one does make those changes above then they should show up in the analyzer report under the Source IP column since we mapped that field to the source_ip variable.

If you wanted to create it's own column for src_ip you could do that by adding a table header and body on lines 124 and 169 in the html template as such:
<th>Custom Column</th>

<td style="overflow:auto"> {{hit.custom_field_that_needs_to_be_baked_in_the_analyzer}} </td>

I am sure the creator of this analyzer will be able to validate these claims :D

Thanks a lot for those info.

I am going to move for ECS, but got a lot to do to translate all I got old style if I must say :) .

[garanews]

@nmprokop thanks for this PR!
Just tested, what I tried:

• local ES and cloud ES
• ES without auth and ES with auth
• predefined index (winlogbeat-*) and custom index (logs-*)
• all stack tried was running v7.10.1

And it works as expected:

What I noticed:

• there is only one template for any search and (of course) fields sometime are not populated:

• is possible to set only one kibana dashboard from config and would be useful to have at least one dashboard address for each observable type (similar to previous point)

• I see the level malicious was not used, it would be nice to have it regarding data coming from the security agent, in order to use the red label when some data is found in the security logs:

But for the v.1.0 I think it is already usable as is.
Going to merge, thanks again!