-
Notifications
You must be signed in to change notification settings - Fork 226
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ElasticAPM crash because of a missing context since version 6.3.0 #1188
Comments
Same problem here |
…rrectly If the elasticsearch span is dropped for some reason, the context object is None, which, if unhandled, leads to an exception fixes elastic#1188
@cccs-sgaron thanks for opening the issue! I went through the code, and the only possible way that I currently see how this could happen is if the Elasticsearch span is a
I opened #1190 to address the issue. The test case in that pull request triggers the same stack trace as the one you added to this issue. |
My case is 99% the 1). In the past we had that note in kibana saying that some spans were dropped since there were too many. but it was working without problem. |
Our issue was most likely due to hitting the transaction_max_spans as well since the API that we were monitoring may recursively loop and issue DB calls to elastic each of them creating a new span. Thanks for the fix, will report back if the issue come up again. |
@cccs-sgaron @esseti Awesome, thanks for the feedback. I just pushed version 6.3.3 of the agent with the fix from #1190 |
…rrectly (elastic#1190) If the elasticsearch span is dropped for some reason, the context object is None, which, if unhandled, leads to an exception fixes elastic#1188
Describe the bug:
Since ElasticAPM release 6.3.0 our application has started to see a ton of crashes that we believed are cause by a recent change in ElasticAPM. The elasticsearch instrumentation package is trying to assign the type of DB to elasticsearch into an None context which causes it to crash.
Here's a snippet of a strack trace from our app:
I believe this crash is related to a recent change in this commit: ee75cb8#diff-c8fb731f92134757656c157f5c3175bcb62e131c1fed1aec5041367603c204d0L62
You can see here, the context was previously assigned it's DB type in a way where even if the context was None it would still work but now it assumes the context is a dictionary. I'm not creating a PR to fix this because I'm not 100% sure if the old way was changed for a reason.
Possible fix
I have very limited understanding on what that context should be before reaching this function but possible fixes include:
Revert to the old way of assigning the DB type.
To test for None context before assigning a type.
To Reproduce
I have no easy way to reproduce this crash because it does not happen all the time.
Environment (please complete the following information)
Aditional Information
Our app in launched in Gunicorn using gevent workers.
The text was updated successfully, but these errors were encountered: