image rendering times out sporadically #582

jluebbe · 2024-11-20T19:46:37Z

What happened:
After updating grafana (to 11.3.1) and the grafana-image-renderer (to 3.11.6), some image rendering requests fail with a timeout, while most work.
As far as I can tell it affects ~25% of requests.

The log message from the renderer is:

logger=plugin.grafana-image-renderer t=2024-11-20T20:22:04.460896286+01:00 level=debug msg="Render request received" url="http://127.0.0.1:3000/d-solo/zglP5XSWk/node?panelId=4&var-node=74acb92689a8&from=now-1d&width=650&height=350&theme=light&render=1"
logger=plugin.grafana-image-renderer t=2024-11-20T20:22:05.585531541+01:00 level=error msg="Browser request failed" url="http://127.0.0.1:3000/api/ds/query?ds_type=influxdb&requestId=SQR100" method=POST failure=net::ERR_ABORTED
logger=plugin.grafana-image-renderer t=2024-11-20T20:22:34.507991251+01:00 level=error msg="Error while waiting for the panels to load" url="http://127.0.0.1:3000/d-solo/zglP5XSWk/node?panelId=4&var-node=74acb92689a8&from=now-1d&width=650&height=350&theme=light&render=1" err="TargetCloseError: Protocol error (Runtime.callFunctionOn): Target closed
    at CallbackRegistry.clear (/snapshot/src/node_modules/puppeteer-core/lib/cjs/puppeteer/common/CallbackRegistry.js:72:36)
    at CdpCDPSession._onClosed (/snapshot/src/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/CDPSession.js:101:25)
    at Connection.onMessage (/snapshot/src/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/Connection.js:130:25)
    at WebSocket.<anonymous> (/snapshot/src/node_modules/puppeteer-core/lib/cjs/puppeteer/node/NodeWebSocketTransport.js:44:32)
    at callListener (/snapshot/src/node_modules/ws/lib/event-target.js:290:14)
    at WebSocket.onMessage (/snapshot/src/node_modules/ws/lib/event-target.js:209:9)
    at WebSocket.emit (node:events:537:28)
    at Receiver.receiverOnMessage (/snapshot/src/node_modules/ws/lib/websocket.js:1220:20)
    at Receiver.emit (node:events:537:28)
    at Immediate.<anonymous> (/snapshot/src/node_modules/ws/lib/receiver.js:601:16)"
logger=plugin.grafana-image-renderer t=2024-11-20T20:22:34.508769665+01:00 level=error msg="Render request failed" url="http://127.0.0.1:3000/d-solo/zglP5XSWk/node?panelId=4&var-node=74acb92689a8&from=now-1d&width=650&height=350&theme=light&render=1" error="Error: Timeout hit: 30000"

The influx log doesn't show anything obviously problematic.

What you expected to happen:
All requests render correctly.

How to reproduce it (as minimally and precisely as possible):
The images are used on public pages: https://freifunk-bs.de/map/#!/en/map/68725132e5cb

Anything else we need to know?:
In the error case, the renderer responds with a truncated error message instead of the image:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
    <meta name="viewport" content="width=device-width" />
    <meta name="theme-color" content="#000" />

    <title>Grafana - Error</title>

    <base href="/grafana/" />

    
    <link rel="stylesheet" href="

Environment:

Grafana Image Renderer version: 3.11.6
Grafana version: 11.3.1
Installed plugin or remote renderer service: installed as plugin
OS Grafana Image Renderer is installed on: Debian 12.8
User OS & Browser: Debian Chromium, Andorid, curl
Others:

The text was updated successfully, but these errors were encountered:

mikerevellesciplay · 2024-11-25T16:46:47Z

had the same issue with 11.3.1, spent more than a week troubleshooting this and it looks like downgrading to 11.2.4 fixed it. directly rendering from the grafana interface (panel -> share -> generate rendered image) works fine but rendering on alert is broken.

manny4566 · 2024-12-02T14:57:40Z

I can confirm the problem

KausL · 2024-12-04T15:29:21Z

I can confirm this issue:

With Grafana v10 I use the rendering with a link like this

https://10.10.20.7:3000/render/d-solo/dtfndb010/10-system-performance-24h-overview/?panelId=34&width=260&height=260&from=1733237940000&to=1733324340000

The image generation takes around 5s depending on the panel.

With Grafana 11.3.1 I use the same url is used but now the image generation takes around 30s which expands the generation of my report email from 5min to 30min which is not really critical but very strange.

After playing around with the renderer settings I found out that using the direct link button via the grafana UI this is wotking as fast as under v10. So the difference is that the link callled by the button contains all the vars defined in the dashboard (no matter if it is used in the panel or not). The Link looks like this now:

https://10.10.20.7:3000/render/d-solo/dtfndb010?orgId=1&from=2024-12-03T15:10:40.882Z&to=2024-12-04T15:10:40.882Z&var-sitea=BRE&var-siteb=HAM&var-sitec=KLN&var-maxOpsRDS=99600&var-maxOpsBDS=28800&var-maxQryBDS=17400&var-maxUpdBDS=3800&var-srvCountBDS=6&var-srvCountRDS=6&var-srvCountBDSOneSite=2&refresh=1m&panelId=panel-34&__feature.dashboardSceneSolo&width=1000&height=500&tz=Europe%2FBerlin

Can anybody explain how this issues/bug can be solved/switched back to the old behaviour?

Thanks in advance!

mikerevellesciplay · 2024-12-05T01:55:20Z

@evictorero @lucychen-grafana do you guys test this stuff?

KausL · 2024-12-05T08:47:53Z

I can also confirm that the issue is solved by downgrading to grafana 11.2.4 as mentioned by @mikerevellesciplay .

jluebbe · 2024-12-11T18:57:00Z

had the same issue with 11.3.1, spent more than a week troubleshooting this and it looks like downgrading to 11.2.4 fixed it.

Thanks for the hint, @mikerevellesciplay. I've downgraded Grafana to 11.2.4 and now rendering is reliable.

Andrew-Hinson · 2024-12-12T03:50:34Z

I'm also dealing with this issue, I have an app that "recreates" grafana dashboards for posting to slack, notably, only panels that query snowflake are having issues, likely due to some small lag as the queries are fairly quick. Before we upgraded Grafana, no issues. Now, I routinely get blank panels sent to slack. Downgrading is not an option for us.

evictorero · 2024-12-12T13:37:53Z

Hi everyone. Thanks for reporting this. We are trying to triage this issue and isolate this case to be able to reproduce it.
As far as I can tell, all of you are having issues after upgrading Grafana to 11.3.1 and consuming the render API from an external app using the same link you used in older versions. We suspect this problem could be related to a retro compatibility issue.

Can you help us answering these questions?

When does it start failing? e.g After upgrading to 11.3.1 from 11.2.0
Which image renderer version are you using?
Is the image renderer configured as a plugin or as an external service?
Which data sources use the panels that are failing?
Does this issue happen randomly or can you reproduce it consistently?
If you grab the URL from the panel UI in Grafana 11.3.1. Does it fail?
Does adding the query param __feature.dashboardSceneSolo to the old URL make it work?

manny4566 · 2024-12-17T12:08:23Z

Hi everyone. Thanks for reporting this. We are trying to triage this issue and isolate this case to be able to reproduce it. As far as I can tell, all of you are having issues after upgrading Grafana to 11.3.1 and consuming the render API from an external app using the same link you used in older versions. We suspect this problem could be related to a retro compatibility issue.

Can you help us answering these questions?

When does it start failing? e.g After upgrading to 11.3.1 from 11.2.0

Which image renderer version are you using?

Is the image renderer configured as a plugin or as an external service?

Which data sources use the panels that are failing?

Does this issue happen randomly or can you reproduce it consistently?

If you grab the URL from the panel UI in Grafana 11.3.1. Does it fail?

Does adding the query param __feature.dashboardSceneSolo to the old URL make it work?

Can you help us answering these questions?

When does it start failing?
e.g After upgrading to 11.3.1 from 11.2.0 -> on of the last updates grafana-11.3.0-1.x86_64 or grafana-11.3.1-1.x86_64. Bevor 11.3.0 i had grafana-11.2.2-1.x86_64

Which image renderer version are you using?
3.11.6

Is the image renderer configured as a plugin or as an external service?
Plugin on the Grafana Host

Which data sources use the panels that are failing?
InfluxDBv2

Does this issue happen randomly or can you reproduce it consistently?
fairly regularly if not always

If you grab the URL from the panel UI in Grafana 11.3.1. Does it fail?
It run into the same timeout error
Does adding the query param __feature.dashboardSceneSolo to the old URL make it work? no. example URL:
http://HOST:3000/d-solo/icinga2/icinga2-with-influxdb?from=now-3h&height=480&orgId=1&panelId=3&render=1&theme=dark&timezone=browser&to=now&var-command=load&var-disk=%24__all&var-hostname=XXX&var-service=Load+1h&var-threadcount=%24__all&width=1000&__feature.dashboardSceneSolo=false

Also not without "&__feature.dashboardSceneSolo=false"

I hope this helps.
Thank you

manny4566 · 2024-12-17T17:49:41Z

Hi,
i think i'm getting closer to the problem.

The problem is the seems to be the panal id. With 11.3.0 the variable for the panal id changed:

v11.2.2
https://XXX/d/icinga2/icinga2-with-influxdb?var-hostname=XXX&var-service=Load%201h&var-command=load&from=now-3h&to=now&orgId=1&viewPanel=3

Here it is &viewPanel=3

v11.3.0
https://XXX/d/icinga2/icinga2-with-influxdb?var-hostname=XXX&var-service=Load%201h&var-command=load&from=now-3h&to=now&orgId=1&viewPanel=panel-3&timezone=browser&var-disk=$__all&var-threadcount=$__all

Here it is &viewPanel=panel-3.

Futhermore automatically "&timezone=browser&var-disk=$__all&var-threadcount=$__all" is added for variable without a definition ($__all).
In 11.2.2 they are not added automatically to the URL

##################
As you can see, it is an Icinga2 implementation. The url is build by Icinga2, concretely the Icinga2 grafana plugin. I think there is a discrepancy between the Icinga2 module and the Grafana panal id variable, which has changed in 11.3.0.
Possibly also in interaction with the image renderer plugin.
I use this environment:

Icinga2 2.14.3
Grafana Plugin v2.0.3 - https://github.com/Mikesch-mp/icingaweb2-module-grafana
same Problem with v3.0.0 - https://github.com/NETWAYS/icingaweb2-module-grafana

Grafana 11.3.2 ( But also with v11.3.0 ). With v11.2.2 it works.
grafana-image-renderer v3.11.6

I hope this will help.
I will also add this comment to other Github tickets.

Pedritod · 2024-12-17T21:16:04Z

We are also facing this issue, by doing the downgrade it works again. Thank you for the nice explanation and investigation of the bug

okossuth · 2024-12-22T13:53:41Z

Same here, i did some tests and there were issues rendering some panels using grafana 11.4.0 and grafana-image-renderer 3.11.6 running as external service. Once i downgraded grafana to 11.2.5, rendering panels worked 100% of the time.

herbetom · 2025-01-15T13:12:52Z

For us the perceived "timeout problem" appears to have turned out to be a missing Variable that was used in the title of the dashboard.

I'm speculating that there previously was some form of redirect or url rewrite taking place which poulated that var (as the rendered image contained it's content before), but that doesn't seem to work anymore.

So everyone who is having issues might want to check that all variables which are needed are provided. It doesn't seem to fail gracefully if not all are present.

Just as a tip, if it's not consitently broken it might of course be something else.

Pedritod · 2025-01-16T08:06:41Z

We thought that missing variables might be the root cause as well, especially since they were used in the dashboard title and sometimes they were not filled. However, after testing over several days, we observed inconsistent behavior — sometimes it worked, and other times it didn’t.

zvarnes · 2025-02-01T03:22:03Z

Downgraded grafana to v11.2.4 and the renderer is working again as expected.

kstange-enfn · 2025-02-06T23:06:36Z

I can confirm this as well, but I can't use 11.2.x because we're trying to use the slack image uploader which was fixed in 11.4.1 and 11.5.0 (grafana/alerting#256), so there's no compatible version of Grafana where both things are working properly right now.

pgassmann · 2025-02-10T17:17:18Z

I am also debugging this issue.
I cannot yet pinpoint what exactly makes it break, sometimes it works on the same panels, and sometimes it doesn't.
Feels like some race condition.
I tested it using the share link > Generate image feature of the panel:
It happens more often on some panels/dashboards than on others. on some dashboard I cannot reproduce the issue, the images are always rendered.

Suraxius · 2025-02-24T18:04:55Z

Tested. Same result. Any news?

vincent-olivert-riera · 2025-03-05T08:38:02Z

We also had to downgrade from 11.5 to 11.2 because of this bug.

vincent-olivert-riera · 2025-03-05T10:08:43Z

I have been digging into this trying to find a consistent way to reproduce the issue, to understand where can the problem be.

I think the problem occurs when the render URL does not include certain variables in the query.
For instance, taking this URL as an example:

https://grafana.*****.com/render/d-solo/ceevt2rdruk1sb/foo?orgId=1&var-service_code=talk&var-space_id=m&panelId=1

This URL works in Grafana <= 11.2.x. The panel does not use any other variables apart from service_code and space_id, which are defined in the URL.

However, that same URL does not work in in Grafana > 11.2.x.

I happen to know that the dashboard has other variables defined, so if I modify the URL to add those variables (even if I don't set any value), it starts working.
For instance, the following URL works in Grafana > 11.2.x:

https://grafana.*****.com/render/d-solo/ceevt2rdruk1sb/foo?orgId=1&var-service_code=talk&var-space_id=m&var-media_type=&var-backend_location=&panelId=1

The only thing I did was adding the media_type and backend_location variables to the URL, without setting any value for them. Even if the variables are not used by that panel, they still need to be defined in the URL, otherwise it does not work and we receive a timeout response.

However, does that not mean we need to add ALL the variables from the dashboard into the URL.
I happen to know that the dashboard has other variables apart from media_type and backend_location, and even without adding those other variables the URL still works. Why?

Well, I am not sure about this, but the type of the media_type and backend_location variables is "query", and the type of those other variables that I didn't include in the URL is "custom".
As long as the panel does not use them, if you remove any "custom" variable from the URL, no problem. But if you remove a "query" variable (even if it's not used by the panel), the render times out.

I didn't have time to investigate more, but I think this should give enough detail for a Grafana developer to quickly find where is the problem.

evictorero added this to Sharing squad Dec 9, 2024

github-project-automation bot moved this to 🗂️ Needs Triage / Escalation in Sharing squad Dec 9, 2024

evictorero added the type/bug Something isn't working label Dec 12, 2024

evictorero mentioned this issue Dec 12, 2024

Grafana 11.3.0 - Share Panel, embed: refresh does not work !? grafana/grafana#95315

Closed

manny4566 mentioned this issue Dec 17, 2024

[Bug]: Rendering of a Grafana Dashboard run into a timeout NETWAYS/icingaweb2-module-grafana#40

Open

1 task

vincent-olivert-riera mentioned this issue Mar 5, 2025

Reopen 95268: grafana image renderer generate an error and a tiemout before handling the image grafana/grafana#95408

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

image rendering times out sporadically #582

image rendering times out sporadically #582

jluebbe commented Nov 20, 2024

mikerevellesciplay commented Nov 25, 2024 •

edited

Loading

manny4566 commented Dec 2, 2024

KausL commented Dec 4, 2024

mikerevellesciplay commented Dec 5, 2024

KausL commented Dec 5, 2024 •

edited

Loading

jluebbe commented Dec 11, 2024 •

edited

Loading

Andrew-Hinson commented Dec 12, 2024

evictorero commented Dec 12, 2024

manny4566 commented Dec 17, 2024 •

edited

Loading

manny4566 commented Dec 17, 2024

Pedritod commented Dec 17, 2024

okossuth commented Dec 22, 2024

herbetom commented Jan 15, 2025

Pedritod commented Jan 16, 2025

zvarnes commented Feb 1, 2025

kstange-enfn commented Feb 6, 2025 •

edited

Loading

pgassmann commented Feb 10, 2025

Suraxius commented Feb 24, 2025 •

edited

Loading

vincent-olivert-riera commented Mar 5, 2025

vincent-olivert-riera commented Mar 5, 2025 •

edited

Loading

image rendering times out sporadically #582

image rendering times out sporadically #582

Comments

jluebbe commented Nov 20, 2024

mikerevellesciplay commented Nov 25, 2024 • edited Loading

manny4566 commented Dec 2, 2024

KausL commented Dec 4, 2024

mikerevellesciplay commented Dec 5, 2024

KausL commented Dec 5, 2024 • edited Loading

jluebbe commented Dec 11, 2024 • edited Loading

Andrew-Hinson commented Dec 12, 2024

evictorero commented Dec 12, 2024

manny4566 commented Dec 17, 2024 • edited Loading

manny4566 commented Dec 17, 2024

Pedritod commented Dec 17, 2024

okossuth commented Dec 22, 2024

herbetom commented Jan 15, 2025

Pedritod commented Jan 16, 2025

zvarnes commented Feb 1, 2025

kstange-enfn commented Feb 6, 2025 • edited Loading

pgassmann commented Feb 10, 2025

Suraxius commented Feb 24, 2025 • edited Loading

vincent-olivert-riera commented Mar 5, 2025

vincent-olivert-riera commented Mar 5, 2025 • edited Loading

mikerevellesciplay commented Nov 25, 2024 •

edited

Loading

KausL commented Dec 5, 2024 •

edited

Loading

jluebbe commented Dec 11, 2024 •

edited

Loading

manny4566 commented Dec 17, 2024 •

edited

Loading

kstange-enfn commented Feb 6, 2025 •

edited

Loading

Suraxius commented Feb 24, 2025 •

edited

Loading

vincent-olivert-riera commented Mar 5, 2025 •

edited

Loading