Error connecting to CertStream - Connection is already closed #35
Comments
|
Facing the same issue again! |
|
Hi. Might be related to CaliDog/certstream-server#59 @Fitblip have you found the time to check this issue already? As far as I can see, your library is sending pings/keepalives over the websocket to indicate that it's still being used. I haven't been able to find the reason behind the random disconnects yet. |
|
Hmm interesting, are you folks using the |
|
I for my part am using your The error message is always "Connection is already closed.". There are basically two places where that exception is raised: websocket/_app.py def send(self, data, opcode=ABNF.OPCODE_TEXT):
[...]
if not self.sock or self.sock.send(data, opcode) == 0:
raise WebSocketConnectionClosedException(
"Connection is already closed.")websocket/_socket.py def recv(sock, bufsize):
if not sock:
raise WebSocketConnectionClosedException("socket is already closed.")
def _recv():
[...]
try:
if sock.gettimeout() == 0:
bytes_ = sock.recv(bufsize)
else:
bytes_ = _recv()
except socket.timeout as e:
[...]
except SSLError as e:
[...]
if not bytes_:
raise WebSocketConnectionClosedException(
"Connection is already closed.")For me the disconnects only appear within the latter (recv) call. It reads 0 bytes This is the full stack trace: |
|
I'm facing the same issue when running |
I exactly have same issue with using calidog server. Error occurred every 1~2 minutes so that connection is not stable. Hope this will be solved. |
|
I am having the same problem with 1.11 from pypi. |
|
This is fixed upstream websocket-client/websocket-client#594 |
|
@Chaz6 the linked PR does not fix the issue for me. I applied the patch locally but am still getting the same issue every two minutes or so. |
What version of Python do you have? |
I am currently developing and testing on |
OK, I am using python 3.9 so it is possible there is a different fix required. |
|
@Chaz6 @d-Rickyy-b I'm facing the same problem. I'm running certstream in the docker container on AWS ECS and python version is 3.9.5. I updated some packages, but still cannot resolve the issue. root@8a07445128d9:/opt/certstream-documentdb# pip freeze |
|
@Fitblip mind looking into this? This is "easy to reproduce" because it literally happens (reliably) every time I use certstream. |
|
Just as a point of reference, today I updated to certstream 1.12 (python 3.9.6 on Linux 64bit) and it works fine. |
|
Hi folks, apologies for the delay in looking into this. I think there's a few things in this one issue to the point where it's over-loaded. Install/packaging/isAlive issuesI'm able to use certstream as expected with docker and the lastest python, which leads me to believe this is not an issue with certstream, but a local issue: $ docker pull python
Using default tag: latest
latest: Pulling from library/python
d272b0d8f7c4: Pull complete
81fabd82f026: Pull complete
0193c0cdceae: Pull complete
967a8f138596: Pull complete
bc1dbd38cda7: Pull complete
80a46c137b0c: Pull complete
f41525227e2c: Pull complete
1354066e326c: Pull complete
ffb5ac5ec588: Pull complete
Digest: sha256:44e7cbba204f9a81ec5bfa1b5de1de176b2590c01d55a02bd299551593517c7d
Status: Downloaded newer image for python:latest
docker.io/library/python:latest
$ docker run --rm -it python bash
root@1477dfc03834:/ $ pip install certstream
Collecting certstream
Downloading certstream-1.12.tar.gz (9.6 kB)
Collecting websocket-client>=0.58.0
Using cached websocket_client-1.2.0-py2.py3-none-any.whl (52 kB)
Collecting termcolor
Using cached termcolor-1.1.0-py3-none-any.whl
Building wheels for collected packages: certstream
Building wheel for certstream (setup.py) ... done
Created wheel for certstream: filename=certstream-1.12-py3-none-any.whl size=4746 sha256=bf05687d2cfdbd224c5873cf0d6d229d6cd4176fa7cd5cd1ace19638ac45ee8c
Stored in directory: /root/.cache/pip/wheels/b9/d5/4b/157f4ada54a1b76c9509a7c00191014478d1039d390eda57c3
Successfully built certstream
Installing collected packages: websocket-client, termcolor, certstream
Successfully installed certstream-1.12 termcolor-1.1.0 websocket-client-1.2.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.2.3; however, version 21.2.4 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
root@1477dfc03834:/ $ certstream
[INFO:certstream] 2021-08-12 20:17:57,734 - Connection established to CertStream! Listening for events...
[2021-08-12T20:17:57.799850] ct.googleapis.com/logs/argon2021/ - imap.sangse.com
[2021-08-12T20:17:57.801696] ct.googleapis.com/logs/argon2021/ - universe.manawac.com
[2021-08-12T20:17:57.803194] ct.googleapis.com/logs/argon2021/ - inspire.devmantra.uk
[2021-08-12T20:17:57.804575] ct.googleapis.com/logs/argon2021/ - mail.airlab.id
[2021-08-12T20:17:57.805963] ct.googleapis.com/logs/argon2021/ - staging.191.com.auDisconnectionsThe certstream server is written in Elixir, using the excellent Cowboy http server. I believe when it doesn't receive any data from the client after a specific timeout (60s according to the docs), it assumes the client is dead and reaps the connection. The fix for this is to have the client spawn a thread to send some data up to the server every 30s or so (a simple I'm not sure I have time to implement this but if someone wants to take a stab at it I'd happily look at a PR! :) |
|
Thank you @Fitblip for your reply! I followed your steps: As you can see I still get the error Also in my custom software where I use certstream as a library: Is this the second part you addressed in your comment? So this could only be fixed with a regular |
|
Thank you @Fitblip @d-Rickyy-b. Even though I updated to certstream 1.12, I still get the same error. It is much appreciated if someone can implement above (ws.send('.')). |
|
@Fitblip @d-Rickyy-b I implemented ws.send in cli.py as following. But, still get the same error. I would like to prevent data loss caused by this error, do you have any ideas? |
|
@Fitblip @d-Rickyy-b I would like to know whether this disconnection happens only in my environment or not. Do you have any success cases (environments) that don't make this data loss error? or do you have any ideas to collect certificates data without data loss using other tools? |
|
I have changed the python version from 3.9.7 to 3.8.12 It working fine without the error |
|
@katesaikishore Thank you for your advice. I have changed the python to python:3.8.12-buster, but it still has the error. Could you show the other libraries versions? My environment is as following |
|
certstram==1.11 And use python 3.8.12 |
|
@katesaikishore Than you for your reply. We used the same versions of the libraries except for heroku (we have to use AWS), but it could not work well. I'll try it in local environment, not cloud. I'll also check the another certstream like Go, JAVA. |
|
@kres20 Oh, I have deployed it in heruko. Even though it will disconnect and reconnect itself. Something is really fishy. |
|
@katesaikishore I've just created the environment in locally (in VM) without docker, but it still has the error. I even used Go version, but still get the same error. @Fitblip Do you think that this problem is caused by server side (wss://certstream.calidog.io/)? |
|
@katesaikishore @Fitblip I set up the certstream server too (https://github.com/CaliDog/certstream-server), and connect there like |
|
Argh, well this is disappointing. :-( I am hitting this problem too and can only reach the same conclusion as @kres20 that the I noticed When I add When I use my preferred public DNS resolver (149.112.121.10), When I use other public DNS resolvers, I'm skeptical that IP anycast is a suitable CDN strategy for services requiring permanent TCP connections, such as the Calidog Certstream service. IMO this problem is likely a completely normal side-effect of the chosen CDN strategy that the Calidog service is using. :-( |
|
@mikenor Yep things are currently hosted behind Cloudflare, and I agree it's not ideal given the CDN nature of cloudflare and the persistent connections for the operation of certstream. Ultimately https://certstream.calidog.io is a service to try out certstream but if you need production grade quality (for analytics/etc) then you should absolutely run your own server. I chose Cloudflare because it was a way for me to spin up a SSL fronted service without much headache and to cut down on potential botspam. For the public instance we also currently fully saturate a gigabit link (https://twitter.com/fitblip/status/1203058921583398914) and pump about a petabyte of traffic into the internet every 3 months, so inevitably that means that certs are dropped and you shouldn't rely on the service to see 100% of everything. Ultimately I'm not interested in spending more time or money per month to host this in a way where it's a rock solid production quality service (certstream.calidog.io, the server software should be solid), but if someone wanted to donate some servers and/or run the service and run it as a cluster with multi-gigabit links then I'm absolutely game to point the DNS records at it and help with the effort! |
I implemented this tool, following the given examples. I found out that sometimes ( usally every 2 minutes, more or less) the connection drops and it takes almost a minute to reconnect to the stream.
I don't know why this happens and this could probably lead to missing some hits i'm looking for.
Here's an example of log ( integrated with python logging module )
Any suggestion about how to fix this?
The text was updated successfully, but these errors were encountered: