CertMgr Empty Verify Issue

Mindwatering Incorporated

Author: Tripp W Black

Created: 06/17/2023 at 09:20 PM

 

Category:
Domino Server Issues Troubleshooting
Other

Issue:
Certificate Store (CertMgr) Error Renewing Certificate. Verification fails.
Domino Server: HCL Domino 12.0.1, Linux, Rocky OS and Ubuntu 20/22.

Error Message:
Certificate store error message:
Verify Failed. Empty reply from Server


Symptom Evidence:
Curl from Domino server(s) running Rocky OS 8 or CentOS 7 show empty response from server when going out to public IP.
$ curl -L -v http://www.mydomain.mindwatering.net/.well-known/acme-challenge/DOMINO-CertMgr-DiagChallenge-HTTP01
* About to connect() to www.mydomain.mindwatering.net port 80 (#0)
* Trying 64.98.86.8...
* Connected to www.mydomain.mindwatering.net (64.98.86.8) port 80 (#0)
> GET /.well-known/acme-challenge/DOMINO-CertMgr-DiagChallenge-HTTP01 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: www.mydomain.mindwatering.net
> Accept: */*
>
* Empty reply from server
* Connection #0 to host www.mydomain.mindwatering.net left intact
curl: (52) Empty reply from server


Curl from Ubuntu 20 machine.
Similar except that it says that the server requires HTST and redirects to HTTPS and fails to find the file. (The Web Site rules do NOT include that header actually.) Regardless, the HTTPS port is behind a reverse-proxy. Although the HCL Git site documentation indicates that any Domino server can serve the challenge. The CertMgr verify appears to have to resolve to that instance of the Domino server. The reverse proxy log shows that the request can go to any of the back-end Domino servers supplying the domain.


Curl from OSX Ventura Mac shows correct curl:
$ curl -L -v http://www.mydomain.mindwatering.net/.well-known/acme-challenge/DOMINO-CertMgr-DiagChallenge-HTTP01
* About to connect() to www.mydomain.mindwatering.net port 80 (#0)
* Trying 64.98.86.8...
* Connected to www.mydomain.mindwatering.net (64.98.86.8) port 80 (#0)
> GET /.well-known/acme-challenge/DOMINO-CertMgr-DiagChallenge-HTTP01 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: www.mydomain.mindwatering.net
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: Lotus-Domino
< Date: Sun, 18 Jun 2023 01:09:41 GMT
< Content-Type: text/html
< Content-Length: 21
< X-FRAME-OPTIONS: SAMEORIGIN
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: same-origin
<
Challenge NOT found
* Connection #0 to host www.mydomain.mindwatering.net left intact
curl: (52) Empty reply from server

This is actually okay, because this is not the correct challenge file, but the generic DXL one from the HCL Git site documentation.


Solution:
The simplest solution was to update the local internal DNS so that internal access to the domain used the "primary" Domino server's IP. However, this can create an outage if the primary Domino server is down for maintenance. We then added 3 A records for the domain for all 3 domain servers. That worked, 1/3 of the time. The Domino server didn't seem to like it's verify going to another Domino server.

Because the number of domains on the Domino servers is less than 100, we elected to hard-code the /etc/hosts for each of the Domino server's to "this" Domino server. That way no matter which one performed the verify it worked. The reverse proxy does the availability check for the backend Domino servers to ensure that an "up" Domino server got the domain request.





previous page

×