Argocd Ingress not working on a Ubuntu Microk8s cluster

Question

I have a working Microk8s cluster. After enabling the argocd community addon, the recommended ingress for argocd sever doesn't seem to be working.

Here's a note I got after enabling the addon:

Infer repository community for addon argocd
Infer repository core for addon helm3
Addon core/helm3 is already enabled
Installing ArgoCD (Helm v4.6.3)
"argo" already exists with the same configuration, skipping
Release "argo-cd" does not exist. Installing it now.
NAME: argo-cd
LAST DEPLOYED: Thu Oct 20 17:34:33 2022
NAMESPACE: argocd
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
In order to access the server UI you have the following options:

1. kubectl port-forward service/argo-cd-argocd-server -n argocd 8080:443

    and then open the browser on http://localhost:8080 and accept the certificate

2. enable ingress in the values file `server.ingress.enabled` and either
      - Add the annotation for ssl passthrough: https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/ingress.md#option-1-ssl-passthrough
      - Add the `--insecure` flag to `server.extraArgs` in the values file and terminate SSL at your ingress: https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/ingress.md#option-2-multiple-ingress-objects-and-hosts


After reaching the UI the first time you can login with username: admin and the random password generated during the installation. You can find the password by running:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

(You should delete the initial secret afterwards as suggested by the Getting Started Guide: https://github.com/argoproj/argo-cd/blob/master/docs/getting_started.md#4-login-using-the-cli)
ArgoCD is installed

Additionally, here's the Ingress that I defined:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    cert-manager.io/cluster-issuer: lets-encrypt
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    # If you encounter a redirect loop or are getting a 307 response code
    # then you need to force the nginx ingress to connect to the backend using HTTPS.
    #
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  rules:
  - host: argocd.DOMAIN.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: argo-cd-argocd-server
            port:
              name: https
  tls:
  - hosts:
    - argocd.DOMAIN.com
    secretName: argocd-secret # do not change, this is provided by Argo CD

When I visit the host address, I get this:

From firefox, here's the certificate being issued.

From the notes I got after enabling this addon, how do I do this part enable ingress in the values file server.ingress.enabled this or get my ingress to correctly work?

UPDATE:

Here's what the ingress description looks like:

is the cert-manager installed and properly configured for lets-encrypt? — dummyuser, Oct 23 '22 at 06:17
@dummyuser yes. It's configured and the same issuer is working well with other ingresses. — Mysterio, Oct 23 '22 at 09:40
Please do the following command: `curl -v https://` (optionally with -k ) check the details of the certificate which is presented by argocd. Does it match your expectations or is it a selfsigned? Use alternativly Firefox, which allows to have a look at the certificate. — dummyuser, Oct 23 '22 at 09:48
@dummyuser updated the question with images from firefox. It seems "enable ingress in the values file `server.ingress.enabled`" thingy has to be done to terminate the internal certificate from the addon before the lets-encrypt issuer can do its thing — Mysterio, Oct 23 '22 at 09:59
so ... fist finding ... ingress seems to present a selfsinged Cert which is not trusted by your fist Browser. I'm not that much in nginx ingress, but the lines `kubernetes.io/tls-acme: "true"` and `nginx.ingress.kubernetes.io/ssl-passthrough: "true"` together do not really make sense to me. I guess the first one can be false. pls check Documentation ( e.g. https://cert-manager.io/docs/usage/ingress/). What happens if you click `Accept the Risk and continue`? — dummyuser, Oct 23 '22 at 10:45
is there a service named `argo-cd-argocd-server` up in the NS argocd and listens to 443? and additionally is there a POD up and running connected to the Service ? — dummyuser, Oct 23 '22 at 11:01
the line `kubernetes.io/tls-acme: "true"` should be removed from ingress. — dummyuser, Oct 23 '22 at 11:03
I guess the servicename in the ingress should be `argocd-server` instead of `argo-cd-argocd-server` — dummyuser, Oct 23 '22 at 11:06
@dummyuser check the løg info. Service name is the one specified and the service seem to be working as expected — Mysterio, Oct 23 '22 at 12:54
souds like you have a big argocd playground now. I'll write an official answer — dummyuser, Oct 23 '22 at 13:24

dummyuser · Answer 1 · 2022-10-24T07:07:06.407

1

The Error message did indicate an invalid Certificate which was not accepted by the browser. The ingess configuration requested a certificate and ssl-passthrough which does not fit together. The line kubernetes.io/tls-acme: "true" had to be removed from Ingress and a minor change of the service name.

The TLS connection to ArgoCD is terminated at the ArgoCD Server and not at the ingress GW. ArgoCD uses the Certificate (and Priv Key) stored in the argocd-secret. ingess should look like

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  rules:
  - host: argocd.<domain>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service: 
            name: argocd-server
            port:
              name: https
  tls:
  - hosts:
    - argocd.<url>
    secretName: argocd-secret # not relevant

and a kubectl get services must contain a line like

argocd-server     ClusterIP   10.99.19.178   <none>        80/TCP,443/TCP               179d

a kubectl get Ingress -n argocdresults in

NAME                    CLASS    HOSTS                 ADDRESS         PORTS     AGE
argocd-server-ingress   <none>   argocd.k3sxx.xx   192.168.xx.xx   80, 443   15m

and details can be seen with kubectl get Ingress -n argocd -o yaml

edited Oct 24 '22 at 07:07

answered Oct 23 '22 at 13:33

dummyuser

943
1
3
9

Hmmm so, I deleted addon and argocd namespace. I installed argocd via the official install guide (https://argo-cd.readthedocs.io/en/stable/getting_started/#1-install-argo-cd) and removed `kubernetes.io/tls-acme: "true"` from the ingress setup and service name set to `argocd-server` BUT I now get a 404 Not Found nginx error in the browser when I got to the URL generated. – Mysterio Oct 23 '22 at 14:39
The ingress description shows that it's correctly hooked into the `argocd-server` service. – Mysterio Oct 23 '22 at 14:39
1

please do a `curl -v -k https://argocd-server.argocd.svc.cluster.local` from any other POD in your cluster (does not work frm argocd NS, as these PODs do not have curl). if you get a http 200 Argos cd is set up properly and there is a ingress issue. I copied my ingress file to my answer. – dummyuser Oct 23 '22 at 15:08
I got an HTTP 200 alright. So argocd is working just fine. It's the ingress which for some reason is fighting over certificates. – Mysterio Oct 23 '22 at 18:19
The only unexplored option is the one from the logs to enable ingress via values file: "enable ingress in the values file `server.ingress.enabled`". Do u have an idea of how to generate and pass this value file? – Mysterio Oct 23 '22 at 18:23
This response could also be a reason for this error: https://stackoverflow.com/a/67263155/8194007 – Mysterio Oct 23 '22 at 18:27
the link just shows how to install argocd without TLS. the option `server.ingress.enabled` defines if the ingress ruel should be enabled at chart installation , then you need to add some parameter to the value file anyway. to me it is a bit easier to deploy the ingess rule separatly. may you update the `ingress file` in your question and show the output of `kubectl get Ingress -n argocd` (should look like `argocd-server-ingress argocd.k3s.xxx.xxx 192.168.xx.xx 80, 443 2m8s` ) – dummyuser Oct 24 '22 at 06:54
updated the question with the description of the ingress as well as the `get` command output. – Mysterio Oct 24 '22 at 07:01
is `server.ingress.enabled` set to false now and ingress is deployed separately ? looks like you still get a certificate from the ingress controller before the 404 – dummyuser Oct 24 '22 at 07:20
@dummuser, where do I check this? the deployment? or service? – Mysterio Oct 24 '22 at 07:43
lets do a quick summery: you installed argo using the file `https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml`, not the helm now? This file does not contain ingress definitions. If so values of the hlemchart (server.ingress.enabledare out of scope) Argo does work, checked with curl, you use an ingress-nginx. Right now, you get a certificate from ingress if trying to access argocd an afterwards a 404. I have a nginx-ingess config with passthrough Ingress running with the config in my answer. – dummyuser Oct 24 '22 at 08:06
Looks like there is anything in the nginx-ingress / Ingress definition which I cannot find remotely. Do you have other (working) ingress definitions on this Server ? – dummyuser Oct 24 '22 at 08:06
yes, I have 4 other ingress definitions working fine using the same certificate issued. The only difference is this service port points to HTTPS instead of some service port number. – Mysterio Oct 24 '22 at 08:29
ok in this case, add the `--insecure` option to the argocd deployment and use one of the other ingress as template - means we move tls termination from Argo to the ingress gw. – dummyuser Oct 24 '22 at 08:33
Which of the command should get this `--insecure` option? Not that well seasoned with the world of Kubernetes as you may have noticed. – Mysterio Oct 24 '22 at 08:42

Argocd Ingress not working on a Ubuntu Microk8s cluster

1 Answers1