cc @bleggett

For helm we may need to have users explicitly do it, but for istioctl we can automatically set trustedZtunnelNamespace based on components.ztunnel.namespace. Should be a 1 line change.

Yeah this is another one that I definitely created a release note for but which didn't make it into the published notes, for reasons that I'm not clear on.

#51311

Behavior:

• If neither pilot.env.CA_TRUSTED_NODE_ACCOUNTS or pilot.trustedZtunnelNamespace is set, assume ztunnel resides in the same namespace as istiod does.
• If pilot.trustedZtunnelNamespace set, use that namespace.
• If pilot.trustedZtunnelNamespace unset, and pilot.env.CA_TRUSTED_NODE_ACCOUNTS set, use the latter.

Oh it is in the change notes: https://istio.io/latest/news/releases/1.23.x/announcing-1.23/change-notes/#installation

The reason we did not simply sustain the previous default (istio-system/ztunnel,kube-system/ztunnel) was that the previous default granted access to two namespaces, only one of which would ever be used, which was arguably a security hole.

Installing ztunnel to a different namespace than istiod requires explicit configuration, in all cases.

Oh it is in the change notes: https://istio.io/latest/news/releases/1.23.x/announcing-1.23/change-notes/#installation

Ok good.

Oh it is in the change notes: https://istio.io/latest/news/releases/1.23.x/announcing-1.23/change-notes/#installation

I don't use the Helm chart however and nothing in that line tells me that previous default behavior changed. Based on the upgrade notes, I expected to be able to install Istio 1.23.0 using my current IstioOperator manifest and the istioctl command.

The reason we did not simply sustain the previous default (istio-system/ztunnel,kube-system/ztunnel) was that the previous default granted access to two namespaces, only one of which would ever be used, which was arguably a security hole.

I definitely agree with that sentiment. In my new configuration I only configured it for kube-system.

Installing ztunnel to a different namespace than istiod requires explicit configuration, in all cases.

Are all of the various places it needs to be configured documented somewhere? I haven't come across a complete list of the places updates are needed, I kind of just figured it out when I initially deployed Ambient mode on 1.22.x.

Oh it is in the change notes: https://istio.io/latest/news/releases/1.23.x/announcing-1.23/change-notes/#installation

I don't use the Helm chart however and nothing in that line tells me that previous default behavior changed. Based on the upgrade notes, I expected to be able to install Istio 1.23.0 using my current IstioOperator manifest and the istioctl command.

Agreed, fair. It could have been more explicit that because the default behavior was a problem, the default behavior has changed. Since ambient is still in beta, maintaining strict back compat is not a requirement, but the relnote (that I wrote) could have been worded more explicitly.

Are all of the various places it needs to be configured documented somewhere? I haven't come across a complete list of the places updates are needed, I kind of just figured it out when I initially deployed Ambient mode on 1.22.x.

istioctl install --set values.pilot.trustedZtunnelNamespace=<MY_ZTUNNEL_NS> or helm install istiod --set pilot.trustedZtunnelNamespace=<MY_ZTUNNEL_NS> at install time should be all that is required.

Or in operator-resource-speak:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    pilot:
      trustedZtunnelNamespace: <MY_ZTUNNEL_NS>

Helm has ways to describe the available values that can be overriden for a particular chart (helm get values),

istioctl is mostly just using the Helm templates under the hood, and the Operator resource itself is a bit vestigial since the operator has been deprecated - I don't think there's a way to do the same there.

This is something generally we are trying to improve.

Agreed, fair. It could have been more explicit that because the default behavior was a problem, the default behavior has changed. Since ambient is still in beta, maintaining strict back compat is not a requirement, but the relnote (that I wrote) could have been worded more explicitly.

Ah to be honest, Ambient mode has worked so well I kind of forgot it is technically a beta feature 😅

Or in operator-resource-speak:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    pilot:
      trustedZtunnelNamespace: <MY_ZTUNNEL_NS>

Thank you! That is simpler than specifying via the environment variable.

Ah to be honest, Ambient mode has worked so well I kind of forgot it is technically a beta feature 😅

Please ping me on Slack so we can publish a case study based on that sentiment :)

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    pilot:
      trustedZtunnelNamespace: <MY_ZTUNNEL_NS>

Using that config results in Error: generate config: error unmarshaling JSON: while decoding JSON: unknown field "trustedZtunnelNamespace" in isitio.operator.v1alpha1.ComponentSpec

It doesn't look to be a valid part of https://istio.io/latest/docs/reference/config/istio.operator.v1alpha1/#ComponentSpec either. I tried nesting it under spec.components.pilot.spec, Istio deploys but istiod has the incorrect CA_TRUSTED_NODE_ACCOUNTS again.

This is the config that is currently working for me but I don't know if it is a recommended way of configuring it. I grepped through the manifests included in the Istio v1.23.0 release and it looks like this env var is the only place .Values.pilot.trustedZtunnelNamespace is used.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    pilot:
      k8s:
        env:
        - name: CA_TRUSTED_NODE_ACCOUNTS
           value: "kube-system/ztunnel"

So we need two things.

1. Add trustedZtunnelNamespace to value_types
2. Auto detect trustedZtunnelNamespace
3. (nice to have) ensure (1) cannot happen again through some testing mechanism

https://github.com/istio/istio/pull/51311/files#diff-74a4ba6cc7c1c0c2070204a32d2f14931b8f379cd72424d5fa4639f625c3587eR982

It should be in values_types already. istioctl (and the Operator resource) is ultimately just an indirection around the helm charts, which support this in a pretty straightforward fashion.

istioctl install --set values.pilot.trustedZtunnelNamespace will work, I'm not familiar with what the IstioOperator does that makes this not-work.

The env-var is equivalent anyway, so that's fine to carry on with if you continue to use IstioOperator.

oh @Phenix66 my bad - it should be

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    pilot:
      trustedZtunnelNamespace: <MY_ZT_NAMESPACE>

Because the IstioOperator indirects the Helm values, but (apparently) in several different ways. (component.pilot = no, values.pilot = yes)

(why do we have 6 ways to do one thing? IDK - I do apologize tho)

oh @Phenix66 my bad - it should be

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    pilot:
      trustedZtunnelNamespace: <MY_ZT_NAMESPACE>

Because the IstioOperator indirects the Helm values, but (apparently) in several different ways. (component.pilot = no, values.pilot = yes)

(why do we have 6 ways to do one thing? IDK - I do apologize tho)

That worked perfectly, thank you!

Closing since we found this is documented in https://istio.io/latest/news/releases/1.23.x/announcing-1.23/change-notes/#installation and issue resolved above. Thanks everyone.