Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Istioctl vs Operator vs Helm path forward #44814

Open
howardjohn opened this issue May 8, 2023 · 15 comments
Open

Istioctl vs Operator vs Helm path forward #44814

howardjohn opened this issue May 8, 2023 · 15 comments
Labels
area/environments lifecycle/automatically-closed Indicates a PR or issue that has been closed automatically. lifecycle/staleproof Indicates a PR or issue has been deemed to be immune from becoming stale and/or automatically closed

Comments

@howardjohn
Copy link
Member

Our different installation methods have caused tension and confusion. This issue aims to discuss some issues we have seen and some ideas for fixing these.

  • Confusion between istioctl vs operator, mostly driven by the IstioOperator API
    • Users think it means "do not use istioctl"
    • Users try to kubectl apply it
  • Confusion with installed-state - users think it is reconciled
  • Feature parity gaps between helm and istioctl
    • Some commands use installed-state; helm doesn't handle this (but it does have its own replacement)
    • Less validation in helm
  • IstioOperator API drives users to monolithic installations, which are hard to maintainer after day0
  • Docs inconsistently document one approach or the other.
  • Wasted effort maintaining installation code, which is outside of our expertise

In general, we don't want to remove any of these to reduce churn. Similarly, we don't want major changes to any of them, either.


Some ideas...

  1. Rename "Istio Operator" in all docs to "Istio In-Cluster Operator"
  2. Remove documentation about "Istio Operator" entirely. Hide commands in istioctl. Pretend it does not exist for all intents and purposes, other than for docs. We can move docs into operator/ or some wiki or point to archives. Add warnings to all istioctl commands using it.
  3. Remove installed-state IstioOperator entirely
    a. Stop installing IstioOperator CRD when not using in-cluster operator
    b. Fix commands depending on it to use alternative inputs (see (4))
  4. Remove all install code, make istioctl install a small shim over helm install
    a. May be less viable for the in-cluster operator, though.
  5. Allow inputs without kind: IstioOperator into istioctl. Update all docs to not have it.
@dgn
Copy link
Contributor

dgn commented May 10, 2023

I very much like the trajectory of this, I have been wanting to simplify the install experience for while.

Our different installation methods have caused tension and confusion. This issue aims to discuss some issues we have seen and some ideas for fixing these.

* Confusion between istioctl vs operator, mostly driven by the IstioOperator API
  
  * Users think it means "do not use istioctl"
  * Users try to kubectl apply it

* Confusion with installed-state - users think it is reconciled

* Feature parity gaps between helm and istioctl
  
  * Some commands use installed-state; helm doesn't handle this (but it does have its own replacement)
  * Less validation in helm

Maybe this could be solved by using schema validation?

* IstioOperator API drives users to monolithic installations, which are hard to maintainer after day0

* Docs inconsistently document one approach or the other.

* Wasted effort maintaining installation code, which is outside of our expertise

In general, we don't want to remove any of these to reduce churn. Similarly, we don't want major changes to any of them, either.

Some ideas...

1. Rename "Istio Operator" in all docs to "Istio In-Cluster Operator"

I think I prefer option 2.

2. Remove documentation about "Istio Operator" entirely. Hide commands in `istioctl`. Pretend it does not exist for all intents and purposes, other than for docs. We can move docs into operator/ or some wiki or point to archives. Add warnings to all istioctl commands using it.

Do we have any information on the user base?

3. Remove `installed-state` IstioOperator entirely
   a. Stop installing  IstioOperator CRD when not using in-cluster operator
   b. Fix commands depending on it to use alternative inputs (see (4))

4. Remove all install code, make `istioctl install` a small shim over `helm install`
   a. May be less viable for the in-cluster operator, though.

This would be great IMO. But keeping the IstioOperator API around would mean maintaining the whole conversion logic.

5. Allow inputs without `kind: IstioOperator` into istioctl. Update all docs to not have it.

This seems like an easy thing to do.

Agreeing with a lot of points here. From my perspective, I think we should strive to make helm the canonical install method and everything else should be done from that perspective. So I would suggest:

  • make the helm values a proper API (ie tougher checks around new settings etc.)
  • add schema validation to helm values
  • make the helm values our default install configuration
    • it's a bit confusing IMO that the spec settings of IstioOperator are different from the fields in the helm values
  • move more things from install config to runtime config to further simplify the install experience
    • e.g. a lot of today's cmdline args and ENV vars could be part of an 'experimental' section in meshconfig instead- there's really no reason why an administrator has to configure them at install time

@bleggett
Copy link
Contributor

bleggett commented May 10, 2023

1 on treating IstioOperator as a non-visible implementation detail (however we do that) - we already tell people not to use it, effectively, following that up by explicitly making it less visible in docs/user-facing porcelain seems good.

@costinm
Copy link
Contributor

costinm commented May 10, 2023

Istioctl install is using the same config as operator. Which is a kind of CRD that is not applied or reconciled.

@costinm
Copy link
Contributor

costinm commented May 10, 2023

I would add that with all refactorings and with ambient behaving in different way leaves an unknown part of the install API in an undeterministic state.

@bleggett
Copy link
Contributor

bleggett commented Aug 9, 2023

Not stale

@istio-policy-bot istio-policy-bot removed the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label Aug 9, 2023
@istio-policy-bot istio-policy-bot added the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label Nov 7, 2023
@hanxiaop
Copy link
Member

hanxiaop commented Nov 7, 2023

not stale

@istio-policy-bot istio-policy-bot removed the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label Nov 7, 2023
@howardjohn
Copy link
Member Author

related: #47838

@AndreaM12345
Copy link
Contributor

I think there are still users of the Istio Operator as its been in the docs. The issue seems to discuss that the initial step will be to remove from the documents the existence of the Istio Operator. Then the next steps are to remove the operator image and the associated testing. I was wondering what the timeline will look like for when the depreciation will be completed.

@zirain
Copy link
Member

zirain commented Jan 18, 2024

I think there are still users of the Istio Operator as its been in the docs.

maybe we can move IstioOperator controller to istio-ecosystem org?

@hanxiaop
Copy link
Member

I think there are still users of the Istio Operator as its been in the docs.

maybe we can move IstioOperator controller to istio-ecosystem org?

Isn't this still implies to users that using this operator is a valid approach with Istio, which in fact is a poor practice?

@AndreaM12345
Copy link
Contributor

I think there are still users of the Istio Operator as its been in the docs.

maybe we can move IstioOperator controller to istio-ecosystem org?

Isn't this still implies to users that using this operator is a valid approach with Istio, which in fact is a poor practice?

Using the istio-ecosystem does give users time to migrate away from the current Istio Operator. I think we need to be more clear what is the deprecation timeline and if there is a migration strategy.

@AndreaM12345
Copy link
Contributor

Also I think there was a plan to announce deprecation at KubeCon. Does that mean that the plan is to remove the Istio Operator from the docs after KubeCon?

@whitneygriffith
Copy link
Contributor

Also I think there was a plan to announce deprecation at KubeCon. Does that mean that the plan is to remove the Istio Operator from the docs after KubeCon?

We have decided to not make an announcement on deprecation until we have finalized a migration strategy

@dgn
Copy link
Contributor

dgn commented Mar 7, 2024

FYI we're working on https://github.com/istio-ecosystem/sail-operator now (initial commit to main branch coming soon), which will be a new operator maintained by us at Red Hat. that could potentially become a migration path for users of the legacy operator. we're sticking to the helm values API, so migration shouldn't be hard. in addition to what the legacy operator provided, sail-operator will support multiple istio versions and istio version upgrades, both automatic in-place upgrades and semi-automatic canary upgrades

@whitneygriffith
Copy link
Contributor

FYI we're working on https://github.com/istio-ecosystem/sail-operator now (initial commit to main branch coming soon), which will be a new operator maintained by us at Red Hat. that could potentially become a migration path for users of the legacy operator. we're sticking to the helm values API, so migration shouldn't be hard. in addition to what the legacy operator provided, sail-operator will support multiple istio versions and istio version upgrades, both automatic in-place upgrades and semi-automatic canary upgrades

This is awesome! Please write an Istio blog post about it

@istio-policy-bot istio-policy-bot added the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label Aug 19, 2024
@istio-policy-bot istio-policy-bot added the lifecycle/automatically-closed Indicates a PR or issue that has been closed automatically. label Sep 3, 2024
@bleggett bleggett reopened this Sep 4, 2024
@istio-policy-bot istio-policy-bot removed the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label Sep 4, 2024
@bleggett bleggett added the lifecycle/staleproof Indicates a PR or issue has been deemed to be immune from becoming stale and/or automatically closed label Sep 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/environments lifecycle/automatically-closed Indicates a PR or issue that has been closed automatically. lifecycle/staleproof Indicates a PR or issue has been deemed to be immune from becoming stale and/or automatically closed
Projects
None yet
Development

No branches or pull requests

9 participants