Feature Overview
Insights Advisor for OpenShift is integrated within OpenShift Cluster Manager. This has some limitations for adding new features and also for sharing codebase between RHEL Advisor and OCM Insights Advisor tab. Insights Advisor for OpenShift lacks certain features from the RHEL UI, the codebase is not 1:1 clone.
As a customer of Insights I will have same/very similar user experience with Insights for OpenShift and Insights for RHEL. The workflows will share the main concepts, the UI elements will be same and features introduced to Advisor will be automatically considered for both all supported platforms.
As OpenShift users I will still see integrations of Insights Advisor within OpenShift Cluster Manager that shows aggregated information for customer account and single cluster view on Advisor data. These integration will point to new Insights Advisor for OpenShift app that will be tightly integrated into OpenShift Cluster Manager.

• Note: The application will be reusing the codebase but will run as a separate app for OpenShift. THere's no intent to merge RHEL and OpenShift workflows into a single app.

Goals

• Q2CY21: Explore possibility to unify codebase between RHEL Advisor and OCM Insights Advisor tab. Identify architecture misalignments, create UI mockups to merge the two existing UIs.
• Q3CY21: Integrate OpenShift into Advisor codebase, standup the Insights Advisor for OpenShift application and change integration in OpenShift Cluster manager to point at the new app
• Q4CY21: Deliver missing screen of Insights Advisor for OpenShift (Systems and Recommendations views)

Requirements

• UX overview of UI elements in both UIs - Marie Doruskova
• Architecture overview/misalignments for both UIs - Jan Zeleny [~fjansen]

Benefits

• Feature parity between RHEL and OpenShift
• Adopting new features developed by RHEL Advisor team quicker
• Smaller maintenance cost

Questions to answer...

• Possible deviations between OpenShift and RHEL
• Remediation workflow different between OpenShift and RHEL

Out of Scope

• Single app that combines RHEL hosts and OpenShift clusters. Goal is still to differentiate between platforms and offer view only for a single platform.
• Direct/Supervised remediations and integration of remediations with Advanced Cluster Manager (as a Service)

Background, and strategic fit

• Insights Advisor for OpenShift follows the goal to introduce multiple applications that add value for OpenShift customers under the Insights brand. The current UI and integration of Advisor into OpenShift cluster manager doesn't follow pattern that other Insights for OpenShift applications can/will follow.

Documentation Considerations

• OCM documentation is impacted, existing workflows described in OCM documentation will persist. The placement of the application within OCM will be different.

OCP/Telco Definition of Done
Feature Template descriptions and documentation.

Feature Overview.

Early customer feedback is that they see SNO as a great solution covering smaller footprint deployment, but are wondering what is the evolution story OpenShift is going to provide where more capacity or high availability are needed in the future.

While migration tooling (moving workload/config to new cluster) could be a mid-term solution, customer desire is not to include extra hardware to be involved in this process.

 For Telecommunications Providers, at the Far Edge they intend to start small and then grow. Many of these operators will start with a SNO-based DU deployment as an initial investment, but as DUs evolve, different segments of the radio spectrum are added, various radio hardware is provisioned and features delivered to the Far Edge, the Telecommunication Providers desire the ability for their Far Edge deployments to scale up from 1 node to 2 nodes to n nodes. On the opposite side of the spectrum from SNO is MMIMO where there is a robust cluster and workloads use HPA.

Goals

• Provide the capability to expand a single replica control plane topology to host more workloads capacity - add worker
• Provide the capability to expand a single replica control plane to be a highly available control plane
• To satisfy MMIMO Telecommunications providers will want the ability to scale a SNO to a multi-node cluster that can support HPA.
• Telecommunications providers do not want workload (DU specifically) downtime when migrating from SNO to a multi-node cluster.
• Telecommunications providers wish to be able to scale from one to two or more nodes to support a variety of radio hardware.
• Support CP scaling (CP HA) for 2 node cluster, 3 node cluster and n node cluster. As the number of nodes in the cluster increases so does the failure domain of the cluster. The cluster is now supporting more cell sectors and therefore has more of a need for HA and resiliency including the cluster CP.

Requirements

• TBD

(Optional) Use Cases

This Section:

• Main success scenarios - high-level user stories
• Alternate flow/scenarios - high-level user stories
• ...

Questions to answer…

• ...

Out of Scope

• …

Background, and strategic fit

This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

• ...

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview

This Feature is a general "catch all" for the time being. There are a number of existing priorities from Q1 that should be aligned with existing priorities below but if not, assign to this feature as needed.

Goals

In order to get a better overall portfolio view, we'll leverage this Feature to gather work that doesn't fall into other existing priorities on this board. As this list grows, the portfolio priority grooming team will look to split out or handle appropriately.

Requirements

A list of specific needs or objectives that a Feature must deliver to satisfy the Feature. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts.  If a non MVP requirement slips, it does not shift the feature.

(Optional) Use Cases

< How will the user interact with this feature? >

< Which users will use this and when will they use it? >

< Is this feature used as part of current user interface? >

Out of Scope

Background, and strategic fit

< What does the person writing code, testing, documenting need to know? >

Assumptions

< Are there assumptions being made regarding prerequisites and dependencies?>

< Are there assumptions about hardware, software or people resources?>

Customer Considerations

< Are there specific customer environments that need to be considered (such as working with existing h/w and software)?>

Documentation Considerations

< What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)? >

<What does success look like?>

< Does this feature have doc impact? Possible values are: New Content, Updates to existing content,  Release Note, or No Doc Impact?>

 <If unsure and no Technical Writer is available, please contact Content Strategy. If yes, complete the following.>

• <What concepts do customers need to understand to be successful in [action]?>
• <How do we expect customers will use the feature? For what purpose(s)?>
• <What reference material might a customer want/need to complete [action]?>
• <Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available. >
• <What is the doc impact (New Content, Updates to existing content, or Release Note)?>

Questions

Problem:

Certain Insights Advisor features differentiate between RHEL and OCP advisor

Goal:

Address top priority UI misalignments between RHEL and OCP advisor. Address UI features dropped from Insights ADvisor for OCP GA.

Scope:

Specific tasks and priority of them tracked in https://issues.redhat.com/browse/CCXDEV-7432

Managing PVs at scale for a fleet creates difficulties where "one size does not fit all". The ability for SRE to deploy prometheus with PVs and have retention based an on a desired size would enable easier management of these volumes across the fleet. 

The prometheus-operator exposes retentionSize.

This is a feature request to enable this configuration option via CMO cluster-monitoring-config ConfigMap.

cc Simon Pasquier  

Problem Alignment

The Problem

Today, all configuration for setting individual, for example, routing configuration is done via a single configuration file that only admins have access to. If an environment uses multiple tenants and each tenant, for example, has different systems that they are using to notify teams in case of an issue, then someone needs to file a request w/ an admin to add the required settings.

That can be bothersome for individual teams, since requests like that usually disappear in the backlog of an administrator. At the same time, administrators might get tons of requests that they have to look at and prioritize, which takes them away from more crucial work.

We would like to introduce a more self service approach whereas individual teams can create their own configuration for their needs w/o the administrators involvement.

Last but not least, since Monitoring is deployed as a Core service of OpenShift there are multiple restrictions that the SRE team has to apply to all OSD and ROSA clusters. One restriction is the ability for customers to use the central Alertmanager that is owned and managed by the SRE team. They can't give access to the central managed secret due to security concerns so that users can add their own routing information.

High-Level Approach

Provide a new API (based on the Operator CRD approach) as part of the Prometheus Operator that allows creating a subset of the Alertmanager configuration without touching the central Alertmanager configuration file.

Please note that we do not plan to support additional individual webhooks with this work. Customers will need to deploy their own version of the third party webhooks.

Goal & Success

• Allow users to deploy individual configurations that allow setting up Alertmanager for their needs without an administrator.

Solution Alignment

Key Capabilities

• As an OpenShift administrator, I want to control who can CRUD individual configuration so that I can make sure that any unknown third person can touch the central Alertmanager instance shipped within OpenShift Monitoring.
• As a team owner, I want to deploy a routing configuration to push notifications for alerts to my system of choice.

Key Flows

Team A wants to send all their important notifications to a specific Slack channel.

• Administrator gives permission to Team A to allow creating a new configuration CR in their individual namespace.
• Team A creates a new configuration CR.
• Team A configures what alerts should go into their Slack channel.
• Open Questions & Key Decisions (optional)
• Do we want to improve anything inside the developer console to allow configuration?

Copy/paste from [_https://github.com/openshift-cs/managed-openshift/issues/60_]

Which service is this feature request for?
OpenShift Dedicated and Red Hat OpenShift Service on AWS

What are you trying to do?
Allow ROSA/OSD to integrate with AWS Managed Prometheus.

Describe the solution you'd like
Remote-write of metrics is supported in OpenShift but it does not work with AWS Managed Prometheus since AWS Managed Prometheus requires AWS SigV4 auth.

• Note that Prometheus supports AWS SigV4 since v2.26 and OpenShift 4.9 uses v2.29.

Describe alternatives you've considered
There is the workaround to use the "AWS SigV4 Proxy" but I'd think this is not properly supported by RH.
https://mobb.ninja/docs/rosa/cluster-metrics-to-aws-prometheus/

Additional context
The customer wants to use an open and portable solution to centralize metrics storage and analysis. If they also deploy to other clouds, they don't want to have to re-configure. Since most clouds offer a Prometheus service (or it's easy to self-manage Prometheus), app migration should be simplified.

Feature Overview

We drive OpenShift cross-market customer success and new customer adoption with constant improvements and feature additions to the existing capabilities of our OpenShift Core Networking (SDN and Network Edge). This feature captures that natural progression of the product.

Goals

• Feature enhancements (performance, scale, configuration, UX, ...)
• Modernization (incorporation and productization of new technologies)

Requirements

• Core Networking Stability
• Core Networking Performance and Scale
• Core Neworking Extensibility (Multus CNIs)
• Core Networking UX (Observability)
• Core Networking Security and Compliance

In Scope

• Network Edge (ingress, DNS, LB)
• SDN (CNI plugins, openshift-sdn, OVN, network policy, egressIP, egress Router, ...)
• Networking Observability

Out of Scope

There are definitely grey areas, but in general:

• CNV
• Service Mesh
• CNF

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

tldr: three basic claims, the rest is explanation and one example

1. We cannot improve long term maintainability solely by fixing bugs.
2. Teams should be asked to produce designs for improving maintainability/debugability.
3. Specific maintenance items (or investigation of maintenance items), should be placed into planning as peer to PM requests and explicitly prioritized against them.

While bugs are an important metric, fixing bugs is different than investing in maintainability and debugability. Investing in fixing bugs will help alleviate immediate problems, but doesn't improve the ability to address future problems. You (may) get a code base with fewer bugs, but when you add a new feature, it will still be hard to debug problems and interactions. This pushes a code base towards stagnation where it gets harder and harder to add features.

One alternative is to ask teams to produce ideas for how they would improve future maintainability and debugability instead of focusing on immediate bugs. This would produce designs that make problem determination, bug resolution, and future feature additions faster over time.

I have a concrete example of one such outcome of focusing on bugs vs quality. We have resolved many bugs about communication failures with ingress by finding problems with point-to-point network communication. We have fixed the individual bugs, but have not improved the code for future debugging. In so doing, we chase many hard to diagnose problem across the stack. The alternative is to create a point-to-point network connectivity capability. this would immediately improve bug resolution and stability (detection) for kuryr, ovs, legacy sdn, network-edge, kube-apiserver, openshift-apiserver, authentication, and console. Bug fixing does not produce the same impact.

We need more investment in our future selves. Saying, "teams should reserve this" doesn't seem to be universally effective. Perhaps an approach that directly asks for designs and impacts and then follows up by placing the items directly in planning and prioritizing against PM feature requests would give teams the confidence to invest in these areas and give broad exposure to systemic problems.

Relevant links:

• Documentation:
  • Edge Diagnostics Scratchpad, our team's internal diagnostic guide.
  • Troubleshooting OCP networking issues - The complete guide, the SDN team's diagnostic guide.
  • Linux Performance, Brendan Gregg's guide to analyzing Linux performance issues.
  • RFC: A proper feedback loop on Alerts.
  • OpenShift Router Reload Technical Overview on Access.
  • Performance Scaling HAProxy with OpenShift on Access.
  • How to collect worker metrics to troubleshoot CPU load, memory pressure and interrupt issues and networking on worker nodes in OCP 4 on Access.
  • OpenShift Performance and Scale Knowledge Base on Mojo, results from OpenShift scalability testing.
  • Scalability and performance, OCP 4.5 documentation about the router's currently known scalability limits.
  • Scaling OpenShift Container Platform HAProxy Router, OCP 3.11 documentation about the manual performance configuration that was possible in OCP 3.
  • Timing web requests with cURL and Chrome from the Cloudflare blog.
  • tcpdump advanced filters, some useful tcpdump commands.
  • OpenShift SDN - Networking, OCP 3.11 documentation on the SDN (useful background reading).
  • Ingress Operator and Controller Status Conditions, design document for improved status condition reporting.
  • Observability tips for HAProxy, a slide deck by Willy Tarreau.
  • Interesting Traces - Out of Order versus Retransmissions, analysis using tshark.
  • The PCP Book: A Complete Documentation of Performance Co-Pilot, by Yogesh Babar.
  • Debugging kernel networking bug, brief guide to using SystemTap on RHCOS.
  • Troubleshooting throughput issues from the OCP 4.5 documentation.
  • Troubleshooting OpenShift Clusters and Workloads.
  • Red Hat Enterprise Linux Network Performance Tuning Guide (PDF).
  • openshift/enhancements#289 stability: point to point network check, a diagnostic built into the kube-apiserver operator.
• Diagnostic tools:
  • dropwatch to watch for packet drops.
  • ethtool to check NIC configuration.
  • iovisor/bcc: BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more to trace and diagnose various issues in the networking stack.
  • r-curler to gather timing information about HTTP/HTTPS connections.
  • route-monitor, to monitor routes for reachability.
  • hping(3), a programmable packet generator.
  • OpenTracing / Jaeger in OpenShift.
  • node-problem-detector, a possible integration point for new diagnostics.
  • Using SystemTap by Brendan Gregg.
  • DTrace SystemTap cheatsheet (PDF).
• Visualization and more sophisticated diagnostic tools:
  • eldadru/ksniff, kubectl plugin for tcpdump & Wireshark.
  • ironcladlou/ditm, Dan's "Dan in the Middle" tool.
  • Skydive, network diagnostic and visualization tool.
  • ali, a "load testing tool capable of performing real-time analysis" with visualization.
• Testing tools:
  • stress-ng, a general stress-loading tool (CPU, filesystem, network, ...).
  • mb, the networking benchmarking tool written and used by Jiri Mencak from our Perf+Scale team.
• Case studies:
  • BZ1763206 is an example of diagnosing DNS latency/timeouts.
  • BZ1829779 Investigation details the diagnosis of route latency.
  • BZ1845545 is an example of diagnosing misconfigured DNS for an external LB.
  • Debugging network stalls on Kubernetes, from the GitHub Blog, about diagnosing Kubernetes performance issues related to ksoftirqd.

Feature Overview

Plugin teams need a mechanism to extend the OCP console that is decoupled enough so they can deliver at the cadence of their projects and not be forced in to the OCP Console release timelines.

The OCP Console Dynamic Plugin Framework will enable all our plugin teams to do the following:

• Extend the Console
• Deliver UI code with their Operator
• Work in their own git Repo
• Deliver at their own cadence

Goals

• • Operators can deliver console plugins separate from the console image and update plugins when the operator updates.
  • The dynamic plugin API is similar to the static plugin API to ease migration.
  • Plugins can use shared console components such as list and details page components.
  • Shared components from core will be part of a well-defined plugin API.
  • Plugins can use Patternfly 4 components.
  • Cluster admins control what plugins are enabled.
  • Misbehaving plugins should not break console.
  • Existing static plugins are not affected and will continue to work as expected.

Out of Scope

• • Initially we don't plan to make this a public API. The target use is for Red Hat operators. We might reevaluate later when dynamic plugins are more mature.
  • We can't avoid breaking changes in console dependencies such as Patternfly even if we don't break the console plugin API itself. We'll need a way for plugins to declare compatibility.
  • Plugins won't be sandboxed. They will have full JavaScript access to the DOM and network. Plugins won't be enabled by default, however. A cluster admin will need to enable the plugin.
  • This proposal does not cover allowing plugins to contribute backend console endpoints.

Requirements

 
 Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?

• Does this feature have doc impact?  

• New Content, Updates to existing content,  Release Note, or No Doc Impact

• If unsure and no Technical Writer is available, please contact Content Strategy.

• What concepts do customers need to understand to be successful in [action]?

• How do we expect customers will use the feature? For what purpose(s)?

• What reference material might a customer want/need to complete [action]?

• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.

• What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview

• This Section:* High-Level description of the feature ie: Executive Summary

• Note: A Feature is a capability or a well defined set of functionality that delivers business value. Features can include additions or changes to existing functionality. Features can easily span multiple teams, and multiple releases.

Goals

• This Section:* Provide high-level goal statement, providing user context and expected user outcome(s) for this feature

• …

Requirements

• This Section:* A list of specific needs or objectives that a Feature must deliver to satisfy the Feature.. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts. If a non MVP requirement slips, it does not shift the feature.

(Optional) Use Cases

This Section: 

• Main success scenarios - high-level user stories

• Alternate flow/scenarios - high-level user stories

• ...

Questions to answer…

• ...

Out of Scope

• …

Background, and strategic fit

This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

• ...

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?

• Does this feature have doc impact?  

• New Content, Updates to existing content,  Release Note, or No Doc Impact

• If unsure and no Technical Writer is available, please contact Content Strategy.

• What concepts do customers need to understand to be successful in [action]?

• How do we expect customers will use the feature? For what purpose(s)?

• What reference material might a customer want/need to complete [action]?

• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.

• What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview

• As an infrastructure owner, I want a repeatable method to quickly deploy the initial OpenShift cluster.

• As an infrastructure owner, I want to install the first (management, hub, “cluster 0”) cluster to manage other (standalone, hub, spoke, hub of hubs) clusters.

Goals

• Enable customers and partners to successfully deploy a single “first” cluster in disconnected, on-premises settings

Requirements

4.11 MVP Requirements

• Customers and partners needs to be able to download the installer
• Enable customers and partners to deploy a single “first” cluster (cluster 0) using single node, compact, or highly available topologies in disconnected, on-premises settings
• Installer must support advanced network settings such as static IP assignments, VLANs and NIC bonding for on-premises metal use cases, as well as DHCP and PXE provisioning environments.
• Installer needs to support automation, including integration with third-party deployment tools, as well as user-driven deployments.
• In the MVP automation has higher priority than interactive, user-driven deployments.
• For bare metal deployments, we cannot assume that users will provide us the credentials to manage hosts via their BMCs.
• Installer should prioritize support for platforms None, baremetal, and VMware.
• The installer will focus on a single version of OpenShift, and a different build artifact will be produced for each different version.
• The installer must not depend on a connected registry; however, the installer can optionally use a previously mirrored registry within the disconnected environment.

Use Cases

• As a Telco partner engineer (Site Engineer, Specialist, Field Engineer), I want to deploy an OpenShift cluster in production with limited or no additional hardware and don’t intend to deploy more OpenShift clusters [Isolated edge experience].
• As a Enterprise infrastructure owner, I want to manage the lifecycle of multiple clusters in 1 or more sites by first installing the first  (management, hub, “cluster 0”) cluster to manage other (standalone, hub, spoke, hub of hubs) clusters [Cluster before your cluster].
• As a Partner, I want to package OpenShift for large scale and/or distributed topology with my own software and/or hardware solution.
• As a large enterprise customer or Service Provider, I want to install a “HyperShift Tugboat” OpenShift cluster in order to offer a hosted OpenShift control plane at scale to my consumers (DevOps Engineers, tenants) that allows for fleet-level provisioning for low CAPEX and OPEX, much like AKS or GKE [Hypershift].
• As a new, novice to intermediate user (Enterprise Admin/Consumer, Telco Partner integrator, RH Solution Architect), I want to quickly deploy a small OpenShift cluster for Poc/Demo/Research purposes.

Questions to answer…

Out of Scope

Out of scope use cases (that are part of the Kubeframe/factory project):

• As a Partner (OEMs, ISVs), I want to install and pre-configure OpenShift with my hardware/software in my disconnected factory, while allowing further (minimal) reconfiguration of a subset of capabilities later at a different site by different set of users (end customer) [Embedded OpenShift].
• As an Infrastructure Admin at an Enterprise customer with multiple remote sites, I want to pre-provision OpenShift centrally prior to shipping and activating the clusters in remote sites.

Background, and strategic fit

• This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

1. The user has only access to the target nodes that will form the cluster and will boot them with the image presented locally via a USB stick. This scenario is common in sites with restricted access such as government infra where only users with security clearance can interact with the installation, where software is allowed to enter in the premises (in a USB, DVD, SD card, etc.) but never allowed to come back out. Users can't enter supporting devices such as laptops or phones.
2. The user has access to the target nodes remotely to their BMCs (e.g. iDrac, iLo) and can map an image as virtual media from their computer. This scenario is common in data centers where the customer provides network access to the BMCs of the target nodes.
3. We cannot assume that we will have access to a computer to run an installer or installer helper software.

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

References

• Bootable Ephemeral Installer Image for Disconnected Cluster 0 Deployments

Feature Overview

The AWS-specific code added in OCPPLAN-6006 needs to become GA and with this we want to introduce a couple of Day2 improvements.
Currently the AWS tags are defined and applied at installation time only and saved in the infrastructure CRD's status field for further operator use, which in turn just add the tags during creation.

Saving in the status field means it's not included in Velero backups, which is a crucial feature for customers and Day2.
Thus the status.resourceTags field should be deprecated in favour of a newly created spec.resourceTags with the same content. The installer should only populate the spec, consumers of the infrastructure CRD must favour the spec over the status definition if both are supplied, otherwise the status should be honored and a warning shall be issued.

Being part of the spec, the behaviour should also tag existing resources that do not have the tags yet and once the tags in the infrastructure CRD are changed all the AWS resources should be updated accordingly.

On AWS this can be done without re-creating any resources (the behaviour is basically an upsert by tag key) and is possible without service interruption as it is a metadata operation.

Tag deletes continue to be out of scope, as the customer can still have custom tags applied to the resources that we do not want to delete.

Due to the ongoing intree/out of tree split on the cloud and CSI providers, this should not apply to clusters with intree providers (!= "external").

Once confident we have all components updated, we should introduce an end2end test that makes sure we never create resources that are untagged.

After that, we can remove the experimental flag and make this a GA feature.

Goals

• Inclusion in the cluster backups
• Flexibility of changing tags during cluster lifetime, without recreating the whole cluster

Requirements

• This Section:* A list of specific needs or objectives that a Feature must deliver to satisfy the Feature.. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts. If a non MVP requirement slips, it does not shift the feature.

List any affected packages or components.

• Installer
• Cluster Infrastructure
• Storage
• Node
• NetworkEdge
• Internal Registry
• CCO

Feature Overview

Customers are asking for improvements to the upgrade experience (both over-the-air and disconnected). This is a feature tracking epics required to get that work done.  

Goals

1. Have an option to do upgrades in more discrete steps under admin control. Specifically, these steps are: 
   • Control plane upgrade
   • Worker nodes upgrade
   • Workload enabling upgrade (i..e. Router, other components) or infra nodes
2. Better visibility into any errors during the upgrades and documentation of what they error means and how to recover. 
3. An user experience around an end-2-end back-up and restore after a failed upgrade 
4. OTA-810  - Better Documentation: 
   • Backup procedures before upgrades. 
   • More control over worker upgrades (with tagged pools between user Vs admin)
   • The kinds of pre-upgrade tests that are run, the errors that are flagged and what they mean and how to address them. 
   • Better explanation of each discrete step in upgrades, and what each CVO Operator is doing and potential errors, troubleshooting and mitigating actions.

References

• OpenShift Upgrades Customer Issues Discussion -6/22/2023
• Select Customer Experiences and Feedback on Upgrades

OCP/Telco Definition of Done
Feature Template descriptions and documentation.
Feature Overview

• Connect OpenShift workloads to Google services with Google Workload Identity

• Enable customers to access Google services from workloads on OpenShift clusters using Google Workload Identity (aka WIF)
  https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity 

Goals

• Customers want to be able to manage and operate OpenShift on Google Cloud Platform with workload identity, much like they do with AWS + STS or Azure + workload identity.
• Customers want to be able to manage and operate operators and customer workloads on top of OCP on GCP with workload identity.

Requirements

• Add support to CCO for the Installation and Upgrade using both UPI and IPI methods with GCP workload identity.
• Support install and upgrades for connected and disconnected/restriction environments.
• Support the use of Operators with GCP workload identity with minimal friction.
• Support for HyperShift and non-HyperShift clusters.
• This Section:* A list of specific needs or objectives that a Feature must deliver to satisfy the Feature.. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts. If a non MVP requirement slips, it does not shift the feature.

(Optional) Use Cases

This Section:

• Main success scenarios - high-level user stories
• Alternate flow/scenarios - high-level user stories
• ...

Questions to answer…

• ...

Out of Scope

• …

Background, and strategic fit

This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

• ...

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview

Enable sharing ConfigMap and Secret across namespaces

Requirements

Questions to answer…

NA

Out of Scope

NA

Background, and strategic fit

Consumption of RHEL entitlements has been a challenge on OCP 4 since it moved to a cluster-based entitlement model compared to the node-based (RHEL subscription manager) entitlement mode. In order to provide a sufficiently similar experience to OCP 3, the entitlement certificates that are made available on the cluster (OCPBU-93) should be shared across namespaces in order to prevent the need for cluster admin to copy these entitlements in each namespace which leads to additional operational challenges for updating and refreshing them. 

Documentation Considerations

Questions to be addressed:
 * What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
 * Does this feature have doc impact?
 * New Content, Updates to existing content, Release Note, or No Doc Impact
 * If unsure and no Technical Writer is available, please contact Content Strategy.
 * What concepts do customers need to understand to be successful in [action]?
 * How do we expect customers will use the feature? For what purpose(s)?
 * What reference material might a customer want/need to complete [action]?
 * Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
 * What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview (aka. Goal Summary)  

Upstream Kuberenetes is following other SIGs by moving it's intree cloud providers to an out of tree plugin format, Cloud Controller Manager, at some point in a future Kubernetes release. OpenShift needs to be ready to action this change  

Goals (aka. expected user outcomes)

Bring together all the cloud controller managers (AWS, GCP, Azure), complete testing and prepare for final GA

Requirements (aka. Acceptance Criteria):

A list of specific needs or objectives that a feature must deliver in order to be considered complete.  Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc.  Initial completion during Refinement status.

Use Cases (Optional):

Include use case diagrams, main success scenarios, alternative flow scenarios.  Initial completion during Refinement status.

Questions to Answer (Optional):

Include a list of refinement / architectural questions that may need to be answered before coding can begin.  Initial completion during Refinement status.

Out of Scope

High-level list of items that are out of scope.  Initial completion during Refinement status.

Background

Provide any additional context is needed to frame the feature.  Initial completion during Refinement status.

Customer Considerations

Provide any additional customer-specific considerations that must be made when designing and delivering the Feature.  Initial completion during Refinement status.

Documentation Considerations

Provide information that needs to be considered and planned so that documentation will meet customer needs.  Initial completion during Refinement status.

Interoperability Considerations

Which other projects and versions in our portfolio does this feature impact?  What interoperability test scenarios should be factored by the layered products?  Initial completion during Refinement status.

Goal

Increase integration of Shipwright, Tekton, Argo CD in OpenShift GitOps with OpenShift platform and related products such as ACM.

Summary (PM+lead)

https://issues.redhat.com/browse/AUTH-2 revealed that, in prinicipal, Pod Security Admission is possible to integrate into OpenShift while retaining SCC functionality.

This epic is about the concrete steps to enable Pod Security Admission by default in OpenShift

Motivation (PM+lead)

Goals (lead)

• Enable Pod Security Admission in "restricted" policy level by default
• Migrate existing core workloads to comply to the "restricted" pod security policy level

Non-Goals (lead)

• Other OpenShift workloads must be migrated by the individual responsible teams.

Deliverables

Proposal (lead)

Enhancement - https://github.com/openshift/enhancements/pull/1010

User Stories (PM)

Dependencies (internal and external, lead)

Previous Work (lead)

Open questions (lead)

1. ...

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

• https://github.com/openshift/cluster-authentication-operator/pull/570
• https://github.com/openshift/cluster-csi-snapshot-controller-operator/pull/120
• https://github.com/openshift/cluster-samples-operator/pull/425

An epic we can duplicate for each release to ensure we have a place to catch things we ought to be doing regularly but can tend to fall by the wayside.

Epic Goal

HyperShift provisions OpenShift clusters with externally managed control-planes. It follows a slightly different process for provisioning clusters. For example, HyperShift uses cluster API as a backend and moves all the machine management bits to the management cluster.  

Why is this important?

showing machine management/cluster auto-scaling tabs in the console is likely to confuse users and cause unnecessary side effects. 

Definition of Done

• MachineConfig and MachineConfigPool should not be present, they should be either removed or hidden when the cluster is spawned using HyperShift. 
• Cluster Settings show say the control plane is externally managed and be read-only.
• Cluster Settings -> Configuration resources should be read-only, maybe hide the tab
• Some resources should go in an allowlist. Most will be hidden
• Review getting started steps

See Design Doc: https://docs.google.com/document/d/1k76JtRRHBdCCEjHPqKcYvbNVsuaGmRhWDLESWIm0mbo/edit#

Setup / Testing

It's based on the SERVER_FLAG controlPlaneTopology being set to External is really the driving factor here; this can be done in one of two ways:

• Locally via a Bridge Variable, export BRIDGE_CONTROL_PLANE_TOPOLOGY_MODE="External"
• Locally / OnCluster via modifying the window.SERVER_FLAGS.controlPlaneTopology to External in the dev tools

To test work related to cluster upgrade process, use a 4.10.3 cluster set on the candidate-4.10 upgrade channel using 4.11 frontend code.

Epic Goal

• Ensure that OCP Admin console can enable dark mode
• Mechanism for switching between light/dark mode, based on user preferences or User Settings in OCP
• Identify changes needed for OCP console's dynamic plugins
• Status as we work through the stories: https://docs.google.com/document/d/18ib0A9ZtSYo1e2aOtM2LtZcsNFs1aRCHXx6NAPsIAso/edit#

Why is this important?

• So the UX satisfies the current trands, where dark mode is becoming a standard for modern services.  

Acceptance Criteria

• OCP admin console must be rendered in a preferred mode based on `prefers-color-scheme` media query
• OCP admin console must be rendered in a preferred mode selected in the User Setting page
• Create an followup epic/story for and listing and tracking changes needed in OCP console's dynamic plugins

Dependencies (internal and external)

1. PatternFly - Dark mode PF variables

Previous Work (Optional):

1. Mike Coker has worked on a POC from the PF point of view on both the admin and dev console, and the screenshot results are listed below along with the repo branch. Also listed is a document covering some of the common issues found when putting together the admin console POC. https://github.com/mcoker/console/tree/dark-theme
   Background POC work completed for reference:

PatternFly Dark Theme Handbook: https://docs.google.com/document/d/1mRYEfUoOjTsSt7hiqjbeplqhfo3_rVDO0QqMj2p67pw/edit

Admin Console -> Workloads & Pods

• Screenshots: https://docs.google.com/presentation/d/1BoOpXpX96_uNUhVJiEqGUCZ-JMYh1A78fXJhk3H86K8/edit#slide=id.p
• Github branch: https://github.com/mcoker/console/tree/dark-theme
• Github PR for Dark Theme WIP by Vikram: https://github.com/openshift/console/pull/10947{}

Dev Console -> Gotcha pages: Observe Dashboard and Metrics, Add, Pipelines: builder, list, log, and run

• Screenshots: https://docs.google.com/presentation/d/13jagAf_JEu82hd_KpEHRmj25xCEqwImWpNufzVopuUs/edit#slide=id.p
• Github branch: https://github.com/openshift/console/pull/10815
• Evaluation: https://docs.google.com/spreadsheets/d/1fBCPPb4Z1sqUDKgQkpueGe8VQZo4pBUA9PV4vnFCnDw/edit#gid=0

Open questions::

1. Who should be responsible for updating DynamicPlugins to be able to render in dark mode?

An epic we can duplicate for each release to ensure we have a place to catch things we ought to be doing regularly but can tend to fall by the wayside.

This epic contains all the Dynamic Plugins related stories for OCP release-4.11 

Epic Goal

• Track all the stories under a single epic

Acceptance Criteria

Goal

• Add the ability for users to select supported but not recommended updates.
• Refine workflow when both "upgradeable=false" and "supported-but-not-recommended" updates occur

Background
RFE: for 4.10, Cincinnati and the cluster-version operator are adding conditional updates (a.k.a. targeted edge blocking): https://issues.redhat.com/browse/OTA-267

High-level plans in https://github.com/openshift/enhancements/blob/master/enhancements/update/targeted-update-edge-blocking.md#update-client-support-for-the-enhanced-schema

Example of what the oc adm upgrade UX will be in https://github.com/openshift/enhancements/blob/master/enhancements/update/targeted-update-edge-blocking.md#cluster-administrator.

The oc implementation landed via https://github.com/openshift/oc/pull/961.

Design

• Use case 01: "supported but not recommended" occurs to the latest version:
  • Add an info icon next to the version on update path with a pop-over to explain about why updating to this version is supported, but not recommended and a link to known risks
  • Identify the difference in "recommended" versions, "supported but not recommended" versions, and "blocked" versions (upgradeable=false) in the + more modal.
  • The latest version is pre-selected in the dropdown in the update modal with an inline alert to inform users about supported-but-not-recommended version with link to known risks. Users can choose to update to another recommended versions, update to a supported-but-not-recommended one, or wait.
  • The "recommended" and "supported but not recommended" updates are separated in the dropdown.
  • If a user selects a "recommended" update, the inline alert disappears.
• Use case 02: When both "upgradeable=false" and "supported but not recommended" occur:
  • Add an alert banner to explain why users shouldn’t update to the latest version and link to how to resolve on the cluster settings details page. Users have the options to resolve the issue, update to a patch version, or wait.
  • If users open the update modal without resolving the "upgradeable=false" issue, the next recommended version is pre-selected. An expandable link "View blocked versions (#)" is included under the dropdown to show "upgradeable=false" versions with resolve link.
  • If users resolve the "upgradeable=false" issue, the cluster settings page will change to use case 01

• • Question: Priority on changing the upgradeable=false alert banner in update modal and blocked versions in dropdown

See design doc: https://docs.google.com/document/d/1Nja4whdsI5dKmQNS_rXyN8IGtRXDJ8gXuU_eSxBLMIY/edit#

See marvel: https://marvelapp.com/prototype/h3ehaa4/screen/86077932

Epic Goal

• Add telemetry so that we know how image stream features are used.

Why is this important?

• We have a long standing epic to create image streams v2. We need to better understand how image streams are used today.

Scenarios

1. ...

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Epic Goal

• Improve CI testing of the image registry components.

Why is this important?

• The image registry, image API and the image pruner had a lot of tests removed during transition 4.0. This may make the platform less stable and/or slow down the team.

Scenarios

1. ...

Acceptance Criteria

• CI - tests should be more stable and have broader coverage

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

<--- Cut-n-Paste the entire contents of this description into your new Epic --->

Epic Goal

• ...

Why is this important?

• …

Scenarios

1. ...

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Goal

Remove Jenkins from the OCP Payload.

Problem

• Jenkins images are "non-trival in size, impact experience around OCP payloads
• Security advisories cannot be handled once, but against all actively supported OCP releases, adding to response time for handling said advisories
• Some customers may now want to upgrade Jenkins as OCP upgrades (making this configurable is more ideal)

Why is this important

• This is an engineering motivated item to reduce costs so we have more cycles for strategic work
• Aside from the team itself, top level OCP architects want this to reduce the image size, improve general OCP upgrade experience
• Sends a mix message with respect to what is startegic CI/CI when Jenkins is baked into OCP, but Tekton/Pipelines is an add-on, day 2 install sort of thing

Dependencies (internal and external)

See epic linking - need alternative non payload image available to provide relatively seamless migration

Also, the EP for this is approved and merged at https://github.com/openshift/enhancements/blob/master/enhancements/builds/remove-jenkins-payload.md

Estimate (xs, s, m, l, xl, xxl):

Questions:

• in addition to needing the CPaaS Image available first, have we confirmed the "deprecate first, then remove" requirements?  Per grooming we've assigned Rob Gormley the task https://issues.redhat.com/browse/JKNS-274 for tracking this down.  I will be reaching out to Ben Parees (he is currently on vacation) to confirm from an OCP staff engineer / architect perspective if his approval of https://github.com/openshift/enhancements/pull/841 is sufficient signoff from that end.

       PARTIAL ANSWER ^^:  confirmed with Ben Parees in https://coreos.slack.com/archives/C014MHHKUSF/p1646683621293839 that EP merging is currently sufficient OCP "technical leadership" approval.

Previous work

Customers

assuming none

Epic Goal

• Remove this UI from our stack that we cannot support.

Why is this important?

• Reduce support burden.
• Remove Bugzilla burden of addressing continuous CVEs found in this project.

Acceptance Criteria

• All Prometheus upstream UI links are removed
• Related documentation is updated
• Ports/routes etc configured to expose access to this UI are removed such that no configuration we provide enables access to this UI or its codepaths.
• There is no reason any CVEs found in this UI would ever require intervention by the Monitoring Team.

Dependencies (internal and external)

1. Make the Prometheus Targets information available in Console UI (https://issues.redhat.com/browse/MON-1079)

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

Epic Goal

• As a CFE team, we would like to enable query logging for all Prometheus read paths
• As part of this, we would like to enable audit & query logging for Prometheus Adapter(aggregated server audit log), Prometheus(query log) and ThanosQuerier(query log)

Why is this important?

• This would help all parties(customers, app-sres, CCX, monitoring team,..) to debug an overloaded Prometheus instance.

Scenarios

1. When a customer faces a high cpu consumption in any of the Prometheus instance, they can enable audit logging in Prometheus Adapter to see which component is calling metrics API
2. When a customer faces a high cpu consumption in any of the Prometheus instance, they can enable query logging in all Prometheus instances(PM & UWM) and ThanosQuerier to see which query is frequently executed
3. https://bugzilla.redhat.com/show_bug.cgi?id=1982302

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• Prometheus Adapter audit logs must be enabled by default
• Prometheus Adapter audit logs must be preserved after each CI run

Open questions::

1. Should we enable ThanosRuler query logs?

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Epic Goal

When users configure CMO to interact with systems outside of an OpenShift cluster, we want to provide an easy way to add the cluster ID to the data send.

Why is this important?

Technically this can be achieved today, by adding an identifying label to the remote_write configuration for a given cluster. The operator adding the remote_write integration needs to take care that the label is unique over the managed fleet of clusters. This however adds management complexity. Any given cluster already has a pseudo-unique datum, that can be used for this purpose.

• Starting in 4.9 we support the Prometheus remote_write feature to send metric data to a storage integration outside of the cluster similar to our own Telemetry service.
• In Telemetry we already use the cluster ID to distinguish the various clusters.
• For users of remote_write this could add an easy way to add such distinguishing information.

Scenarios

1. An organisation with multiple OpenShift clusters want to store their metric data centralized in a dedicated system and use remote_write in all their clusters to send this data. When querying their centralized storage, metadata (here a label) is needed to separate the data of the various clusters.
2. Service providers who manage multiple clusters for multiple customers via a centralized storage system need distinguishing metadata too. See https://issues.redhat.com/browse/OSD-6573 for example

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• Document how to use this feature

Dependencies (internal and external)

1. none

Previous Work (Optional):

1. none

Open questions::

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Implementation proposal:

Expose a flag in the CMO configuration, that is false by default (keeps backward compatibility) and when set to true will add the _id label to a remote_write configuration. More specifically it will be added to the top of a remote_write relabel_config list via the replace action. This will add the label as expect, but additionally a user could alter this label in a later relabel config to suit any specific requirements (say rename the label or add additional information to the value).
The location of this flag is the remote_write Spec, so this can be set for individual remote_write configurations.

Epic Goal

• Offer the option to double the scrape intervals for CMO controlled ServiceMonitors in single node deployments
• Alternatively automatically double the same scrape intervals if CMO detects an SNO setup

The potential target ServiceMonitors are:

• kubelet
• kube-state-metrics
• node-exporter
• etcd
• openshift-state-metrics

Why is this important?

• Reduce CPU usage in SNO setups
• Specifically doubling the scrape interval is important because:

1. we are confident that this will have the least chance to interfere with existing rules. We typically have rate queries over the last 2 minutes (no shorter time window). With 30 second scrape intervals (the current default) this gives us 4 samples in any 2 minute window. rate needs at least 2 samples to work, we want another 2 for failure tolerance. Doubling the scrape interval will still give us 2 samples in most 2 minute windows. If a scrape fails, a few rule evaluations might fail intermittently.
2. We expect a measureable reduction of CPU resources (see previous work)

Scenarios

1. RAN deployments (Telco Edge) are SNO deployments. In these setups a full CMO deployment is often not needed and the default setup consumes too many resources. OpenShift as a whole has only very limited CPU cycles available and too many cycles are spend on Monitoring

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Previous Work (Optional):

1. https://issues.redhat.com/browse/MON-1569

Open questions:

1. Whether doubling some scrape intervals reduces CPU usage to fit into the assigned budget

Non goals

• Allow arbitrarily long scrape intervals. This will interfere with alert and recoring rules
• Implement a global override to scrape intervals.

Description:

Openshift on RHV is composed of the following subproject the team maintains:

• installer
• cluster-api-provider-ovirt
• ovirt-csi-driver
• ovirt-csi-driver-operator
• terraform-provider-ovirt 

Each of those projects currently uses the generated oVirt API project go-ovirt.

This leads to a number of issues:

1. Duplicated code between the subprojects: Since the go-ovirt is a thin layer around the API then a lot of the code which interacts with oVirt is duplicated between the projects, which leads to all the classic duplication problems such as maintaining the project, lack of clear conventions, and so on.
2. Bad error handling and unclear errors:
   1. Since the go-ovirt is a thin layer there is a lot of error handling and checking which needs to be done, since a lot of the times it looks like a certain error should be ignored, it is never checked which could lead to unexpected situations.
   2. Since the errors which are returned from the oVirt Engine are sometimes unclear, when we return those errors to the users or log them is hard to understand what is the actual issue.
3. Lack of retries: sometimes an operation can take some time due to some condition that needs to be met, or an operation can fail due to infrastructure issues, the go-ovirt library doesn't contain any retry logic which means each client needs to implement its own retry logic which is not done at the moment and will cause more duplicated code.
4. Poor logging: The current go-ovirt library doesn't log anything, and all the logs come from the subprojects, this leads to:
   1. Inconsistent logging between the projects.
   2. Lack of logs.
5. Almost no test coverage:
   1. It's very hard to mock and write tests with go-ovirt since there are so many calls, but will be much easier to mock and write tests with go-ovirt-clent.
   2. go-ovirt only has rudimentary tests.

Then came go-ovirt-client, go-ovirt-client-log, go-ovirt-client-log-klog and k8sOVirtCredentialsMonitor to the rescue!

The go-ovirt-client is a wrapper around the go-ovirt which contains all the error handling/retry logic/logs/tests needed to provide a decent user experience and an easy-to-use API to the oVirt engine.

go-ovirt-client-log is a library to unify the logging logic between the projects, it is used by go-ovirt-client and should be used by all the sub-projects.

go-ovirt-client-log-klog is a companion library to go-ovirt-client-log enabling logging via the Kubernetes "klog" facility.

k8sOVirtCredentialsMonitor is a utility for monitoring the oVirt credentials secret, which will automatically update the ovirt credentials is they are changed. 

We aim to move all projects which are using the go-ovirt to use go-ovirt-client, go-ovirt-client-log and k8sOVirtCredentialsMonitor instead.

Benefits for the eng:

• Possible to write unit tests.
• Easier to maintain since less code duplication - reduce the amount of code.
• Test coverage exists on the ovirt-client as well.
• No(Less) bugs regarding operations that needed a retry or polling logic.
• Solves a number of existing bugs

Benefits for the customers:

• Clearer error messages and logs.
• Fewer bugs.

Acceptance criteria:

1. All sub-projects are not using go-ovirt directly - at least 90% of the calls to go-ovirt should be migrated to go-ovirt-client.
2. All sub-projects should use the corresponding go-ovirt-client-log for logging.
3. All csi-driver and cluster provide use k8sOVirtCredentialsMonitor.
4. CI tests are green for all components.

How to test:

1. QE regression - make sure all flows are still working.
2. Green CI on all jobs.
3. Keep an eye out for log messages that might confuse customers.

T-shirt size: M

Goal:

Provide an easy and successful experience for front end developers to build and deploy their applications

Why is it important?

Currently, the front end dev experience is not positive. It's much easier for them to use other platforms. Improving the front end dev experience will enable us to gain more marketshare

Use cases:

1. Need to be able to override the npm command when using Node Builder Image
2. Need to expose target port
3. Need access to the URL to access my application

Although we provide the ability for 2 & 3 today, the current journey does not match with the mental model of the front end developer

Acceptance criteria:

1. When importing an app, I should be able to easily provide the npm build and run commands
2. When opting in to create a route, the target port should be exposed without having to open any Advanced Options
3. After importing my app, if a route is exposed, I should be able to access/copy that URL

Dependencies (External/Internal):

Design Artifacts:

Desired UX experience

• enable user to provide the *Build Command* when Node Builder image is being used
• enable user to provide the *Run Command* when Node Builder image is being used
• 

• expose the Target Port under the *Create a route to the Application *rather than inside Show advanced Routing options

• NEED TO FINALIZE HOW TO PROVIDE THE ROUTE TO EASILY COPY – Inline Notification maybe? As well as side panel?

Note:

Problem:

This epic is mainly focused on the 4.10 Release QE activities

Goal:

1. Identify the scenarios for automation
2. Segregate the test Scenarios into smoke, Regression and other user stories
a. Update the https://docs.jboss.org/display/ODC/Automation+Status+Report
3. Align with layered operator teams for updating scripts
3. Work closely with dev team for epic automation
4. Create the automation scripts using cypress
5. Implement CI for nightly builds
6. Execute scripts on sprint basis

Why is it important?

To the track the QE progress at one place in 4.10 Release Confluence page

Use cases:

1. <case>

Acceptance criteria:

1. <criteria>

Dependencies (External/Internal):

Design Artifacts:

Exploration:

Note:

Goal:

This epic covers a number of customer requests(RFEs) as well as increases usability.

Why is it important?

Customer satisfaction as well as improved usability.

Acceptance Criteria

1. Allow user to re-arrange the resources which have ben added to nav by the user
2. Improved user experience (form based experience)
   1. Form based editing of Routes
   2. Form based creation and editing of Config Maps
   3. Form base creation of Deployments
3. Improved discovery
   1. Include Share my project on the Add page to increase discoverability
   2. NS Helm Chart Repo
      1. Add tile to Add page for discoverability
      2. Provide a form driven creation experience
      3. User should be able to switch back and forth from Form/YAML
      4. change the intro text to the below & have the link in the intro text bring up the full page form
         1. Browse for charts that help manage complex installations and upgrades. Cluster administrators can customize the content made available in the catalog. Alternatively, developers can try to configure their own custom Helm Chart repository.

Dependencies (External/Internal):

None

Exploration:

Miro board from Epic Exploration

Problem:

Currently we are only able to get limited telemetry from the Dev Sandbox, but not from any of our managed clusters or on prem clusters.

Goals:

1. Enable gathering segment telemetry whenever cluster telemetry is enabled on OSD clusters
2. Have our OSD clusters opt into telemetry by default
3. Work with PM & UX to identify additional metrics to capture in addition to what we have enabled currently on Sandbox.
4. Ability to get a single report from woopra across all of our Sandbox and OSD clusters.
5. Be able to generate a report including metrics of a single cluster or all clusters of a certain type ( sandbox, or OSD)

Why is it important?

In order to improve properly analyze usage and the user experience, we need to be able to gather as much data as possible.

Goal:
Enhance oc adm release new (and related verbs info, extract, mirror) with heterogeneous architecture support

Epic Goal

• Port all remaining Protractor tests to Cypress

Why is this important?

• Protractor is very hard to debug when tests fail/flake
• Once all protractor tests are ported we can remove all Protractor dependencies, scripts, and configuration files.
• Cypress has better debugging, plug-ins, and reporting tools

Acceptance Criteria

• CI - MUST be running successfully with tests automated

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Epic Goal

• Make the image registry distributed across availability zones.

Why is this important?

• The registry should be highly available and zone failsafe.

Scenarios

1. As an administrator I want to rely on a default configuration that spreads image registry pods across topology zones so that I don't suffer from a long recovery time (>6 mins) in case of a complete zone failure if all pods are impacted.

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.

Dependencies (internal and external)

1. Pod's topologySpreadConstraints

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: https://github.com/openshift/cluster-image-registry-operator/pull/730
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Epic Goal

• Update image registry dependencies (Kubernetes and OpenShift) to the latest versions.

Why is this important?

• New versions usually bring improvements that are needed by the registry and help with getting updates for z-stream.

Scenarios

1. As an OpenShift engineer, I want my components to use the versions of dependencies, so that they get fixes for known issues and can be easily updated in z-stream.

Acceptance Criteria

• CI - MUST be running successfully with tests automated

Dependencies (internal and external)

1. Kubernetes 1.24

Previous Work (Optional):

1. IR-210

Done Checklist

• CI - CI is running, tests are automated and merged.
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• QE - Test plans in Polarion: <link or reference to Polarion>

Epic Goal

• Provide a dedicated dashboard for NVIDIA GPU usage visualization in the OpenShift Console.

Why is this important?

• Customers that use GPUs in their clusters usually have the GPU workloads as the main purpose of their cluster. As such, it makes much more sense to have the details about the usage they are doing of GPGPU resources AND CPU/RAM rather than just CPU/RAM

Scenarios

1. As an admin of a cluster dedicated to data science, I want to quickly find out how much of my very costly resources are currently in use and if things are getting queued due to lack of resources

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.

Dependencies (internal and external)

1. The NVIDIA GPU Operator must export to prometheus the relevant data

Open questions::

1. Will NVIDIA agree to these extra data exports in their GPU Operator?

I asked Zvonko Kaiser and he seemed open to it. I need to confirm with Shiva Merla

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

<--- Cut-n-Paste the entire contents of this description into your new Epic --->

Epic Goal

• Run OpenShift builds that do not execute as the "root" user on the host node.

Why is this important?

• OpenShift builds require an elevated set of capabilities to build a container image
• Builds currently run as root to maintain adequate performance
• Container workloads should run as non-root from the host's perspective. Containers running as root are a known security risk.
• Builds currently run as root and require a privileged container. See BUILD-225 for removing the privileged container requirement.

Scenarios

1. Run BuildConfigs in a multi-tenant environment
2. Run BuildConfigs in a heightened security environment/deployment

Acceptance Criteria

• Developers can opt into running builds in a cri-o user namespace by providing an environment variable with a specific value.
• When the correct environment variable is provided, builds run in a cri-o user namespace, and the build pod does not require the "privileged: true" security context.
• User namespace builds can pass basic test scenarios for the Docker and Source strategy build.
• Steps to run unprivileged builds are documented.

Dependencies (internal and external)

1. Buildah supports running inside a non-privileged container
2. CRI-O allows workloads to opt into running containers in user namespaces.

Previous Work (Optional):

1. BUILD-225 - remove privileged requirement for builds.

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Problem:

Complete all the 4.9 epic features automation user stories and merge it to master branch.

Goal:

4.9 epics automation completion

Why is it important?

Tech debt should be completed

Use cases:

1. <case>

Acceptance criteria:

Create the pr's for 4.9 epic user stories automation
Review it
Merge it to 4.10 master branch and 4.9 master branch

Dependencies (External/Internal):

Design Artifacts:

Exploration:

Note:

Node Sass is deprecated. See https://github.com/sass/node-sass

• https://github.com/openshift/console/pull/10149

Acceptance criteria:

• All tests (including e2e) pass
• No regressions are introduced
• openshift/api points to a recent commit on the master branch

• https://github.com/openshift/cluster-image-registry-operator/pull/728

• https://github.com/openshift/cluster-autoscaler-operator/pull/226

Add a Makefile rule in CMO to execute all the different rule that are used for verification and validation. Currenctly, some of them might not be at the right place, for example `check-assets` which is part of `generate` despite not being responsible of any generation. https://github.com/openshift/cluster-monitoring-operator/pull/1151/files#r629371735

DoD:

• Add a new rule in CMO to handle verification
• Add a CI job for this rule

• https://github.com/openshift/cluster-monitoring-operator/pull/1499
• https://github.com/openshift/cluster-monitoring-operator/pull/1492

• https://github.com/openshift/cluster-monitoring-operator/pull/1338

[Updated story request]

Decision is to always display Red Hat OpenShift logo for OCP instead of conditionally. And also update the OCP login, errors, providers templates. https://openshift.github.io/oauth-templates/

Related note in comments.

[Original request]

If the ACM or the ACS dynamic plugin is enabled and there is not a custom branding set, then the default "Red Hat Openshift" branding should be shown.

This was identified as an issue during the Hybrid Console Scrum on 11/15/20201

PRs associated with this change

https://github.com/openshift/console/pull/10940 [merged]

https://github.com/openshift/oauth-templates/pull/20 [merged]

https://github.com/openshift/cluster-authentication-operator/pull/540 [merged]

• https://github.com/openshift/console/pull/10940

The static authorizer feature has landed in upstream kube-rbac-proxy. Lets use it by configuring a static authorizer for all requests that hit a /metrics endpoint.

DoD:

• Downstream kube-rbac-proxy is synced.
• All CMO operands are configured with static authorization.
• Bugzillas created for all non-monitoring components using kube-rbac-proxy for metrics authn/authz.

• https://github.com/openshift/cluster-monitoring-operator/pull/1318

New release https://github.com/kubernetes/kube-state-metrics/releases

• https://github.com/openshift/kube-state-metrics/pull/61

The CMO e2e tests create a bunch of resources. These should be cleaned up on a successful run. However:

• Some test failures leave the create resource behind, which have to be cleaned up before a re-run.
• There have been developer reports that even successful runs don't tidy up everything.

In a CI context this is rarely a problem, however running the tests locally can be made quite awkward, especially repeated runs on the same cluster.

We should tag all resources created by the e2e tests with a label (app.kubernetes.io/created-by: cmo-e2e-test).
This will allow easy cleanup by deleting all resources with that label and will allow for checking proper clean-up.

DoD:
All e2e resources get properly tagged.
It is straight forward to ensure that future code changes don't skip adding this tag.

• https://github.com/openshift/cluster-monitoring-operator/pull/1397

• https://github.com/openshift/cluster-monitoring-operator/pull/1379

 As mentioned in [1], the cluster monitoring operator doesn't define the relatedObjects field in the ClusterOperator manifest which is initially deployed by CVO [2].
If the CMO pod fails to start, the must-gather might miss information from the monitoring namespace. Note that once CMO runs, it will update the initial ClusterOperator object with the proper information [3].

[1] http://mailman-int.corp.redhat.com/archives/aos-devel/2021-May/msg00139.html
[2] https://github.com/openshift/cluster-monitoring-operator/blob/master/manifests/0000_50_cluster-monitoring-operator_06-clusteroperator.yaml
[3] https://github.com/openshift/cluster-monitoring-operator/blob/a6bc9824035ceb8dbfe7c53cf0c138bfb2ec5643/pkg/client/status_reporter.go#L49-L63

• https://github.com/openshift/cluster-monitoring-operator/pull/1483

Goal

We have several use cases where dynamic plugins need to proxy to another service on the cluster. One example is the Helm plugin. We would like to move the backend code for Helm to a separate service on the cluster, and the Helm plugin could proxy to that service for its requests. This is required to make Helm a dynamic plugin. Similarly if we want to have ACM contribute any views through dynamic plugins, we will need a way for ACM to proxy to its services (e.g., for Search).

It's possible for plugins to make requests to services exposed through routes today, but that has several problems:

1. It requires that the service be exposed outside the cluster, which is not always desired.
2. It requires the service support CORS headers for the console.
3. There is no way to specify a CA file for the route if it's not trusted by the browser.
4. Plugins will not have access to the user's access token on the client, which means that there is no simple way to handle auth.

Plugins need a way to declare in-cluster services that they need to connect to. The console backend will need to set up proxies to those services on console load. This also requires that the console operator be updated to pass the configuration to the console backend.

This work will apply only to single clusters.

Open Questions

• What happens when a multitenant isolated network policy is configured on the cluster?

https://docs.openshift.com/container-platform/4.7/networking/network_policy/multitenant-network-policy.html

• How do we (and can we?) support this for multi-cluster where console is running on a different hub cluster?
• Do we need to auth for all requests?

Acceptance Criteria

• Plugins can declare a service to proxy to in the ConsolePlugin resource
• Plugins can specify a CA cert for the service
• Console falls back to the service signing CA if none is specified
• Plugins have a way of specifying whether the user's authentication token is included in requests through the service proxy
• Dynamic plugin enhancement is updated with the implementation details
• Support for server-side events (SSE) for ACM
• Add support, or a flag, if auth is needed for each request.  

cc Ali Mobrem [~christianmvogt]

• https://github.com/openshift/console/pull/10215
• https://github.com/openshift/console-operator/pull/603

As per [1], the jsonnet code for managing thanos-ruler resources should reuse the upstream kube-thanos project.

[1] https://github.com/openshift/cluster-monitoring-operator/blob/399c84dbca596b611b0c30a0d2df63a5d2b0b8cc/jsonnet/components/thanos-ruler.libsonnet#L1

• https://github.com/openshift/cluster-monitoring-operator/pull/1478

Test description

• https://github.com/openshift/driver-toolkit/pull/77

*USER STORY:*

 As a customer or OpenShift engineer, I want to see the user agent for anything calling from OpenShift -> vSphere to eliminate troubleshooting guesswork.

*DESCRIPTION:*

A question in #forum-vmware was raised where we identified that the user-agent may not be configured for all OpenShift components calling to vSphere API.

https://coreos.slack.com/archives/CH06KMDRV/p1627368902058800

*Required:*

Audit of OpenShift components calling to vSphere API to make sure user agent strings are set appropriately.

*Nice to have:*

How can this be prevented in the future? How can we minimize maintenance costs added by new PRs/bugs reported from this spike?

*ACCEPTANCE CRITERIA:*