Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The Submariner Automated Upgrade command is updating only one component #2859

Closed
BhavaniYalamanchili opened this issue Jan 1, 2024 · 7 comments
Closed

The Submariner Automated Upgrade command is updating only one component #2859

BhavaniYalamanchili opened this issue Jan 1, 2024 · 7 comments
Labels
stale support

Comments

@BhavaniYalamanchili
Copy link

ISSUE:
We have used the Automated Upgrade command to upgrade the submariner from v0.15.2 to v0.16.2, in the documentation, it is mentioned that all the components will be upgraded, here we see that only one component of the submariner is upgraded.

Document Referenced:
https://submariner.io/operations/upgrading/#automated-upgrade

Outputs:

From Site2, did subctl upgrade on site2

sh-4.4$ /tmp/new_subctl/bin/subctl upgrade --to-version v0.16.0 --kubeconfig /tmp/local-kubeconfig
Cluster "local-config"
 ✓ Checking if the Broker is installed
 ✓ Checking if the Operator is installed
 ✓ Upgrading the Operator to v0.16.0 
 ✓ Created operator namespace: submariner-operator
 ✓ Upgrading the Connectivity component to v0.16.0 
 ✓ Upgrading Service Discovery to v0.16.0
 
sh-4.4$ /root/.local/bin/subctl version
subctl version: v0.15.2
sh-4.4$ /root/.local/bin/subctl show all --kubeconfig /tmp/local-kubeconfig 
Cluster "local-config"
 ✓ Detecting broker(s)
 ✓ No brokers found

 ✓ Showing Connections
GATEWAY                          CLUSTER   REMOTE IP       NAT   CABLE DRIVER   SUBNETS                        STATUS      RTT avg.     
control-1-ru2.rackae1.mydomain   site1     172.20.200.25   no    libreswan      172.30.0.0/16, 10.128.0.0/14   connected   324.836µs    

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP     PUBLIC IP     CABLE DRIVER   TYPE     
site2     172.20.102.26   129.41.87.3   libreswan      local    
site2     172.20.102.27   129.41.87.3   libreswan      local    
site1     172.20.200.25   129.41.87.2   libreswan      remote   

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY                               
control-1-ru3.rackae2.mydomain   passive     There are no connections              
control-1-ru4.rackae2.mydomain   active      All connections (1) are established   

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  OVNKubernetes
        Service CIDRs:   [172.31.0.0/16]
        Cluster CIDRs:   [10.132.0.0/14]

 ✓ Showing versions
COMPONENT                         REPOSITORY           VERSION   
submariner-gateway                quay.io/submariner   0.15.2    
submariner-routeagent             quay.io/submariner   0.15.2    
submariner-metrics-proxy          quay.io/submariner   0.15.2    
submariner-operator               quay.io/submariner   v0.16.0   
submariner-lighthouse-agent       quay.io/submariner   0.15.2    
submariner-lighthouse-coredns     quay.io/submariner   0.15.2    
submariner-networkplugin-syncer   quay.io/submariner   0.15.2    


sh-4.4$ /root/.local/bin/subctl show all --kubeconfig /connection/kube-config/71a5f5abfc/kubeconfig 
Cluster "default-cluster"
 ✓ Detecting broker(s)
NAMESPACE               NAME                COMPONENTS                        GLOBALNET   GLOBALNET CIDR   DEFAULT GLOBALNET SIZE   DEFAULT DOMAINS   
submariner-k8s-broker   submariner-broker   service-discovery, connectivity   no          242.0.0.0/8      65536                                      

 ✓ Showing Connections
GATEWAY                          CLUSTER   REMOTE IP       NAT   CABLE DRIVER   SUBNETS                        STATUS      RTT avg.     
control-1-ru4.rackae2.mydomain   site2     172.20.102.27   no    libreswan      172.31.0.0/16, 10.132.0.0/14   connected   369.425µs    

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP     PUBLIC IP     CABLE DRIVER   TYPE     
site1     172.20.200.25   129.41.87.2   libreswan      local    
site2     172.20.102.27   129.41.87.3   libreswan      remote   
site1     172.20.200.26   129.41.87.2   libreswan      local    
site1     172.20.200.27   129.41.87.2   libreswan      local    

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY                               
control-1-ru2.rackae1.mydomain   active      All connections (1) are established   
control-1-ru3.rackae1.mydomain   passive     There are no connections              
control-1-ru4.rackae1.mydomain   passive     There are no connections              

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  OVNKubernetes
        Service CIDRs:   [172.30.0.0/16]
        Cluster CIDRs:   [10.128.0.0/14]

 ✓ Showing versions
COMPONENT                         REPOSITORY           VERSION   
submariner-gateway                quay.io/submariner   0.15.2    
submariner-routeagent             quay.io/submariner   0.15.2    
submariner-metrics-proxy          quay.io/submariner   0.15.2    
submariner-operator               quay.io/submariner   0.15.2    
submariner-lighthouse-agent       quay.io/submariner   0.15.2    
submariner-lighthouse-coredns     quay.io/submariner   0.15.2    
submariner-networkplugin-syncer   quay.io/submariner   0.15.2

From Site2, did subctl upgrade on site1

sh-4.4$ /tmp/new_subctl/bin/subctl upgrade --to-version v0.16.0 --kubeconfig /connection/kube-config/71a5f5abfc/kubeconfig 
Cluster "default-cluster"
 ✓ Checking if the Broker is installed
 ✓ Upgrading the Broker to v0.16.0
 ✓ Setting up broker RBAC 
 ✓ Deploying the Submariner operator 
 ✓ Created operator namespace: submariner-operator
 ✓ Deploying the broker 
 ✓ Upgrading the Connectivity component to v0.16.0 
 ✓ Upgrading Service Discovery to v0.16.0

sh-4.4$ /root/.local/bin/subctl show all --kubeconfig /tmp/local-kubeconfig 
Cluster "local-config"
 ✓ Detecting broker(s)
 ✓ No brokers found

 ✓ Showing Connections
GATEWAY                          CLUSTER   REMOTE IP       NAT   CABLE DRIVER   SUBNETS                        STATUS      RTT avg.    
control-1-ru2.rackae1.mydomain   site1     172.20.200.25   no    libreswan      172.30.0.0/16, 10.128.0.0/14   connected   274.04µs    

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP     PUBLIC IP     CABLE DRIVER   TYPE     
site2     172.20.102.26   129.41.87.3   libreswan      local    
site2     172.20.102.27   129.41.87.3   libreswan      local    
site1     172.20.200.25   129.41.87.2   libreswan      remote   

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY                               
control-1-ru3.rackae2.mydomain   passive     There are no connections              
control-1-ru4.rackae2.mydomain   active      All connections (1) are established   

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  OVNKubernetes
        Service CIDRs:   [172.31.0.0/16]
        Cluster CIDRs:   [10.132.0.0/14]

 ✓ Showing versions 
COMPONENT                         REPOSITORY           VERSION   
submariner-gateway                quay.io/submariner   0.15.2    
submariner-routeagent             quay.io/submariner   0.15.2    
submariner-metrics-proxy          quay.io/submariner   0.15.2    
submariner-operator               quay.io/submariner   v0.16.0   
submariner-lighthouse-agent       quay.io/submariner   0.15.2    
submariner-lighthouse-coredns     quay.io/submariner   0.15.2    
submariner-networkplugin-syncer   quay.io/submariner   0.15.2    


sh-4.4$ /root/.local/bin/subctl show all --kubeconfig /connection/kube-config/71a5f5abfc/kubeconfig 
Cluster "default-cluster"
 ✓ Detecting broker(s)
NAMESPACE               NAME                COMPONENTS                        GLOBALNET   GLOBALNET CIDR   DEFAULT GLOBALNET SIZE   DEFAULT DOMAINS   
submariner-k8s-broker   submariner-broker   service-discovery, connectivity   no          242.0.0.0/8      65536                                      

 ✓ Showing Connections
GATEWAY                          CLUSTER   REMOTE IP       NAT   CABLE DRIVER   SUBNETS                        STATUS      RTT avg.     
control-1-ru4.rackae2.mydomain   site2     172.20.102.27   no    libreswan      172.31.0.0/16, 10.132.0.0/14   connected   368.614µs    

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP     PUBLIC IP     CABLE DRIVER   TYPE     
site1     172.20.200.25   129.41.87.2   libreswan      local    
site2     172.20.102.27   129.41.87.3   libreswan      remote   
site1     172.20.200.26   129.41.87.2   libreswan      local    
site1     172.20.200.27   129.41.87.2   libreswan      local    

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY                               
control-1-ru2.rackae1.mydomain   active      All connections (1) are established   
control-1-ru3.rackae1.mydomain   passive     There are no connections              
control-1-ru4.rackae1.mydomain   passive     There are no connections              

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  OVNKubernetes
        Service CIDRs:   [172.30.0.0/16]
        Cluster CIDRs:   [10.128.0.0/14]

 ✓ Showing versions 
COMPONENT                         REPOSITORY           VERSION   
submariner-gateway                quay.io/submariner   0.15.2    
submariner-routeagent             quay.io/submariner   0.15.2    
submariner-metrics-proxy          quay.io/submariner   0.15.2    
submariner-operator               quay.io/submariner   v0.16.0   
submariner-lighthouse-agent       quay.io/submariner   0.15.2    
submariner-lighthouse-coredns     quay.io/submariner   0.15.2    
submariner-networkplugin-syncer   quay.io/submariner   0.15.2

We couldn't do subctl gather due to the issue that we have here
#2857

Questions:
Till now the process that we followed for upgradation is to uninstall the older version and install the newer version. This process takes about 10 mins, during this, there will be downtime.

  1. As this is a frequent process, such a long downtime would be detrimental to the product, is there any alternate way to address this issue?

We discovered that there is an automated upgrade command.
subctl upgrade --to-version v0.16.0 --kubeconfig <kubeconfig_path>

  1. Is this the right way to follow for the upgrade of the submariner?
  2. Is it expected that only one component of the submariner will be upgraded, while the command is run?
  3. If so is there any workaround to be followed to make other components of the submariner to be upgraded?
@skitt
Copy link
Member

skitt commented Jan 3, 2024

The recommended way to deploy Submariner on OpenShift is to use Advanced Cluster Management; this takes care of upgrades for you.

In this particular instance, we need some more information; to start with, can you show the output of

/tmp/new_subctl/bin/subctl show all --kubeconfig /tmp/local-kubeconfig

(The new version of subctl shows more information than version 0.15.2.)

@BhavaniYalamanchili
Copy link
Author

Hi @skitt
We have uninstalled the submariner and installed the new version of it manually due to urgent requirements.
However, we have tried the upgrade command on other clusters, it's the same result.

Outputs for upgrade commands:

sh-4.4$ /tmp/new_subctl/bin/subctl upgrade --to-version v0.16.0 --kubeconfig /tmp/local-kubeconfig
Cluster "local-config"
 ✓ Checking if the Broker is installed
 ✓ Checking if the Operator is installed
 ✓ Upgrading the Operator to v0.16.0 
 ✓ Created operator namespace: submariner-operator
 ✓ Upgrading the Connectivity component to v0.16.0 
 ✓ Upgrading Service Discovery to v0.16.0

sh-4.4$ /tmp/new_subctl/bin/subctl upgrade --to-version v0.16.0 --kubeconfig /connection/kube-config/c0672c4c63/kubeconfig 
Cluster "default-cluster"
 ✓ Checking if the Broker is installed
 ✓ Upgrading the Broker to v0.16.0
 ✓ Setting up broker RBAC
 ✓ Deploying the Submariner operator 
 ✓ Created operator namespace: submariner-operator
 ✓ Deploying the broker 
 ✓ Upgrading the Connectivity component to v0.16.0 
 ✓ Upgrading Service Discovery to v0.16.0

Outputs for commands show all, diagnose all, and subctl gather
For SITE 2

sh-4.4$ /tmp/new_subctl/bin/subctl show all --kubeconfig /tmp/local-kubeconfig
Cluster "local-config"
 ✓ Detecting broker(s)
 ✓ No brokers found

 ✓ Showing Connections
GATEWAY                          CLUSTER   REMOTE IP      NAT   CABLE DRIVER   SUBNETS                        STATUS      RTT avg.     
control-1-ru4.isf-racki.rtp.ra   site1     9.42.107.216   no    libreswan      172.30.0.0/16, 10.128.0.0/14   connected   267.558µs    

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP    PUBLIC IP     CABLE DRIVER   TYPE     
site2     9.42.107.234   129.41.87.2   libreswan      local    
site1     9.42.107.216   129.41.87.0   libreswan      remote   
site2     9.42.107.235   129.41.87.3   libreswan      local    
site2     9.42.107.236   129.41.87.4   libreswan      local    

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY                               
control-1-ru2.isf-rackk.rtp.ra   active      All connections (1) are established   
control-1-ru3.isf-rackk.rtp.ra   passive     There are no connections              
control-1-ru4.isf-rackk.rtp.ra   passive     There are no connections              

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  OVNKubernetes
        Service CIDRs:   [172.31.0.0/16]
        Cluster CIDRs:   [10.132.0.0/14]

 ✓ Showing versions 
COMPONENT                       REPOSITORY           CONFIGURED   RUNNING       
submariner-gateway              quay.io/submariner   0.15.2       Unavailable   
submariner-routeagent           quay.io/submariner   0.15.2       Unavailable   
submariner-metrics-proxy        quay.io/submariner   0.15.2       Unavailable   
submariner-operator             quay.io/submariner   v0.16.0      Unavailable   
submariner-lighthouse-agent     quay.io/submariner   0.15.2       Unavailable   
submariner-lighthouse-coredns   quay.io/submariner   0.15.2       Unavailable   

sh-4.4$ /tmp/new_subctl/bin/subctl diagnose all --kubeconfig /tmp/local-kubeconfig
Cluster "local-config"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.25.14+bcb9a60" is supported

 ✓ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
 ✓ Checking DaemonSet "submariner-gateway"
 ✓ Checking DaemonSet "submariner-routeagent"
 ✓ Checking DaemonSet "submariner-metrics-proxy"
 ✓ Checking Deployment "submariner-lighthouse-agent"
 ✓ Checking Deployment "submariner-lighthouse-coredns"
 ✗ Checking the status of all Submariner pods
 ✗ Pod "submariner-operator-5b7c6958b6-l2kfd" is not running. (current state is Pending)

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OVNKubernetes") is supported
 ✓ Checking OVN version 
 ✓ The ovn-nb database version 7.0.0 is supported
 ✓ Checking gateway connections
 ✓ Checking Submariner support for the kube-proxy mode
 ✓ Cluster is running with "OVNKubernetes" CNI which internally implements kube-proxy functionality
 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic

 ✓ Checking that services have been exported properly

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

subctl version: v0.16.0

sh-4.4$ /tmp/new_subctl/bin/subctl subctl gather --kubeconfig /tmp/local-kubeconfig
Error: unknown command "subctl" for "subctl"
Run 'subctl --help' for usage.
unknown command "subctl" for "subctl"
sh-4.4$ cd /tmp
sh-4.4$ /tmp/new_subctl/bin/subctl gather --kubeconfig /tmp/local-kubeconfig
Cluster "local-config"
Gathering information from cluster "local-config"
 ✓ Gathering connectivity logs 
 ✓ Found 3 pods matching label selector "app=submariner-gateway"
 ✓ Found 6 pods matching label selector "app=submariner-routeagent"
 ✓ Found 3 pods matching label selector "app=submariner-metrics-proxy"
 ✓ Found 0 pods matching label selector "app=submariner-globalnet"
 ✓ Found 0 pods matching label selector "app=submariner-addon"
 ✗ Gathering connectivity resources 
 ✓ Gathering CNI data from 6 pods matching label selector "app=submariner-routeagent"
 ✓ Gathering CNI data from 3 pods matching label selector "app=submariner-gateway"
 ✓ Gathering cable driver data from 3 pods matching label selector "app=submariner-gateway"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-master-jkbwb"
 ✗ Failed to gather gatewayroutes: error listing "gatewayroutes": the server could not find the requested resource
 ✗ Failed to gather nongatewayroutes: error listing "nongatewayroutes": the server could not find the requested resource
 ✓ Found 2 endpoints in namespace "submariner-operator"
 ✓ Found 2 clusters in namespace "submariner-operator"
 ✓ Found 3 gateways in namespace "submariner-operator"
 ✓ Found 0 clusterglobalegressips in namespace ""
 ✓ Found 0 globalegressips in namespace ""
 ✓ Found 0 globalingressips in namespace ""
 ✓ Gathering service-discovery logs 
 ✓ Found 3 pods matching label selector "component=submariner-lighthouse"
 ✓ Found 6 pods matching label selector "dns.operator.openshift.io/daemonset-dns=default"
 ✓ Gathering service-discovery resources 
 ✓ Found 0 serviceexports in namespace ""
 ✓ Found 0 serviceimports in namespace ""
 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""
 ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"
 ✓ Found 1 configmaps by field selector "metadata.name=dns-default" in namespace "openshift-dns"
 ✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""
 ✓ Gathering broker logs
 ✓ Gathering broker resources
 ✓ Found 2 endpoints in namespace "submariner-k8s-broker"
 ✓ Found 2 clusters in namespace "submariner-k8s-broker"
 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace "submariner-k8s-broker"
 ✓ Found 0 serviceimports in namespace "submariner-k8s-broker"
 ✗ Gathering operator logs 
 ✓ Found 2 pods matching label selector "name=submariner-operator"
 ✗ Error outputting current log for pod "submariner-operator-5b7c6958b6-l2kfd": error opening log stream: container "submariner-operator" in pod "submariner-operator-5b7c6958b6-l2kfd" is waiting to start: trying and failing to pull image
 ✓ Gathering operator resources
 ✓ Found 1 submariners in namespace "submariner-operator"
 ✓ Found 1 servicediscoveries in namespace "submariner-operator"
 ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"
 ✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"
Files are stored under directory "submariner-20240105173047/local-config"

For SITE 1

sh-4.4$ /tmp/new_subctl/bin/subctl show all --kubeconfig /connection/kube-config/c0672c4c63/kubeconfig 
Cluster "default-cluster"
 ✓ Detecting broker(s)
NAMESPACE               NAME                COMPONENTS                        GLOBALNET   GLOBALNET CIDR   DEFAULT GLOBALNET SIZE   DEFAULT DOMAINS   
submariner-k8s-broker   submariner-broker   service-discovery, connectivity   no          242.0.0.0/8      65536                                      

 ✓ Showing Connections
GATEWAY                          CLUSTER   REMOTE IP      NAT   CABLE DRIVER   SUBNETS                        STATUS      RTT avg.     
control-1-ru2.isf-rackk.rtp.ra   site2     9.42.107.234   no    libreswan      172.31.0.0/16, 10.132.0.0/14   connected   324.011µs    

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP    PUBLIC IP     CABLE DRIVER   TYPE     
site1     9.42.107.214   129.41.87.6   libreswan      local    
site1     9.42.107.215   129.41.87.7   libreswan      local    
site1     9.42.107.216   129.41.87.0   libreswan      local    
site2     9.42.107.234   129.41.87.2   libreswan      remote   

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY                               
control-1-ru2.isf-racki.rtp.ra   passive     There are no connections              
control-1-ru3.isf-racki.rtp.ra   passive     There are no connections              
control-1-ru4.isf-racki.rtp.ra   active      All connections (1) are established   

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  OVNKubernetes
        Service CIDRs:   [172.30.0.0/16]
        Cluster CIDRs:   [10.128.0.0/14]

 ✓ Showing versions 
COMPONENT                       REPOSITORY           CONFIGURED   RUNNING       
submariner-gateway              quay.io/submariner   0.15.2       Unavailable   
submariner-routeagent           quay.io/submariner   0.15.2       Unavailable   
submariner-metrics-proxy        quay.io/submariner   0.15.2       Unavailable   
submariner-operator             quay.io/submariner   v0.16.0      Unavailable   
submariner-lighthouse-agent     quay.io/submariner   0.15.2       Unavailable   
submariner-lighthouse-coredns   quay.io/submariner   0.15.2       Unavailable   

sh-4.4$ /tmp/new_subctl/bin/subctl diagnose all --kubeconfig /connection/kube-config/c0672c4c63/kubeconfig 
Cluster "default-cluster"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.25.14+bcb9a60" is supported

 ✓ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
 ✓ Checking DaemonSet "submariner-gateway"
 ✓ Checking DaemonSet "submariner-routeagent"
 ✓ Checking DaemonSet "submariner-metrics-proxy"
 ✓ Checking Deployment "submariner-lighthouse-agent"
 ✓ Checking Deployment "submariner-lighthouse-coredns"
 ✗ Checking the status of all Submariner pods
 ✗ Pod "submariner-operator-7bfcdc5d48-6jv62" is not running. (current state is Pending)

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OVNKubernetes") is supported
 ✓ Checking OVN version 
 ✓ The ovn-nb database version 7.0.0 is supported
 ✓ Checking gateway connections
 ✓ Checking Submariner support for the kube-proxy mode
 ✓ Cluster is running with "OVNKubernetes" CNI which internally implements kube-proxy functionality
 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic

 ✓ Checking that services have been exported properly

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

subctl version: v0.16.0

sh-4.4$ /tmp/new_subctl/bin/subctl gather --kubeconfig /connection/kube-config/c0672c4c63/kubeconfig 
Cluster "default-cluster"
Gathering information from cluster "default-cluster"
 ✓ Gathering broker logs
 ✓ Gathering broker resources
 ✓ Found 2 endpoints in namespace "submariner-k8s-broker"
 ✓ Found 2 clusters in namespace "submariner-k8s-broker"
 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace "submariner-k8s-broker"
 ✓ Found 0 serviceimports in namespace "submariner-k8s-broker"
 ✗ Gathering operator logs 
 ✓ Found 2 pods matching label selector "name=submariner-operator"
 ✗ Error outputting current log for pod "submariner-operator-7bfcdc5d48-6jv62": error opening log stream: container "submariner-operator" in pod "submariner-operator-7bfcdc5d48-6jv62" is waiting to start: trying and failing to pull image
 ✓ Gathering operator resources
 ✓ Found 1 submariners in namespace "submariner-operator"
 ✓ Found 1 servicediscoveries in namespace "submariner-operator"
 ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"
 ✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"
 ✓ Gathering connectivity logs 
 ✓ Found 3 pods matching label selector "app=submariner-gateway"
 ✓ Found 6 pods matching label selector "app=submariner-routeagent"
 ✓ Found 3 pods matching label selector "app=submariner-metrics-proxy"
 ✓ Found 0 pods matching label selector "app=submariner-globalnet"
 ✓ Found 0 pods matching label selector "app=submariner-addon"
 ✓ Gathering connectivity resources 
 ✓ Gathering CNI data from 6 pods matching label selector "app=submariner-routeagent"
 ✓ Gathering CNI data from 3 pods matching label selector "app=submariner-gateway"
 ✓ Gathering cable driver data from 3 pods matching label selector "app=submariner-gateway"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-master-lttjt"
 ✓ Found 0 gatewayroutes in namespace ""
 ✓ Found 0 nongatewayroutes in namespace ""
 ✓ Found 2 endpoints in namespace "submariner-operator"
 ✓ Found 2 clusters in namespace "submariner-operator"
 ✓ Found 3 gateways in namespace "submariner-operator"
 ✓ Found 0 clusterglobalegressips in namespace ""
 ✓ Found 0 globalegressips in namespace ""
 ✓ Found 0 globalingressips in namespace ""
 ✓ Gathering service-discovery logs 
 ✓ Found 3 pods matching label selector "component=submariner-lighthouse"
 ✓ Found 6 pods matching label selector "dns.operator.openshift.io/daemonset-dns=default"
 ✓ Gathering service-discovery resources 
 ✓ Found 0 serviceexports in namespace ""
 ✓ Found 0 serviceimports in namespace ""
 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""
 ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"
 ✓ Found 1 configmaps by field selector "metadata.name=dns-default" in namespace "openshift-dns"
 ✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""
Files are stored under directory "submariner-20240105173120/default-cluster"

Subctl Gather Files :
submariner-site1-gather.zip
submariner-site2-gather.zip

@dfarrell07 dfarrell07 mentioned this issue Jan 9, 2024
Closed
@dfarrell07
Copy link
Member

We know of an issue (submariner-io/subctl#971) with 0.16.0 upgrades, but we think they are fixed in later 0.16.x releases. Any reason not to upgrade to 0.16.2?

@BhavaniYalamanchili
Copy link
Author

Hi @dfarrell07
We can upgrade to 0.16.2 if it is compatible with OCP 4.12.x, 4.13.x and 4.14.x

Yesterday we tried to use the automated upgrade command using the subctl version v0.16.2
But, this is also doing the same(upgrading only one component of the submariner)

Before using upgrade command

sh-4.4$ /tmp/new_subctl/bin/subctl show all --kubeconfig /tmp/local-kubeconfig 
Cluster "local-config"
 ✓ Detecting broker(s)
 ✓ No brokers found

 ✓ Showing Connections
GATEWAY                          CLUSTER   REMOTE IP      NAT   CABLE DRIVER   SUBNETS                        STATUS      RTT avg.     
control-1-ru4.isf-racki.rtp.ra   site1     9.42.107.216   no    libreswan      172.30.0.0/16, 10.128.0.0/14   connected   329.319µs    

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP    PUBLIC IP     CABLE DRIVER   TYPE     
site2     9.42.107.234   129.41.87.2   libreswan      local    
site2     9.42.107.235   129.41.87.3   libreswan      local    
site1     9.42.107.216   129.41.87.0   libreswan      remote   
site2     9.42.107.236   129.41.87.4   libreswan      local    

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY                               
control-1-ru2.isf-rackk.rtp.ra   passive     There are no connections              
control-1-ru3.isf-rackk.rtp.ra   active      All connections (1) are established   
control-1-ru4.isf-rackk.rtp.ra   passive     There are no connections              

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  OVNKubernetes
        Service CIDRs:   [172.31.0.0/16]
        Cluster CIDRs:   [10.132.0.0/14]

 ✓ Showing versions 
COMPONENT                       REPOSITORY           CONFIGURED   RUNNING       ARCH          
submariner-gateway              quay.io/submariner   0.15.2       Unavailable   Unavailable   
submariner-routeagent           quay.io/submariner   0.15.2       Unavailable   Unavailable   
submariner-metrics-proxy        quay.io/submariner   0.15.2       Unavailable   Unavailable   
submariner-operator             quay.io/submariner   0.15.2       Unavailable   Unavailable   
submariner-lighthouse-agent     quay.io/submariner   0.15.2       Unavailable   Unavailable   
submariner-lighthouse-coredns   quay.io/submariner   0.15.2       Unavailable   Unavailable

Upgrade command for site 2

sh-4.4$ /tmp/new_subctl/bin/subctl upgrade --to-version v0.16.2 --kubeconfig /tmp/local-kubeconfig
Cluster "local-config"
 ✓ Checking if the Broker is installed
 ✓ Checking if the Operator is installed
 ✓ Upgrading the Operator to v0.16.2 
 ✓ Created operator namespace: submariner-operator
 ✓ Upgrading the Connectivity component to v0.16.2 
 ✓ Upgrading Service Discovery to v0.16.2

Upgrade command for site 1

sh-4.4$ /tmp/new_subctl/bin/subctl upgrade --to-version v0.16.2 --kubeconfig /connection/kube-config/c0672c4c63/kubeconfig 
Cluster "default-cluster"
 ✓ Checking if the Broker is installed
 ✓ Upgrading the Broker to v0.16.2
 ✓ Setting up broker RBAC
 ✓ Deploying the Submariner operator 
 ✓ Created operator namespace: submariner-operator
 ✓ Deploying the broker 
 ✓ Upgrading the Connectivity component to v0.16.2 
 ✓ Upgrading Service Discovery to v0.16.2

Show all command for Site 2

sh-4.4$ /tmp/new_subctl/bin/subctl show all --kubeconfig /tmp/local-kubeconfig 
Cluster "local-config"
 ✓ Detecting broker(s)
 ✓ No brokers found

 ✓ Showing Connections
GATEWAY                          CLUSTER   REMOTE IP      NAT   CABLE DRIVER   SUBNETS                        STATUS      RTT avg.   
control-1-ru4.isf-racki.rtp.ra   site1     9.42.107.216   no    libreswan      172.30.0.0/16, 10.128.0.0/14   connected   0s         

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP    PUBLIC IP     CABLE DRIVER   TYPE     
site2     9.42.107.234   129.41.87.2   libreswan      local    
site2     9.42.107.235   129.41.87.3   libreswan      local    
site1     9.42.107.216   129.41.87.0   libreswan      remote   
site2     9.42.107.236   129.41.87.4   libreswan      local    

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY                               
control-1-ru2.isf-rackk.rtp.ra   passive     There are no connections              
control-1-ru3.isf-rackk.rtp.ra   active      All connections (1) are established   
control-1-ru4.isf-rackk.rtp.ra   passive     There are no connections              

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  OVNKubernetes
        Service CIDRs:   [172.31.0.0/16]
        Cluster CIDRs:   [10.132.0.0/14]

 ✓ Showing versions 
COMPONENT                       REPOSITORY           CONFIGURED   RUNNING       ARCH          
submariner-gateway              quay.io/submariner   0.15.2       Unavailable   Unavailable   
submariner-routeagent           quay.io/submariner   0.15.2       Unavailable   Unavailable   
submariner-metrics-proxy        quay.io/submariner   0.15.2       Unavailable   Unavailable   
submariner-operator             quay.io/submariner   v0.16.2      Unavailable   Unavailable   
submariner-lighthouse-agent     quay.io/submariner   0.15.2       Unavailable   Unavailable   
submariner-lighthouse-coredns   quay.io/submariner   0.15.2       Unavailable   Unavailable

Show all command for site 1

sh-4.4$ /tmp/new_subctl/bin/subctl show all --kubeconfig /connection/kube-config/c0672c4c63/kubeconfig 
Cluster "default-cluster"
 ✓ Detecting broker(s)
NAMESPACE               NAME                COMPONENTS                        GLOBALNET   GLOBALNET CIDR   DEFAULT GLOBALNET SIZE   DEFAULT DOMAINS   
submariner-k8s-broker   submariner-broker   service-discovery, connectivity   no          242.0.0.0/8      65536                                      

 ✓ Showing Connections
GATEWAY                          CLUSTER   REMOTE IP      NAT   CABLE DRIVER   SUBNETS                        STATUS      RTT avg.     
control-1-ru3.isf-rackk.rtp.ra   site2     9.42.107.235   no    libreswan      172.31.0.0/16, 10.132.0.0/14   connected   372.694µs    

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP    PUBLIC IP     CABLE DRIVER   TYPE     
site1     9.42.107.214   129.41.87.6   libreswan      local    
site1     9.42.107.215   129.41.87.7   libreswan      local    
site1     9.42.107.216   129.41.87.0   libreswan      local    
site2     9.42.107.235   129.41.87.3   libreswan      remote   

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY                               
control-1-ru2.isf-racki.rtp.ra   passive     There are no connections              
control-1-ru3.isf-racki.rtp.ra   passive     There are no connections              
control-1-ru4.isf-racki.rtp.ra   active      All connections (1) are established   

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  OVNKubernetes
        Service CIDRs:   [172.30.0.0/16]
        Cluster CIDRs:   [10.128.0.0/14]

 ✓ Showing versions 
COMPONENT                       REPOSITORY           CONFIGURED   RUNNING       ARCH          
submariner-gateway              quay.io/submariner   0.15.2       Unavailable   Unavailable   
submariner-routeagent           quay.io/submariner   0.15.2       Unavailable   Unavailable   
submariner-metrics-proxy        quay.io/submariner   0.15.2       Unavailable   Unavailable   
submariner-operator             quay.io/submariner   v0.16.2      Unavailable   Unavailable   
submariner-lighthouse-agent     quay.io/submariner   0.15.2       Unavailable   Unavailable   
submariner-lighthouse-coredns   quay.io/submariner   0.15.2       Unavailable   Unavailable

Diagnose all for site 2

sh-4.4$ /tmp/new_subctl/bin/subctl diagnose all --kubeconfig /tmp/local-kubeconfig 
Cluster "local-config"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.26.11+7dfc52e" is supported

 ✓ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
 ✓ Checking DaemonSet "submariner-gateway"
 ✓ Checking DaemonSet "submariner-routeagent"
 ✓ Checking DaemonSet "submariner-metrics-proxy"
 ✓ Checking Deployment "submariner-lighthouse-agent"
 ✓ Checking Deployment "submariner-lighthouse-coredns"
 ✗ Checking the status of all Submariner pods
 ✗ Pod "submariner-operator-578485964d-nkvlj" is not running. (current state is Pending)

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OVNKubernetes") is supported
 ✓ Checking OVN version
 ✓ The ovn-nb database version 7.0.4 is supported
 ✓ Checking gateway connections
 ✓ Checking Submariner support for the kube-proxy mode
 ✓ Cluster is running with "OVNKubernetes" CNI which internally implements kube-proxy functionality
 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic

 ✓ Checking that services have been exported properly

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

subctl version: v0.16.2

sh-4.4$ /tmp/new_subctl/bin/subctl diagnose all --kubeconfig /connection/kube-config/c0672c4c63/kubeconfig 
Cluster "default-cluster"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.26.11+7dfc52e" is supported

 ✓ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
 ✓ Checking DaemonSet "submariner-gateway"
 ✓ Checking DaemonSet "submariner-routeagent"
 ✓ Checking DaemonSet "submariner-metrics-proxy"
 ✓ Checking Deployment "submariner-lighthouse-agent"
 ✓ Checking Deployment "submariner-lighthouse-coredns"
 ✗ Checking the status of all Submariner pods
 ✗ Pod "submariner-operator-578555d5bb-mb7jn" is not running. (current state is Pending)

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OVNKubernetes") is supported
 ✓ Checking OVN version
 ✓ The ovn-nb database version 7.0.4 is supported
 ✓ Checking gateway connections
 ✓ Checking Submariner support for the kube-proxy mode
 ✓ Cluster is running with "OVNKubernetes" CNI which internally implements kube-proxy functionality
 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic

 ✓ Checking that services have been exported properly

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

subctl version: v0.16.2

Outputs for subctl gather
For Site 2

sh-4.4$ /tmp/new_subctl/bin/subctl gather --kubeconfig /tmp/local-kubeconfig
Cluster "local-config"
Gathering information from cluster "local-config"
 ✓ Gathering broker logs
 ✓ Gathering broker resources
 ✓ Found 2 endpoints in namespace "submariner-k8s-broker"
 ✓ Found 2 clusters in namespace "submariner-k8s-broker"
 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace "submariner-k8s-broker"
 ✓ Found 0 serviceimports in namespace "submariner-k8s-broker"
 ✗ Gathering operator logs 
 ✓ Found 2 pods matching label selector "name=submariner-operator"
 ✗ Error outputting current log for pod "submariner-operator-578485964d-nkvlj": error opening log stream: container "submariner-operator" in pod "submariner-operator-578485964d-nkvlj" is waiting to start: trying and failing to pull image
 ✓ Gathering operator resources
 ✓ Found 1 submariners in namespace "submariner-operator"
 ✓ Found 1 servicediscoveries in namespace "submariner-operator"
 ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"
 ✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"
 ✓ Gathering connectivity logs 
 ✓ Found 3 pods matching label selector "app=submariner-gateway"
 ✓ Found 6 pods matching label selector "app=submariner-routeagent"
 ✓ Found 3 pods matching label selector "app=submariner-metrics-proxy"
 ✓ Found 0 pods matching label selector "app=submariner-globalnet"
 ✓ Found 0 pods matching label selector "app=submariner-addon"
 ✓ Gathering connectivity resources 
 ✓ Gathering CNI data from 6 pods matching label selector "app=submariner-routeagent"
 ✓ Gathering CNI data from 3 pods matching label selector "app=submariner-gateway"
 ✓ Gathering cable driver data from 3 pods matching label selector "app=submariner-gateway"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-master-7xrzp"
 ✓ Found 0 gatewayroutes in namespace ""
 ✓ Found 0 nongatewayroutes in namespace ""
 ✓ Found 2 endpoints in namespace "submariner-operator"
 ✓ Found 2 clusters in namespace "submariner-operator"
 ✓ Found 3 gateways in namespace "submariner-operator"
 ✓ Found 0 clusterglobalegressips in namespace ""
 ✓ Found 0 globalegressips in namespace ""
 ✓ Found 0 globalingressips in namespace ""
 ✓ Gathering service-discovery logs 
 ✓ Found 3 pods matching label selector "component=submariner-lighthouse"
 ✓ Found 6 pods matching label selector "dns.operator.openshift.io/daemonset-dns=default"
 ✓ Gathering service-discovery resources 
 ✓ Found 0 serviceexports in namespace ""
 ✓ Found 0 serviceimports in namespace ""
 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""
 ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"
 ✓ Found 1 configmaps by field selector "metadata.name=dns-default" in namespace "openshift-dns"
 ✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""
Files are stored under directory "submariner-20240110101256/local-config"

For site 1

sh-4.4$ /tmp/new_subctl/bin/subctl gather --kubeconfig /connection/kube-config/c0672c4c63/kubeconfig 
Cluster "default-cluster"
Gathering information from cluster "default-cluster"
 ✓ Gathering connectivity logs 
 ✓ Found 3 pods matching label selector "app=submariner-gateway"
 ✓ Found 6 pods matching label selector "app=submariner-routeagent"
 ✓ Found 3 pods matching label selector "app=submariner-metrics-proxy"
 ✓ Found 0 pods matching label selector "app=submariner-globalnet"
 ✓ Found 0 pods matching label selector "app=submariner-addon"
 ✓ Gathering connectivity resources 
 ✓ Gathering CNI data from 6 pods matching label selector "app=submariner-routeagent"
 ✓ Gathering CNI data from 3 pods matching label selector "app=submariner-gateway"
 ✓ Gathering cable driver data from 3 pods matching label selector "app=submariner-gateway"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-master-r8sbp"
 ✓ Found 1 gatewayroutes in namespace ""
 ✓ Found 0 nongatewayroutes in namespace ""
 ✓ Found 2 endpoints in namespace "submariner-operator"
 ✓ Found 2 clusters in namespace "submariner-operator"
 ✓ Found 3 gateways in namespace "submariner-operator"
 ✓ Found 0 clusterglobalegressips in namespace ""
 ✓ Found 0 globalegressips in namespace ""
 ✓ Found 0 globalingressips in namespace ""
 ✓ Gathering service-discovery logs 
 ✓ Found 3 pods matching label selector "component=submariner-lighthouse"
 ✓ Found 6 pods matching label selector "dns.operator.openshift.io/daemonset-dns=default"
 ✓ Gathering service-discovery resources 
 ✓ Found 0 serviceexports in namespace ""
 ✓ Found 0 serviceimports in namespace ""
 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""
 ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"
 ✓ Found 1 configmaps by field selector "metadata.name=dns-default" in namespace "openshift-dns"
 ✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""
 ✓ Gathering broker logs
 ✓ Gathering broker resources
 ✓ Found 2 endpoints in namespace "submariner-k8s-broker"
 ✓ Found 2 clusters in namespace "submariner-k8s-broker"
 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace "submariner-k8s-broker"
 ✓ Found 0 serviceimports in namespace "submariner-k8s-broker"
 ✗ Gathering operator logs 
 ✓ Found 2 pods matching label selector "name=submariner-operator"
 ✗ Error outputting current log for pod "submariner-operator-578555d5bb-mb7jn": error opening log stream: container "submariner-operator" in pod "submariner-operator-578555d5bb-mb7jn" is waiting to start: trying and failing to pull image
 ✓ Gathering operator resources
 ✓ Found 1 submariners in namespace "submariner-operator"
 ✓ Found 1 servicediscoveries in namespace "submariner-operator"
 ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"
 ✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"
Files are stored under directory "submariner-20240110101322/default-cluster"

Gather logs
site1_gather_logs.zip
site2_gather_logs.zip

@BhavaniYalamanchili
Copy link
Author

@dfarrell07

Alongside can you provide the Submariner versions compatibility with the OCP versions?
This matrix will be really helpful for us to easily decide on the required submariner version with the OCP versions that we upgrade to.

@BhavaniYalamanchili
Copy link
Author

@dfarrell07
Any update on the above queries?

@github-actions GitHub Actions
Copy link
Contributor

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further
activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label May 17, 2024
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale May 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale support
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dfarrell07 @skitt @BhavaniYalamanchili