kubeadm init “--apiserver-advertise-address=publicIP” not working, private IP works 1.13 version #1390

phanikumarp · 2019-02-06T11:40:22Z

BUG REPORT

Versions

kubeadm version : v.1.13
Environment: Ubuntu-16.04.5 LTS (Xenial Xerus)

Kubernetes version "v1.13.3
Cloud provider or hardware configuration: GCP
OS (e.g. from /etc/os-release): ubuntu-16.04
Kernel (e.g. uname -a): Linux ubuntu 4.15.0-1026-gcp Permissions #27~16.04.1-Ubuntu

What happened?

"sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=32.xxx.xx.xxx(PublicIP)"
when u we using Public IP got error , when using Private IP , got success.

What you expected to happen?

expected success when we give Public IP, but got failed.

How to reproduce it (as minimally and precisely as possible)?

after installed "apt-get install -y kubelet kubeadm kubectl" , trying to get single node cluster with kubadm

Anything else we need to know?

After installing Docker version 18.09.1, i tried single node cluster.

The text was updated successfully, but these errors were encountered:

neolit123 · 2019-02-06T20:24:28Z

this hints a bout a problem with your networking setup.
do you have connectivity to the remote?

what is the output of kubeadm if you add --v=2

phanikumarp · 2019-02-07T05:14:46Z

this hints a bout a problem with your networking setup.
do you have connectivity to the remote?

what is the output of kubeadm if you add --v=2

No, not connecting remotely, using one of GCP VM directly. you said issue in network setup.
added my log below. how to reslove this issue?
https://pastebin.com/nNstHxvx

my network route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.140.0.1 0.0.0.0 UG 0 0 0 ens4
10.140.0.1 * 255.255.255.255 UH 0 0 0 ens4
172.17.0.0 * 255.255.0.0 U 0 0 0 docker0

neolit123 · 2019-02-07T14:07:39Z

i cannot see the log for some reason.
but if you can't connect using tools like curl and ping then this is not a kubeadm bug.

phanikumarp · 2019-02-07T14:28:16Z

Iam able to access curl and ping.

…

On Thu 7 Feb, 2019, 7:37 PM Lubomir I. Ivanov ***@***.*** wrote: i cannot see the log for some reason. but if you can't connect using tools like curl and ping then this is not a kubeadm bug. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1390 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AINnY4CcC0fNGf4oX4vf4r4Q-OpuToR4ks5vLDMugaJpZM4alDvn> .

neolit123 · 2019-02-07T16:35:30Z

/sig network

spockmang · 2019-02-12T12:18:58Z

Hi,
I have reproduced your issue and found the root cause. Please find my analysis below.

During kubeadm init, kubelet will try to create kube-proxy, kube-apiserver, kube-controller, and kube-scheduller. It will try to bind all these services with the public IP address(GCP assigned) of the VM.

But the problem in GCP is, the public IP address does not rest on the VM, but it rather a NAT function. The tricky part here is to understand is, if you are sending a packet to the NAT it will forward the packet to the VM and vice-versa. But your process/application cannot bind to that NAT IP address. The IP address with which you intend to create the cluster has to reside on the VM.

That is why it is working with internal IP address but not with public IP address.

You can verify this by checking 'tail -f /var/log/syslog' while creating the cluster.

Please let me know if this addressed your issue.

-M

sensre · 2019-02-18T21:19:14Z

thanks, @spockmang for the insight. do you have the recommendation or step to make it work with public IP add ?

phanikumarp · 2019-02-19T05:02:42Z

what is final solution for getting Public IP fof kubernetes

bcheerla4509 · 2019-02-25T18:31:50Z

Hi,
Even i have the exact problem with trying on Azure VMs (Redhat 7 OS). Anyone has suggestions to resolve the issue?

This is what i get every time i do kubeadm init .. I tried re-installing kubelet and kubeadm.. tried yum update and restarted VM.

[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
timed out waiting for the condition

This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster

Thanks
Sagar

phanisowjanyavutukuri · 2019-03-18T05:55:58Z

Even am facing the same issue
kubeadm init is failing with public ip and saying that kubelet is misconfigured
This error is likely caused by:

The kubelet is not running
The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

fabriziopandini · 2019-03-18T07:17:45Z

I'm for closing the issue here because this is not a kubeadm problem

The IP address with which you intend to create the cluster has to reside on the VM.

Probably the best option to get a suggestion here is to reopen the issue in k/k and tag SIG network and SIG cloud provider
@neolit123 opinions?

neolit123 · 2019-03-18T07:43:03Z

its definitely not a kubeadm problem, because kubeadm does not manage the node networking.

for the GCP case, this seems to be the root of the problem as explained by @spockmang:

But your process/application cannot bind to that NAT IP address. The IP address with which you intend to create the cluster has to reside on the VM

i think we should not track this issue here but rather in the kubernetes/kubernetes tracker.
also tag /sig gcp, /sig aws, /sig network.

@phanikumarp please re-open the ticket in k/k if you'd like and reference this one.
thank you.

bobbydeveaux · 2019-03-26T21:55:09Z

having same issue @phanikumarp - did you find a solution to use the external IP?

phanikumarp · 2019-03-27T04:51:06Z

I have solution ....for fixing this issue

…

On Wed, Mar 27, 2019 at 3:25 AM Bobby DeVeaux ***@***.***> wrote: having same issue @phanikumarp <https://github.com/phanikumarp> - did you find a solution to use the external IP? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1390 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AINnY7-sbUIsrF24brq-MJ2KQG5mcQ4cks5vapdBgaJpZM4alDvn> .

swapblue · 2019-04-12T04:42:16Z

Thank you @spockmang for the explanation. Just for the record, i faced the same issue on Openstack.

x22n · 2019-04-14T18:22:15Z

@phanikumarp What was the solution?

Experiencing this issue on Scaleway too.

x22n · 2019-04-17T13:13:03Z

I found a solution.

Instead of using --apiserver-advertise-address=publicIp use --apiserver-cert-extra-sans=publicIp

Don't forget to replace the private ip for the public ip in your .kube/config if you use kubectl from remote.

ASLanin · 2019-05-28T14:16:48Z

--apiserver-cert-extra-sans=publicIp do not solve the problem.
Yes, it adds public IP to the Certs, but do not affects connection procedure.
The worker nodes will look for apiserver-advertise-address during join. So they will not connect to the ptivateIP, if no route exist.
Api-server itself have two params
--advertise-address ip and --bind-address ip. It looks reasonable. But, how can this addresses configured during kubeadm init?

pankajcheema · 2019-06-06T11:44:16Z

I am trying on different cloud provider but no luck . Any suggestion or solution ?

atshakil · 2019-07-02T01:38:00Z

I found a solution.

Instead of using --apiserver-advertise-address=publicIp use --apiserver-cert-extra-sans=publicIp

Don't forget to replace the private ip for the public ip in your .kube/config if you use kubectl from remote.

@x22n Your solution results in a verification error.

$ kubectl get nodes
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

Update

It turns out that the verification issue is due to leftover credentials (in $HOME/.kube) from the last master, before kubeadm reset was performed.

This can be resolved by,

mv  $HOME/.kube $HOME/.kube.bak
mkdir $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

And, the following seems to be a clean solution.

kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=0.0.0.0 --apiserver-cert-extra-sans=<PRIVATE_IP>[,<PUBLIC_IP>,...]

I just tested the solution above on Compute Engine.

asymness · 2019-07-10T01:23:39Z

I solved this problem by forwarding the private IP of the master node to the public IP of the master node on the worker node. Specifically, this was the command that I ran on worker node before running kubeadm join:
sudo iptables -t nat -A OUTPUT -d <Private IP of master node> -j DNAT --to-destination <Public IP of master node>

Noobgam · 2019-07-20T13:40:13Z

sudo iptables -t nat -A OUTPUT -d <Private IP of master node> -j DNAT --to-destination <Public IP of master node>

Thanks a lot, solved issue for me. But keep in mind that you'll also have to forward worker private ips the same way on the master node to make everything work correctly (if they suffer from the same issue of being covered by cloud provider NAT)

KevinWang15 · 2019-08-05T05:16:15Z

Using iptables seems to be promising, but I still cannot get kube-proxy to work, and I cannot use ClusterIP services that way

kioyong · 2019-08-29T09:05:09Z

I solved this problem by forwarding the private IP of the master node to the public IP of the master node on the worker node. Specifically, this was the command that I ran on worker node before running kubeadm join:
sudo iptables -t nat -A OUTPUT -d <Private IP of master node> -j DNAT --to-destination <Public IP of master node>

thanks for your solution, i solved this issue~

Zhang21 · 2019-09-24T03:14:26Z

@x22n

I found a solution.
Instead of using --apiserver-advertise-address=publicIp use --apiserver-cert-extra-sans=publicIp
Don't forget to replace the private ip for the public ip in your .kube/config if you use kubectl from remote.

This soltuion is just for kubeclt uses ~.kube/config， but not for nodes kube join cross VPC.

@asymness Iptables also a temporary solution.

# on node
sudo iptables -t nat -A OUTPUT -d <Private IP of master> -j DNAT --to-destination <Public IP of master>


# on master
sudo iptables -t nat -A OUTPUT -d <Private IP of WAN node> -j DNAT --to-destination <Public IP of WAN node>

There is no final solvtion?

Thank you!

Zhang21 · 2019-09-24T06:50:10Z

But it also cause a new issue, the flannel on WAN node create faile, can't connect to kubernest svc(10.96.0.1)。

What should I do?

janjangao · 2019-09-27T06:27:40Z

But it also cause a new issue, the flannel on WAN node create faile, can't connect to kubernest svc(10.96.0.1)。

What should I do?

same problem，flannel pod always error，dial tcp 10.96.0.1:443: i/o timeout

Zhang21 · 2019-09-27T07:26:10Z

@hayond
Creating a k8s cluster which nodes over WAN is diffcult, I just give it up. So I just create a k8s cluster which nodes over LAN, but can use kubectl(config) from remote client.

janjangao · 2019-10-09T12:40:31Z

@Zhang21
after a lot of day and night research... i finally found a way to let k8s work over WAN.

short answer

--apiserver-advertise-address=publicIP is necessary, this tell k8s worker communicate to master over publicIP. default is privateIP, will lead to 10.96.0.1:443: i/o timeout.
node annotation flannel.alpha.coreos.com/public-ip-overwrite=publicIP is necessary，this set flannel pod node-ip to publicIP

full answer

first, run ifconfig on master, check whether there have publicIP info on master interfaces. some cloud provider use Elastic IP, they dont have any publicIP interface info. you must add publicIP interface info yourself. follow Cloud_floating_IP_persistent. if you dont add publicIP interface, kubeadm init --apiserver-advertise-address=publicIP will not success.
kubeadm init with --apiserver-advertise-address=publicIP, i use --control-plane-endpoint=publicIP --upload-certs --apiserver-advertise-address=publicIP for my own. i think just apiserver-advertise-address also will be ok.
apply flannel.yaml and join worker node.
on worker node，add kubelet args --node-ip

vim /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS=--cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --node-ip=publicIP
systemctl restart kubelet

set worker node public-ip-overwrite annotate, and restart worker kube-flannel pod

kubectl annotate node NODE_NAME flannel.alpha.coreos.com/public-ip-overwrite=publicIP

# on worker node
systemctl restart docker

heangratha · 2020-04-11T13:39:53Z

Hi,
I have tried this on AWS EC2

on master:
iptables -t nat -A OUTPUT -d <Private IP of node> -j DNAT --to-destination <Public IP of node>
on node:
iptables -t nat -A OUTPUT -d <Private IP of master> -j DNAT --to-destination <Public IP of master>
kube join
Everything working well I can see master and node

`ubuntu@ip-200-200-20-10:~$ kubectl get nodes
NAME               STATUS   ROLES    AGE     VERSION
ip-200-200-20-10   Ready    master   3m29s   v1.18.1
node-ohio          Ready    <none>   66s     v1.18.1
`

But when I install calico and ingress, both STATUS ContainerCreating

`ubuntu@ip-200-200-20-10:~$ kubectl get po --all-namespaces -o wide
NAMESPACE     NAME                                       READY   STATUS           
kube-system   calico-kube-controllers-5b8b769fcd-dlncq   0/1     ContainerCreating
kube-system   calico-node-94xr2                          1/1     Running          
kube-system   calico-node-p9gnr                          0/1     CrashLoopBackOff 
kube-system   coredns-66bff467f8-26fc4                   1/1     Running          
kube-system   coredns-66bff467f8-4l2tb                   1/1     Running          
kube-system   etcd-ip-200-200-20-10                      1/1     Running          
kube-system   kube-apiserver-ip-200-200-20-10            1/1     Running          
kube-system   kube-controller-manager-ip-200-200-20-10   1/1     Running          
kube-system   kube-proxy-85l2s                           1/1     Running          
kube-system   kube-proxy-r9zmv                           1/1     Running          
kube-system   kube-scheduler-ip-200-200-20-10            1/1     Running          
kube-system   traefik-ingress-controller-zlxst           0/1     ContainerCreating
`

And in the long
81" network for pod "traefik-ingress-controller-58nmh": networkPlugin cni failed to set up pod "traefik-ingress-controller-58nmh_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/ Warning FailedCreatePodSandBox 21m kubelet, node-ohio Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "edd1f8697da91c99e9498cee624f6ff0343cd469b66e363224527696b069769d" network for pod "traefik-ingress-controller-58nmh": networkPlugin cni failed to set up pod "traefik-ingress-controller-58nmh_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/ Normal SandboxChanged 6m26s (x508 over 21m) kubelet, node-ohio Pod sandbox changed, it will be killed and re-created. Warning FailedCreatePodSandBox 86s (x747 over 21m) kubelet, node-ohio (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fbbb8840cee381f881ffddfb0c3afeb6f13a606cfe7b75022a214339e926e36a" network for pod "traefik-ingress-controller-58nmh": networkPlugin cni failed to set up pod "traefik-ingress-controller-58nmh_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

s977120 · 2020-04-20T10:47:19Z

its definitely not a kubeadm problem, because kubeadm does not manage the node networking.

for the GCP case, this seems to be the root of the problem as explained by @spockmang:

But your process/application cannot bind to that NAT IP address. The IP address with which you intend to create the cluster has to reside on the VM

i think we should not track this issue here but rather in the kubernetes/kubernetes tracker.
also tag /sig gcp, /sig aws, /sig network.

@phanikumarp please re-open the ticket in k/k if you'd like and reference this one.
thank you.

Please reopen this issue, binding to public ip is not a kubeadm problem, But there is still a problem with binding to 0.0.0.0.
Perhaps by adding new options, distinguish between bind ip and advertise ip

saarram · 2020-06-03T10:18:07Z

@phanikumarp - What's the solution you found for this problem ? We are facing same issue and need a way to advertise external IP for kube-api server.

hocgin · 2021-09-30T07:02:18Z

@Zhang21 after a lot of day and night research... i finally found a way to let k8s work over WAN.

short answer

--apiserver-advertise-address=publicIP is necessary, this tell k8s worker communicate to master over publicIP. default is privateIP, will lead to 10.96.0.1:443: i/o timeout.

node annotation flannel.alpha.coreos.com/public-ip-overwrite=publicIP is necessary，this set flannel pod node-ip to publicIP

full answer

first, run ifconfig on master, check whether there have publicIP info on master interfaces. some cloud provider use Elastic IP, they dont have any publicIP interface info. you must add publicIP interface info yourself. follow Cloud_floating_IP_persistent. if you dont add publicIP interface, kubeadm init --apiserver-advertise-address=publicIP will not success.

kubeadm init with --apiserver-advertise-address=publicIP, i use --control-plane-endpoint=publicIP --upload-certs --apiserver-advertise-address=publicIP for my own. i think just apiserver-advertise-address also will be ok.

apply flannel.yaml and join worker node.

on worker node，add kubelet args --node-ip
vim /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS=--cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --node-ip=publicIP
systemctl restart kubelet
set worker node public-ip-overwrite annotate, and restart worker kube-flannel pod
kubectl annotate node NODE_NAME flannel.alpha.coreos.com/public-ip-overwrite=publicIP
# on worker node
systemctl restart docker

⚠️ I tried this solution.
It can solve the apiserver timeout problem.
but, It will prevent flannel packets from being sent to the destination in Alibaba Cloud.
I suggest not to try [Cloud_floating_IP_persistent].

adamwithit · 2021-10-12T06:55:03Z

@Zhang21 after a lot of day and night research... i finally found a way to let k8s work over WAN.
short answer

--apiserver-advertise-address=publicIP is necessary, this tell k8s worker communicate to master over publicIP. default is privateIP, will lead to 10.96.0.1:443: i/o timeout.

node annotation flannel.alpha.coreos.com/public-ip-overwrite=publicIP is necessary，this set flannel pod node-ip to publicIP

full answer

first, run ifconfig on master, check whether there have publicIP info on master interfaces. some cloud provider use Elastic IP, they dont have any publicIP interface info. you must add publicIP interface info yourself. follow Cloud_floating_IP_persistent. if you dont add publicIP interface, kubeadm init --apiserver-advertise-address=publicIP will not success.

kubeadm init with --apiserver-advertise-address=publicIP, i use --control-plane-endpoint=publicIP --upload-certs --apiserver-advertise-address=publicIP for my own. i think just apiserver-advertise-address also will be ok.

apply flannel.yaml and join worker node.

on worker node，add kubelet args --node-ip
vim /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS=--cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --node-ip=publicIP
systemctl restart kubelet
set worker node public-ip-overwrite annotate, and restart worker kube-flannel pod
kubectl annotate node NODE_NAME flannel.alpha.coreos.com/public-ip-overwrite=publicIP
# on worker node
systemctl restart docker
warning I tried this solution. It can solve the apiserver timeout problem. but, It will prevent flannel packets from being sent to the destination in Alibaba Cloud. I suggest not to try [Cloud_floating_IP_persistent].

I spend a lot of time in this solution. It work great in the beginning, I can create nodes in two vm (which do not have public ip, all rely on port forwarding ) across two wan, flannel and ingress create successfully. however, when come to udp connection between two node, udp package seen like dropped by router because of the src ip is a WAN IP. so I believe unless you have two machine directly connect to wan or you can counter the anti spoofing rule in your router, this is not a good attend in this topic. I also wonder if there anyway we can send a udp package with WAN src ip from a machine in LAN. Any trick?

janjangao · 2021-10-12T07:38:35Z

@Zhang21 after a lot of day and night research... i finally found a way to let k8s work over WAN.
short answer

--apiserver-advertise-address=publicIP is necessary, this tell k8s worker communicate to master over publicIP. default is privateIP, will lead to 10.96.0.1:443: i/o timeout.

node annotation flannel.alpha.coreos.com/public-ip-overwrite=publicIP is necessary，this set flannel pod node-ip to publicIP

full answer

first, run ifconfig on master, check whether there have publicIP info on master interfaces. some cloud provider use Elastic IP, they dont have any publicIP interface info. you must add publicIP interface info yourself. follow Cloud_floating_IP_persistent. if you dont add publicIP interface, kubeadm init --apiserver-advertise-address=publicIP will not success.

kubeadm init with --apiserver-advertise-address=publicIP, i use --control-plane-endpoint=publicIP --upload-certs --apiserver-advertise-address=publicIP for my own. i think just apiserver-advertise-address also will be ok.

apply flannel.yaml and join worker node.

on worker node，add kubelet args --node-ip
vim /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS=--cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --node-ip=publicIP
systemctl restart kubelet
set worker node public-ip-overwrite annotate, and restart worker kube-flannel pod
kubectl annotate node NODE_NAME flannel.alpha.coreos.com/public-ip-overwrite=publicIP
# on worker node
systemctl restart docker
warning I tried this solution. It can solve the apiserver timeout problem. but, It will prevent flannel packets from being sent to the destination in Alibaba Cloud. I suggest not to try [Cloud_floating_IP_persistent].
I spend a lot of time in this solution. It work great in the beginning, I can create nodes in two vm (which do not have public ip, all rely on port forwarding ) across two wan, flannel and ingress create successfully. however, when come to udp connection between two node, udp package seen like dropped by router because of the src ip is a WAN IP. so I believe unless you have two machine directly connect to wan or you can counter the anti spoofing rule in your router, this is not a good attend in this topic. I also wonder if there anyway we can send a udp package with WAN src ip from a machine in LAN. Any trick?

yes... still have problem... I am tired, It's not worth to speed lot of time for it, I finally buy an another machine under the lan, I found spending money makes me happy.

janjangao · 2021-10-12T07:44:19Z

@Zhang21 after a lot of day and night research... i finally found a way to let k8s work over WAN.
short answer

--apiserver-advertise-address=publicIP is necessary, this tell k8s worker communicate to master over publicIP. default is privateIP, will lead to 10.96.0.1:443: i/o timeout.

node annotation flannel.alpha.coreos.com/public-ip-overwrite=publicIP is necessary，this set flannel pod node-ip to publicIP

full answer

first, run ifconfig on master, check whether there have publicIP info on master interfaces. some cloud provider use Elastic IP, they dont have any publicIP interface info. you must add publicIP interface info yourself. follow Cloud_floating_IP_persistent. if you dont add publicIP interface, kubeadm init --apiserver-advertise-address=publicIP will not success.

kubeadm init with --apiserver-advertise-address=publicIP, i use --control-plane-endpoint=publicIP --upload-certs --apiserver-advertise-address=publicIP for my own. i think just apiserver-advertise-address also will be ok.

apply flannel.yaml and join worker node.

on worker node，add kubelet args --node-ip
vim /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS=--cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --node-ip=publicIP
systemctl restart kubelet
set worker node public-ip-overwrite annotate, and restart worker kube-flannel pod
kubectl annotate node NODE_NAME flannel.alpha.coreos.com/public-ip-overwrite=publicIP
# on worker node
systemctl restart docker
⚠️ I tried this solution. It can solve the apiserver timeout problem. but, It will prevent flannel packets from being sent to the destination in Alibaba Cloud. I suggest not to try [Cloud_floating_IP_persistent].

still have problem in alibaba cloud, I gave up, finally buy an another machine under the lan ^_^, I found I should spend more time in my life, rather than this stupid damn k8s.

neolit123 · 2021-10-12T10:56:40Z

Alternatively try another cni plugin. The kubeadm team no longer recommends flannel due to bugs in the past. Also passing a custom ip to all components is difficult to maintain and instead the host network / default route can be customized...but there are no docs for that in the k8s.io website yet.

adamwithit · 2021-11-03T08:59:51Z

finally I make it work, first I follow @hayond step to init two node across two WAN, after init and apply flannel, I change the interface back to LAN ip address eg. 192.168.x.x (using netplan) and do the iptables trick to route the traffic to actual WAN IP. That make udp package sent correctly with flannel. now my flannel using LAN ip address and work perfect so I wonder It write down the interface IP during kubeadm init process somewhere and use it in health check but using the current interface ip for udp VXLAN communication.

mk2134226 · 2022-08-07T23:36:20Z

@adamwithit Did your setting kept working or caused some other issues down the track.

Ryder05 · 2022-09-17T11:52:19Z

I manged to solve the problem by enabling inbound traffic in port 6443 and used the flag --control-plane-endpoint instead of --apiserver-advertise-address

HairukanLin · 2023-07-29T21:35:19Z

I manged to solve the problem by enabling inbound traffic in port 6443 and used the flag --control-plane-endpoint instead of --apiserver-advertise-address

Thank you. This helped me. I am using a gcp virtual machine and your instructions helped. Hope other people will find this as well.

Tuanshu · 2023-11-01T02:00:49Z

I manged to solve the problem by enabling inbound traffic in port 6443 and used the flag --control-plane-endpoint instead of --apiserver-advertise-address

Thank you a lot. this solved my problem as well. Hope this can be better documented in tutorial.

neolit123 added the priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. label Feb 6, 2019

k8s-ci-robot added the sig/network Categorizes an issue or PR as relevant to SIG Network. label Feb 7, 2019

neolit123 closed this as completed Mar 18, 2019

kubeadm init “--apiserver-advertise-address=publicIP” not working, private IP works 1.13 version #1390

kubeadm init “--apiserver-advertise-address=publicIP” not working, private IP works 1.13 version #1390

Comments

phanikumarp commented Feb 6, 2019 • edited Loading

Versions

What happened?

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

neolit123 commented Feb 6, 2019

phanikumarp commented Feb 7, 2019 • edited Loading

neolit123 commented Feb 7, 2019

phanikumarp commented Feb 7, 2019 via email

neolit123 commented Feb 7, 2019

spockmang commented Feb 12, 2019

sensre commented Feb 18, 2019

phanikumarp commented Feb 19, 2019

bcheerla4509 commented Feb 25, 2019

phanisowjanyavutukuri commented Mar 18, 2019

fabriziopandini commented Mar 18, 2019

neolit123 commented Mar 18, 2019

bobbydeveaux commented Mar 26, 2019

phanikumarp commented Mar 27, 2019 via email

swapblue commented Apr 12, 2019

x22n commented Apr 14, 2019

x22n commented Apr 17, 2019

ASLanin commented May 28, 2019

pankajcheema commented Jun 6, 2019

atshakil commented Jul 2, 2019 • edited Loading

Update

asymness commented Jul 10, 2019

Noobgam commented Jul 20, 2019

KevinWang15 commented Aug 5, 2019

kioyong commented Aug 29, 2019

Zhang21 commented Sep 24, 2019 • edited Loading

Zhang21 commented Sep 24, 2019 • edited Loading

janjangao commented Sep 27, 2019

Zhang21 commented Sep 27, 2019

janjangao commented Oct 9, 2019 • edited Loading

heangratha commented Apr 11, 2020

s977120 commented Apr 20, 2020

saarram commented Jun 3, 2020

hocgin commented Sep 30, 2021

adamwithit commented Oct 12, 2021

janjangao commented Oct 12, 2021 • edited Loading

janjangao commented Oct 12, 2021

neolit123 commented Oct 12, 2021 via email

adamwithit commented Nov 3, 2021 • edited Loading

mk2134226 commented Aug 7, 2022

Ryder05 commented Sep 17, 2022

HairukanLin commented Jul 29, 2023

Tuanshu commented Nov 1, 2023

phanikumarp commented Feb 6, 2019 •

edited

Loading

phanikumarp commented Feb 7, 2019 •

edited

Loading

atshakil commented Jul 2, 2019 •

edited

Loading

Zhang21 commented Sep 24, 2019 •

edited

Loading

Zhang21 commented Sep 24, 2019 •

edited

Loading

janjangao commented Oct 9, 2019 •

edited

Loading

janjangao commented Oct 12, 2021 •

edited

Loading

adamwithit commented Nov 3, 2021 •

edited

Loading