节点规划

节点名称节点最低配置集群角色IP地址docker版本rancher版本操作系统版本
node012c8gmaster192.168.189.2018.09.3rancher2.3Ubuntu18
node022c8gnode192.168.189.2118.09.3rancher2.3Ubuntu18
nginx2c1gnginx192.168.188.100nginx1.15.1 Ubuntu18

一、节点基础配置

注意:本章节操作需要在每个节点执行,用非root账户 使用sudo 加以操作

1.配置apt源

sudo cp /etc/apt/sources.list /etc/apt/sources.list.bak
sudo sed -i 's/http\:\/\/cn.archive.ubuntu.com/https\:\/\/mirrors.aliyun.com/g' /etc/apt/sources.list
sudo apt update

注:http://cn.archive.ubuntu.com 为原sources.list文件的apt源码仓库地址

2.关闭ufw(防火墙),配置hosts解析

sudo ufw disable
sudo vim /etc/hosts
192.168.189.20 node1
192.168.189.21 node2
#nginx服务代理ingrees服务
192.168.188.100 nginx abc.yefxx.cn

3.开启内核转发

sudo -i
cat >> /etc/sysctl.conf<<EOF
net.bridge.bridge-nf-call-ip6tables=1
net.bridge.bridge-nf-call-iptables=1
net.ipv4.ip_forward=1
net.ipv4.conf.all.forwarding=1
net.ipv4.neigh.default.gc_thresh1=4096
net.ipv4.neigh.default.gc_thresh2=6144
net.ipv4.neigh.default.gc_thresh3=8192
net.ipv4.neigh.default.gc_interval=60
net.ipv4.neigh.default.gc_stale_time=120
EOF
exit

4.时间同步

sudo ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
sudo apt install ntpdate
sudo crontab -e
#添加定时任务
*/2 * * * * ntpdate ntp1.aliyun.com > /dev/null

二、安装rancher

安装docker(公网环境下执行)

curl -fsSL https://get.docker.com |sudo bash -s docker --mirror Aliyun
#镜像加速
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
    "oom-score-adjust": -1000,
    "log-driver": "json-file",
    "log-opts": {
    "max-size": "100m",
    "max-file": "3"
    },
    "max-concurrent-downloads": 10,
    "max-concurrent-uploads": 10,
    "bip": "169.254.123.1/24",
    "registry-mirrors":   ["https://7bezldxe.mirror.aliyuncs.com","https://ip3m2hdn.mirror.aliyuncs.com"],
    "storage-driver": "overlay2",
    "storage-opts": [
    "overlay2.override_kernel_check=true"
    ]
}
EOF
#修改docker.service
sudo vim /lib/systemd/system/docker.service
#防止docker服务OOM: 
OOMScoreAdjust=-1000
ExecStartPost=/sbin/iptables -P FORWARD ACCEPT
sudo systemctl daemon-reload && sudo systemctl restart docker
#以下为可执行优化项 请谨慎执行
#解决docker info提示WARNING: No swap limit support
#sudo sed -i 's/GRUB_CMDLINE_LINUX="\(.*\)"/GRUB_CMDLINE_LINUX=" #cgroup_enable=memory swapaccount=1 \1"/g' /etc/default/grub
#sudo update-grub
#sudo reboot

1573014381709

普通用户执行docker

sudo usermod -aG docker spms

配置spms用户节点互相互信

ssh-keygen -f /home/spms/.ssh/id_rsa -N '' 
ssh-copy-id -i /home/spms/.ssh/id_rsa.pub node1
ssh-copy-id -i /home/spms/.ssh/id_rsa.pub node2
scp /home/spms/.ssh/id_rsa node01:/home/spms/.ssh/
scp /home/spms/.ssh/id_rsa node02:/home/spms/.ssh/

rke部署k8s集群

创建集群配置文件

cat >> rancher-cluster.yml <<EOF
nodes:
  - address: 168.168.189.20
    user: spms
    role: [controlplane,worker,etcd]
  - address: 168.168.189.21
    user: spms
    role: [controlplane,worker,etcd]


services:
  etcd:
    snapshot: true
    creation: 6h
    retention: 24h
EOF

创建k8s集群

wget https://www.rancher.cn/download/rke/v0.3.2-rke_linux-amd64
sudo chmod 777 v0.3.2-rke_linux-amd64 && sudo mv v0.3.2-rke_linux-amd64 /usr/local/bin/rke
rke up --config rancher-cluster.yml

下载并配置kubectl

wget https://www.rancher.cn/download/kubernetes/linux-amd64-v1.16.2-kubectl
sudo chmod 777 kubectl_amd64-linux
sudo mv kubectl_amd64-linux /usr/local/bin/kubectl
echo "source <(kubectl completion bash)" >> ~/.bashrc
source ~/.bashrc
mkdir ~/.kube
cp /home/spms/kube_config_rancher-cluster.yml /spms/.kube/config

检查k8s集群pod状态

kubectl get pods --all-namespaces
[spms@node1 ~]$ kubectl get pods --all-namespaces
NAMESPACE       NAME                                      READY   STATUS      RESTARTS   AGE
ingress-nginx   default-http-backend-7f8fbb85db-rxs9r     1/1     Running     0          106s
ingress-nginx   nginx-ingress-controller-9vhbj            1/1     Running     0          10m
ingress-nginx   nginx-ingress-controller-lhvk4            1/1     Running     0          10m
kube-system     canal-9lhlr                               2/2     Running     0          10m
kube-system     canal-xxz5p                               2/2     Running     0          10m
kube-system     kube-dns-5fd74c7488-54dgp                 3/3     Running     0          10m
kube-system     kube-dns-autoscaler-c89df977f-fb42z       1/1     Running     0          10m
kube-system     metrics-server-7fbd549b78-8hftl           1/1     Running     0          10m
kube-system     rke-ingress-controller-deploy-job-8c9c2   0/1     Completed   0          10m
kube-system     rke-kubedns-addon-deploy-job-lp5tc        0/1     Completed   0          10m
kube-system     rke-metrics-addon-deploy-job-j585d        0/1     Completed   0          10m
kube-system     rke-network-plugin-deploy-job-xssrc       0/1     Completed   0          10m

pod的状态只有以上两种状态为正常状态,若有其他状态则需要查看pod日志

kubectl describe pod pod-xxx -n namespace

helm将rancher部署在k8s集群

安装并配置helm客户端

wget https://www.rancher.cn/download/helm/helm-v2.15.2-linux-amd64.tar.gz
tar xf helm-v2.15.2-linux-amd64.tar.gz
sudo cp linux-amd64/helm /usr/local/bin/
sudo cp linux-amd64/tiller /usr/local/bin/

配置helm客户端具有访问k8s集群的权限

kubectl -n kube-system create serviceaccount tiller
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller

将helm server(titler)部署到k8s集群

helm init --service-account tiller   --tiller-image sapcc/tiller:v2.15.2 --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
helm init --service-account tiller --canary-image

检查tiller pod运行正常,若tiller pod不正常需要修复。

kubectl get pods --all-namespaces

为helm客户端配置chart仓库

helm repo add rancher-stable https://releases.rancher.com/server-charts/stable

检查rancher chart仓库可用

helm search rancher
NAME                      CHART VERSION    APP VERSION    DESCRIPTION                                                 
rancher-stable/rancher    2.3.2            v2.3.2         Install Rancher Server to manage Kubernetes clusters acro...

安装证书管理器

helm install stable/cert-manager   --name cert-manager   --namespace kube-system   --set rancherImage=rancher/rancher:stable

检查证书管理器pod是否正常

kubectl get pods --all-namespaces|grep cert-manager

安装rancher

helm install rancher-stable/rancher   --name rancher   --namespace cattle-system   --set hostname=abc.yefxx.cn

配置nginx

cat /etc/nginx/conf/rancher.conf
upstream rancher {
        server 192.168.189.20:443 max_fails=3 fail_timeout=5s;
        server 192.168.189.21:443 max_fails=3 fail_timeout=5s;
}

map $http_upgrade $connection_upgrade {
    default Upgrade;
    ''      close;
}

server {
    listen 443 ssl http2; # 如果是升级或者全新安装v2.2.2,需要禁止http2,其他版本不需修改。
    server_name abc.yefxx.cn;
    ssl_certificate /root/.acme.sh/yefxx.cn/fullchain.cer;
    ssl_certificate_key /root/.acme.sh/yefxx.cn/yefxx.cn.key;
location / {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass https://rancher;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    # This allows the ability for the execute shell window to remain open for up to 15 minutes. 
    ## Without this parameter, the default is 1 minute and will automatically close.
    proxy_read_timeout 900s;
    proxy_buffering off;
    }
}
server {
        listen 80;
        server_name abc.yefxx.cn;
        return 301 https://$server_name$request_uri;
    }

由于rancher HA需要通过域名访问,所有在本地添加hosts解析后。浏览器访问nginx

https://abc.yefxx.cn