阿里云K8S私有Ingress Controller的配置和使用
创建集群
进入阿里云容器服务控制台，创建一个新的k8s集群，此时集群会自动生成一个公网的Ingress Controller和一个公网的SLB监听着Worker的80和443端口。
默认的公网Ingress Controller
默认的公网SLB（SLB名字是自己起的，为了方便看）
创建私有SLB
进入负载均衡控制台创建一个私有SLB，实例规格按实际业务需求。注意：专有网络必须和刚才创建的集群的一样！！！
配置私有Ingress Controller
由于Ingress Controller Pods中的serviceAccountName是引用集群默认创建的，所以在此就不再配置ServiceAccount、ClusterRole和ClusterRoleBinding。
-
Nginx ConfigMap
apiVersion: v1 kind: ConfigMap metadata:name: private-nginx-configuration #名字可以自己改
namespace: kube-system
labels:
app: ingress-nginxdata:proxy-body-size: 20m
proxy-connect-timeout: "10"
max-worker-connections: "65536"
enable-underscores-in-headers: "true"
reuse-port: "true"
worker-cpu-affinity: "auto"
server-tokens: "false"
ssl-redirect: "false"
allow-backend-server-header: "true"
ignore-invalid-headers: "true"
generate-request-id: "true"
#forwarded-for-header: "X-Real-IP"
#compute-full-forwarded-for: "true"
#hsts: "false"
#enable-vts-status: "true"
#use-proxy-protocol: "true"--- # nginx tcp stream config map kind: ConfigMap apiVersion: v1 metadata:name: private-tcp-services
namespace: kube-system--- # nginx udp stream config map kind: ConfigMap apiVersion: v1 metadata:name: private-udp-services
namespace: kube-system -
Nginx Ingress Controller Pods
apiVersion: apps/v1 kind: Deployment metadata:name: private-nginx-ingress-controller
labels:
app: private-ingress-nginx
namespace: kube-system
annotations:
component.version: '0.22.0'
component.revision: '5'spec:replicas: 2
selector:
matchLabels:app: private-ingress-nginxtemplate:
metadata:labels: app: private-ingress-nginx annotations: prometheus.io/port: "10254" prometheus.io/scrape: "true"spec:
#tolerations: # - key: node-role.kubernetes.io/master # effect: NoSchedule affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - ingress-nginx topologyKey: "kubernetes.io/hostname" #use default serviceAccountName serviceAccountName: nginx-ingress-controller initContainers: - name: init-sysctl image: registry-vpc.cn-hongkong.aliyuncs.com/acs/busybox:latest command: - /bin/sh - -c - | sysctl -w net.core.somaxconn=65535 sysctl -w net.ipv4.ip_local_port_range="1024 65535" sysctl -w fs.file-max=1048576 sysctl -w fs.inotify.max_user_instances=16384 sysctl -w fs.inotify.max_user_watches=524288 sysctl -w fs.inotify.max_queued_events=16384 securityContext: privileged: true containers: - name: nginx-ingress-controller image: registry-vpc.cn-hongkong.aliyuncs.com/acs/aliyun-ingress-controller:v0.22.0.5-552e0db-aliyun args: - /nginx-ingress-controller - --configmap=$(POD_NAMESPACE)/private-nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/private-tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/private-udp-services - --annotations-prefix=nginx.ingress.kubernetes.io - --publish-service=$(POD_NAMESPACE)/private-nginx-ingress-lb - --ingress-class=private #自定义名 - --v=2 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 - name: https containerPort: 443 livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 securityContext: capabilities: drop: - ALL add: - NET_BIND_SERVICE runAsUser: 33 volumeMounts: - name: localtime mountPath: /etc/localtime readOnly: true nodeSelector: beta.kubernetes.io/os: linux volumes: - name: localtime hostPath: path: /etc/localtime type: File -
Nginx Ingress Service
apiVersion: v1 kind: Service metadata:name: private-nginx-ingress-lb
namespace: kube-system
labels:
app: private-nginx-ingress-lb
annotations:
# set loadbalancer to the specified slb id
service.beta.kubernetes.io/alicloud-loadbalancer-id: lb-xxxx
# set loadbalancer address type to intranet if using private slb instance
service.beta.kubernetes.io/alicloud-loadbalancer-address-type: intranet
service.beta.kubernetes.io/alicloud-loadbalancer-force-override-listeners: 'true'spec:type: LoadBalancer
do not route traffic to other nodes
and reserve client ip for upstream
externalTrafficPolicy: "Local"
ports:- port: 80
name: http
targetPort: 80
- port: 443
name: https
targetPort: 443
selector:
# select app=private-ingress-nginx pods
app: private-ingress-nginx - port: 80
部署私有Ingress Controller
kubectl apply -f private-ingress-controller.yml- Private Ingress Pod
- Private Ingress LB Service
更新Clusterrole:nginx-ingress-controller
由于在配置私有Ingress Controller Pod时是引用集群默认的ServiceAccount，新生成的ingress-controller-leader-private配置项没有更新到默认的ClusterRole所以导致Service启动时会报没权限，此时我们需要在默认的ClusterRole中的resourceNames下添加ingress-controller-leader-private。
kubectl edit clusterrole nginx-ingress-controller -o yaml
使用阿里云DNS PrivateZone绑定SLB IP
- 进入云解析DNS控制台，开通PrivateZone并添加域名。
- 关联vpc
- 添加域名解析，绑定私有SLB IP
部署测试服务
apiVersion: apps/v1beta2
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: '1'
generation: 1
labels:
app: demo
name: demo
namespace: default
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: demo
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: demo
spec:
containers:
- image: >-
registry-vpc.cn-hongkong.aliyuncs.com/xxxx/demo:1.0.6-1
imagePullPolicy: Always
name: demo
resources:
limits:
cpu: 2048m
memory: 4Gi
requests:
cpu: 2048m
memory: 4Gi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
livenessProbe:
httpGet:
path: /v1/health
port: 80
initialDelaySeconds: 180
periodSeconds: 10
readinessProbe:
httpGet:
path: /v1/health
port: 80
initialDelaySeconds: 180
periodSeconds: 10
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 300
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: private
nginx.ingress.kubernetes.io/rewrite-target: /$2
generation: 1
name: private-demo-ingress
namespace: default
spec:
tls:
- hosts:
- k8s-test.internal.abc.com
# 配置TLS证书
secretName: abc.com
rules:
- host: k8s-test.internal.abc.com
http:
paths:
- backend:
serviceName: demo-svc
servicePort: 80
path: /abc(/|$)(.*)
---
apiVersion: v1
kind: Service
metadata:
name: demo-svc
namespace: default
spec:
clusterIP: None
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: demo
sessionAffinity: None
type: ClusterIP
测试结果
在同一个VPC下的ECS访问集群服务结果如下
