如果想要在K8S环境上部署某软件的话，那么首先先要了解的是该软件在Bare Metal裸金属环境下的手动安装方式、如何配置，启动等等；然后再考虑如何将软件Docker化，怎么编写Dockerfile，Docker环境下如何配置、启动等。最后根据之前Docker的方式选用K8S的YAML资源来创建该服务，例如：Deployment、DaemonSet、Service，ConfigMap，Secret。最后，建议做成Helm Chart的方式，来发布，部署，管理。
按照这个步骤，我们之前已经了解了手动安装Nacos集群的方式，参考：手动安装Nacos集群 ，那么这里我们先了解一下Docker化的Nacos，以及如何部署在K8S环境中。
官网：https://nacos.io/
GitHub：https://github.com/alibaba/nacos
DockerHub：https://hub.docker.com/r/nacos/nacos-server
Docker：https://github.com/nacos-group/nacos-docker
Nacos On Kubernetes：https://github.com/nacos-group/nacos-k8s
环境说明：
Kubernetes 1.15.5
Nacos 1.1.4
了解Nacos Docker化
参考：
https://nacos.io/zh-cn/docs/quick-start-docker.html
https://github.com/nacos-group/nacos-docker
首先拿到Nacos的镜像为：nacos/nacos-server，然后到Dockerhub 中找到对应的镜像，一般里面包含了使用配置说明，以及Dockerfile的GitHub。
这里我们拿到了Nacos Dockerfile的GitHub：https://www.github.com/nacos-group/nacos-docker，经查看大部分的配置都能通过环境变量的方式配置，这也是Docker化的推荐方式，下面是具体的配置项：
环境变量配置项
name
description
option
MODE
cluster模式/standalone模式
cluster/standalone default cluster
NACOS_SERVERS
nacos cluster地址
eg. ip1,ip2,ip3
PREFER_HOST_MODE
是否支持hostname
hostname/ip default ip
NACOS_SERVER_PORT
nacos服务器端口
default 8848
NACOS_SERVER_IP
多网卡下的自定义nacos服务器IP
SPRING_DATASOURCE_PLATFORM
standalone 支持 mysql
mysql / empty default empty
MYSQL_MASTER_SERVICE_HOST
mysql 主节点host地址
MYSQL_MASTER_SERVICE_PORT
mysql 主节点端口
default : 3306
MYSQL_MASTER_SERVICE_DB_NAME
mysql 主节点数据库名称
MYSQL_MASTER_SERVICE_USER
数据库用户名
MYSQL_MASTER_SERVICE_PASSWORD
数据库密码
MYSQL_SLAVE_SERVICE_HOST
mysql从节点host地址
MYSQL_SLAVE_SERVICE_PORT
mysql从节点端口
default :3306
MYSQL_DATABASE_NUM
数据库数量
default :2
JVM_XMS
-Xms
default :2g
JVM_XMX
-Xmx
default :2g
JVM_XMN
-Xmn
default :1g
JVM_MS
-XX:MetaspaceSize
default :128m
JVM_MMS
-XX:MaxMetaspaceSize
default :320m
NACOS_DEBUG
开启远程调试
y/n default :n
TOMCAT_ACCESSLOG_ENABLED
server.tomcat.accesslog.enabled
default :false
知道了Nacos 在docker中的配置方式是使用环境变量，以及具体的环境变量对应的配置，但是我们不了解是如何使用环境变量配置Nacos的，接着往下看：
找到具体的Dockerfile 文件：
https://github.com/nacos-group/nacos-docker/tree/master/build
位置：
https://github.com/nacos-group/nacos-docker/blob/master/build/Dockerfile
FROM centos:7.5 .1804 MAINTAINER pader "huangmnlove@163.com" ENV MODE="cluster" \ PREFER_HOST_MODE="ip" \ BASE_DIR="/home/nacos" \ CLASSPATH=".:/home/nacos/conf:$CLASSPATH" \ CLUSTER_CONF="/home/nacos/conf/cluster.conf" \ FUNCTION_MODE="all" \ JAVA_HOME="/usr/lib/jvm/java-1.8.0-openjdk" \ NACOS_USER="nacos" \ JAVA="/usr/lib/jvm/java-1.8.0-openjdk/bin/java" \ JVM_XMS="2g" \ JVM_XMX="2g" \ JVM_XMN="1g" \ JVM_MS="128m" \ JVM_MMS="320m" \ NACOS_DEBUG="n" \ TOMCAT_ACCESSLOG_ENABLED="false" \ TIME_ZONE="Asia/Shanghai" ARG NACOS_VERSION=1.1 .4 WORKDIR /$BASE_DIR RUN set -x \ && yum update -y \ && yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel wget iputils nc vim libcurl\ && wget https://github.com/alibaba/nacos/releases/download/${NACOS_VERSION} /nacos-server-${NACOS_VERSION} .tar.gz -P /home \ && tar -xzvf /home/nacos-server-${NACOS_VERSION} .tar.gz -C /home \ && rm -rf /home/nacos-server-${NACOS_VERSION} .tar.gz /home/nacos/bin/* /home/nacos/conf/*.properties /home/nacos/conf/*.example /home/nacos/conf/nacos-mysql.sql \ && yum autoremove -y wget \ && ln -snf /usr/share/zoneinfo/$TIME_ZONE /etc/localtime && echo '$TIME_ZONE' > /etc/timezone \ && yum clean all ADD bin/docker-startup.sh bin/docker-startup.sh ADD conf/application.properties conf/application.properties ADD init.d/custom.properties init.d/custom.properties RUN mkdir -p logs \ && cd logs \ && touch start.out \ && ln -sf /dev/stdout start.out \ && ln -sf /dev/stderr start.out RUN chmod +x bin/docker-startup.sh EXPOSE 8848 ENTRYPOINT ["bin/docker-startup.sh" ]
其次查看bin/docker-startup.sh 启动脚本：
位置：https://github.com/nacos-group/nacos-docker/blob/master/build/bin/docker-startup.sh
#!/bin/bash set -xexport DEFAULT_SEARCH_LOCATIONS="classpath:/,classpath:/config/,file:./,file:./config/" export CUSTOM_SEARCH_LOCATIONS=${DEFAULT_SEARCH_LOCATIONS} ,file:${BASE_DIR} /conf/,${BASE_DIR} /init.d/export CUSTOM_SEARCH_NAMES="application,custom" PLUGINS_DIR="/home/nacos/plugins/peer-finder" function print_servers if [[ ! -d "${PLUGINS_DIR} " ]]; then echo "" > "$CLUSTER_CONF " for server in ${NACOS_SERVERS} ; do echo "$server " >> "$CLUSTER_CONF " done else bash $PLUGINS_DIR /plugin.sh sleep 30 4fi } if [[ "${MODE} " == "standalone" ]]; then JAVA_OPT="${JAVA_OPT} -Xms512m -Xmx512m -Xmn256m" JAVA_OPT="${JAVA_OPT} -Dnacos.standalone=true" else JAVA_OPT="${JAVA_OPT} -server -Xms${JVM_XMS} -Xmx${JVM_XMX} -Xmn${JVM_XMN} -XX:MetaspaceSize=${JVM_MS} -XX:MaxMetaspaceSize=${JVM_MMS} " if [[ "${NACOS_DEBUG} " == "y" ]]; then JAVA_OPT="${JAVA_OPT} -Xdebug -Xrunjdwp:transport=dt_socket,address=9555,server=y,suspend=n" fi JAVA_OPT="${JAVA_OPT} -XX:-OmitStackTraceInFastThrow -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${BASE_DIR} /logs/java_heapdump.hprof" JAVA_OPT="${JAVA_OPT} -XX:-UseLargePages" print_servers fi if [[ "${FUNCTION_MODE} " == "config" ]]; then JAVA_OPT="${JAVA_OPT} -Dnacos.functionMode=config" elif [[ "${FUNCTION_MODE} " == "naming" ]]; then JAVA_OPT="${JAVA_OPT} -Dnacos.functionMode=naming" fi if [[ ! -z "${NACOS_SERVER_IP} " ]]; then JAVA_OPT="${JAVA_OPT} -Dnacos.server.ip=${NACOS_SERVER_IP} " fi if [[ ! -z "${USE_ONLY_SITE_INTERFACES} " ]]; then JAVA_OPT="${JAVA_OPT} -Dnacos.inetutils.use-only-site-local-interfaces=${USE_ONLY_SITE_INTERFACES} " fi if [[ ! -z "${PREFERRED_NETWORKS} " ]]; then JAVA_OPT="${JAVA_OPT} -Dnacos.inetutils.preferred-networks=${PREFERRED_NETWORKS} " fi if [[ ! -z "${IGNORED_INTERFACES} " ]]; then JAVA_OPT="${JAVA_OPT} -Dnacos.inetutils.ignored-interfaces=${IGNORED_INTERFACES} " fi if [[ "${PREFER_HOST_MODE} " == "hostname" ]]; then JAVA_OPT="${JAVA_OPT} -Dnacos.preferHostnameOverIp=true" fi JAVA_MAJOR_VERSION=$($JAVA -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p' ) if [[ "$JAVA_MAJOR_VERSION " -ge "9" ]] ; then JAVA_OPT="${JAVA_OPT} -cp .:${BASE_DIR} /plugins/cmdb/*.jar:${BASE_DIR} /plugins/mysql/*.jar" JAVA_OPT="${JAVA_OPT} -Xlog:gc*:file=${BASE_DIR} /logs/nacos_gc.log:time,tags:filecount=10,filesize=102400" else JAVA_OPT="${JAVA_OPT} -Djava.ext.dirs=${JAVA_HOME} /jre/lib/ext:${JAVA_HOME} /lib/ext:${BASE_DIR} /plugins/cmdb:${BASE_DIR} /plugins/mysql" JAVA_OPT="${JAVA_OPT} -Xloggc:${BASE_DIR} /logs/nacos_gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M" fi JAVA_OPT="${JAVA_OPT} -Dnacos.home=${BASE_DIR} " JAVA_OPT="${JAVA_OPT} -jar ${BASE_DIR} /target/nacos-server.jar" JAVA_OPT="${JAVA_OPT} ${JAVA_OPT_EXT} " JAVA_OPT="${JAVA_OPT} --spring.config.location=${CUSTOM_SEARCH_LOCATIONS} " JAVA_OPT="${JAVA_OPT} --spring.config.name=${CUSTOM_SEARCH_NAMES} " JAVA_OPT="${JAVA_OPT} --logging.config=${BASE_DIR} /conf/nacos-logback.xml" JAVA_OPT="${JAVA_OPT} --server.max-http-header-size=524288" echo "nacos is starting,you can check the ${BASE_DIR} /logs/start.out" echo "$JAVA ${JAVA_OPT} " > ${BASE_DIR} /logs/start.out 2>&1 &nohup $JAVA ${JAVA_OPT} > ${BASE_DIR} /logs/start.out 2>&1 < /dev/null
再来看一下application.properties：
位置：
https://github.com/nacos-group/nacos-docker/blob/master/build/conf/application.properties
# spring server.servlet.contextPath=${SERVER_SERVLET_CONTEXTPATH:/nacos} server.contextPath=/nacos server.port=${NACOS_SERVER_PORT:8848} spring.datasource.platform=${SPRING_DATASOURCE_PLATFORM:""} nacos.cmdb.dumpTaskInterval=3600 nacos.cmdb.eventTaskInterval=10 nacos.cmdb.labelTaskInterval=300 nacos.cmdb.loadDataAtStart=false db.num=${MYSQL_DATABASE_NUM:2} db.url.0=jdbc:mysql://${MYSQL_MASTER_SERVICE_HOST}:${MYSQL_MASTER_SERVICE_PORT:3306}/${MYSQL_MASTER_SERVICE_DB_NAME}?characterEncoding=utf8&connectTimeout=1000&socketTimeout=3000&autoReconnect=true db.url.1=jdbc:mysql://${MYSQL_SLAVE_SERVICE_HOST}:${MYSQL_SLAVE_SERVICE_PORT:3306}/${MYSQL_MASTER_SERVICE_DB_NAME}?characterEncoding=utf8&connectTimeout=1000&socketTimeout=3000&autoReconnect=true db.user=${MYSQL_MASTER_SERVICE_USER} db.password=${MYSQL_MASTER_SERVICE_PASSWORD} server.tomcat.accesslog.enabled=${TOMCAT_ACCESSLOG_ENABLED:false} server.tomcat.accesslog.pattern=%h %l %u %t "%r" %s %b %D # default current work dir server.tomcat.basedir= ## spring security config ### turn off security nacos.security.ignore.urls=/,/**/*.css,/**/*.js,/**/*.html,/**/*.map,/**/*.svg,/**/*.png,/**/*.ico,/console-fe/public/**,/v1/auth/login,/v1/console/health/**,/v1/cs/**,/v1/ns/**,/v1/cmdb/**,/actuator/**,/v1/console/server/** # metrics for elastic search management.metrics.export.elastic.enabled=false management.metrics.export.influx.enabled=false nacos.naming.distro.taskDispatchThreadCount=10 nacos.naming.distro.taskDispatchPeriod=200 nacos.naming.distro.batchSyncKeyCount=1000 nacos.naming.distro.initDataRatio=0.9 nacos.naming.distro.syncRetryDelay=5000 nacos.naming.data.warmup=true
这里就是如何通过环境变量的方式配置Nacos的啦，如果变量名为空，那么将使用冒号后面的默认值。
你还记得BASH中如何设置成这种效果嘛？
$ echo ${name} $ echo ${name:-NAME} NAME $ echo ${name} $ echo ${name} $ echo ${name:=NAME} NAME ~$ echo ${name} NAME $ echo ${age} $ echo ${age:?AGE NULL} -bash: age: AGE NULL $ echo ${age:?} -bash: age: parameter null or not set $ age='' $ echo ${age:?} -bash: age: parameter null or not set $ age=22 $ echo ${age:?} 22 $ echo ${sex:+3} $ sex=1 $ echo ${sex:+3} 3 $ echo ${sex} 1
好了，Nacos如何实现Docker化的我们大概了解了，下面就是如何在Kubernetes安装Docker化的Nacos。
在Kubernetes环境下安装Nacos集群
参考：
https://nacos.io/zh-cn/docs/use-nacos-with-kubernetes.html
https://github.com/nacos-group/nacos-k8s/tree/master/deploy/nacos
这里我们是用nfs-client-provisioner 来提供持久化，关于nfs-client-provisioner 的安装请参考：在Kubernetes上安装nfs-client-provisioner来提供StorageClass
修改基础文件：
$ wget -c "https://github.com/nacos-group/nacos-k8s/raw/master/deploy/nacos/nacos-pvc-nfs.yaml"
YAML 修改成如下：
$ cat nacos-pvc-nfs.yaml --- apiVersion: v1 kind: Service metadata: name: nacos-headless labels: app: nacos annotations: service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" spec: ports: - port: 8848 name: server targetPort: 8848 clusterIP: None selector: app: nacos --- apiVersion: v1 kind: ConfigMap metadata: name: nacos-cm data: spring.datasource.platform: "mysql" mysql.database.num: "1" mysql.master.host: "x.x.x.x" mysql.master.db.name: "nacos" mysql.master.port: "3306" mysql.master.user: "nacos" mysql.master.password: "nacos-password" nacos_server_port: "8848" prefer_host_mode: "hostname" nacos_servers: "nacos-0.nacos-headless.default.svc.cluster.local:8848 nacos-1.nacos-headless.default.svc.cluster.local:8848 nacos-2.nacos-headless.default.svc.cluster.local:8848" --- apiVersion: apps/v1 kind: StatefulSet metadata: name: nacos spec: serviceName: nacos-headless replicas: 3 template: metadata: labels: app: nacos annotations: pod.alpha.kubernetes.io/initialized: "true" spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: In values: - nacos topologyKey: "kubernetes.io/hostname" containers: - name: nacos imagePullPolicy: IfNotPresent image: nacos/nacos-server:1.1.4 resources: requests: memory: "2Gi" cpu: "100m" limits: memory: "2Gi" cpu: "300m" ports: - containerPort: 8848 name: client-port env: - name: NACOS_REPLICAS value: "3" - name: SERVICE_NAME value: "nacos-headless" - name: POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: SPRING_DATASOURCE_PLATFORM valueFrom: configMapKeyRef: name: nacos-cm key: spring.datasource.platform - name: MYSQL_MASTER_SERVICE_HOST valueFrom: configMapKeyRef: name: nacos-cm key: mysql.master.host - name: MYSQL_MASTER_SERVICE_DB_NAME valueFrom: configMapKeyRef: name: nacos-cm key: mysql.master.db.name - name: MYSQL_MASTER_SERVICE_PORT valueFrom: configMapKeyRef: name: nacos-cm key: mysql.master.port - name: MYSQL_DATABASE_NUM valueFrom: configMapKeyRef: name: nacos-cm key: mysql.database.num - name: MYSQL_MASTER_SERVICE_USER valueFrom: configMapKeyRef: name: nacos-cm key: mysql.master.user - name: MYSQL_MASTER_SERVICE_PASSWORD valueFrom: configMapKeyRef: name: nacos-cm key: mysql.master.password - name: JVM_XMS_NUM valueFrom: resourceFieldRef: divisor: 1Gi resource: requests.memory - name: JVM_XMS value: $(JVM_XMS_NUM)g - name: JVM_XMX_NUM valueFrom: resourceFieldRef: divisor: 1Gi resource: limits.memory - name: JVM_XMX value: $(JVM_XMX_NUM)g - name: JVM_XMN value: 1g - name: NACOS_SERVER_PORT valueFrom: configMapKeyRef: name: nacos-cm key: nacos_server_port - name: PREFER_HOST_MODE valueFrom: configMapKeyRef: name: nacos-cm key: prefer_host_mode - name: NACOS_SERVERS valueFrom: configMapKeyRef: name: nacos-cm key: nacos_servers readinessProbe: httpGet: port: client-port path: /nacos/v1/console/health/readiness initialDelaySeconds: 90 timeoutSeconds: 3 livenessProbe: httpGet: port: client-port path: /nacos/v1/console/health/liveness initialDelaySeconds: 90 timeoutSeconds: 3 volumeMounts: - name: datadir mountPath: /home/nacos/data - name: logdir mountPath: /home/nacos/logs volumeClaimTemplates: - metadata: name: datadir spec: storageClassName: nfs-client accessModes: [ "ReadWriteMany" ] resources: requests: storage: 1Gi - metadata: name: logdir spec: storageClassName: nfs-client accessModes: [ "ReadWriteMany" ] resources: requests: storage: 1Gi selector: matchLabels: app: nacos --- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nacos annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: nacos.test.aws.test.com http: paths: - path: /nacos backend: serviceName: nacos-headless servicePort: server
修改说明：
没有使用官方的initContainers的nacos/nacos-peer-finder-plugin:latest，经测试启动正常，也能看到页面，但是，一直无法查看到集群主机列表，一直提示：server is STARTING now, please try again later!，所有的API接口也提示这个。
所以直接通过环境变量指定了所有的Nacos Cluster 节点的地址。
使用resource中的资源充当-Xms,-Xmx，但是-Xmn,需要手动指定。
将环境变量的值，都从configmap获取。
带来的问题：扩缩容比较麻烦
应用yaml：
$ kubectl apply -f nacos-pvc-nfs.yaml
然后配置域名解析nacos.test.aws.test.com指定到nginx-ingress-controller的公网地址。
访问nacos.test.aws.test.com，查看集群节点列表：
注意：这里只是映射了Nacos的Dashboard，但是如果开发人员需要通过公网使用Nacos呢？
这里使用nginx-ingress-controller的TCP转发功能：
首先：确认nginx-ingress-controller参数中使用的TCP的configmap是哪个？
如果是通过Helm安装的话：在values.yaml如果tcp和udp的配置是空的，那么默认不会添加这两个参数的：--tcp-services-configmap和--udp-services-configmap 建议通过Helm update进行升级。
containers: - args: - /nginx-ingress-controller - --default-backend-service=kube-system/nginx-ingress-default-backend - --election-id=ingress-controller-leader - --ingress-class=nginx - --configmap=kube-system/nginx-ingress-controller - --tcp-services-configmap=kube-system/nginx-ingress-tcp - --udp-services-configmap=kube-system/nginx-ingress-udp
我们来编辑这个configMap：
$ kubectl -n kube-system edit cm nginx-ingress-tcp apiVersion: v1 data: "2181": default/zookeeper:2181 "8848": default/nacos-headless:8848 kind: ConfigMap metadata: creationTimestamp: "2019-11-27T08:58:01Z" labels: app: nginx-ingress chart: nginx-ingress-1.6.0 component: controller heritage: Tiller release: dev-microoak-cn-ingress name: nginx-ingress-tcp namespace: kube-system resourceVersion: "96042220" selfLink: /api/v1/namespaces/kube-system/configmaps/nginx-ingress-tcp uid: 04158536 -10f4-11ea-9238-0296672f1eaa
在data下添加：
"8848": microoak/nacos-headless:8848
格式如下：
PORT: <namespace>/<service name>:<service port> PORT 为nginx-ingress-controller监听的端口，并将流量转发到后端 <namespace> 为nginx-ingress-controller将流量转发到的后端所在的namespace <service name> 为nginx-ingress-controller将流量转发到的后端的service name，这里是nacos-headless <service port> 为nginx-ingress-controller将流量转发到的后端的service port，这里是8848
然后就可以通过：
<nginx-ingress-controller IP>:PORT 访问了；
如果nginx-ingress-controller有多个，最好前边再加个LB。通过访问LB IP+LB PORT的方式访问。
本文到这里就结束了，欢迎期待后面的文章。您可以关注下方的公众号二维码，在第一时间查看新文章。
