360公司的集群管理平台 wayne搭建

阅读量：6241 次

发布时间：2019-06-22

本文共 11447 字，大约阅读时间需要 38 分钟。

首先部署 MySQL 服务，如果你系统中有一个可访问的 MySQL 服务的话就可以跳过这一步，我们这里在 Kubernetes 集群中部署一个简单的 MySQL 服务，对应的资源清单文件如下：(db.yaml)

[root@kubemaster Wayne]# cat db.yaml

apiVersion: apps/v1beta1

kind: Deployment

metadata:

namespace: kube-system

labels:

app: mysql

spec:

template:

metadata:

labels:

app: mysql

spec:

containers:

- name: mysql

image: mysql:5.7.14

imagePullPolicy: IfNotPresent

ports:

- containerPort: 3306

env:

- name: MYSQL_ROOT_PASSWORD

value: rootPassw0rd

volumeMounts:

- name: db

mountPath: /var/lib/mysql

volumes:

- name: db

emptyDir: {}

---

apiVersion: v1

kind: Service

metadata:

namespace: kube-system

spec:

selector:

app: mysql

ports:

- name: mysqlport

protocol: TCP

port: 3306

targetPort: dbport

kubectl create -f db.yaml

kubectl get pods -n kube-system

[root@kubemaster Wayne]# kubectl get pods -n kube-system

NAME READY STATUS RESTARTS AGE

coredns-86c58d9df4-5ztlc 1/1 Running 0 2d2h

coredns-86c58d9df4-dzwz9 1/1 Running 0 2d2h

etcd-kubemaster 1/1 Running 1 2d2h

kube-apiserver-kubemaster 1/1 Running 0 2d2h

kube-controller-manager-kubemaster 1/1 Running 0 2d2h

kube-flannel-ds-amd64-7ghpg 1/1 Running 0 41h

kube-flannel-ds-amd64-j2d65 1/1 Running 0 41h

kube-flannel-ds-amd64-jdvvq 1/1 Running 0 2d2h

kube-proxy-2lfnm 1/1 Running 0 41h

kube-proxy-9xqnh 1/1 Running 0 2d2h

kube-proxy-zbgxc 1/1 Running 0 41h

kube-scheduler-kubemaster 1/1 Running 0 2d2h

mysql-7c8bc9c996-29dcf 0/1 ContainerCreating 0 2m32s

[root@kubemaster Wayne]#

查看详细的信息，发现还在pull镜像

[root@kubemaster Wayne]# kubectl describe pods -n kube-system mysql-7c8bc9c996-29dcf

Name: mysql-7c8bc9c996-29dcf

Namespace: kube-system

Priority: 0

PriorityClassName: <none>

Node: kubenode2/10.83.32.133

Start Time: Tue, 26 Feb 2019 15:10:09 +0800

Labels: app=mysql

pod-template-hash=7c8bc9c996

Annotations: <none>

Status: Pending

IP:

Controlled By: ReplicaSet/mysql-7c8bc9c996

Containers:

mysql:

Container ID:

Image: mysql:5.7.14

Image ID:

Port: 3306/TCP

Host Port: 0/TCP

State: Waiting

Reason: ContainerCreating

Ready: False

Restart Count: 0

Environment:

MYSQL_ROOT_PASSWORD: rootPassw0rd

Mounts:

/var/lib/mysql from db (rw)

/var/run/secrets/kubernetes.io/serviceaccount from default-token-x5lmr (ro)

Conditions:

Type Status

Initialized True

Ready False

ContainersReady False

PodScheduled True

Volumes:

db:

Type: EmptyDir (a temporary directory that shares a pod's lifetime)

Medium:

default-token-x5lmr:

Type: Secret (a volume populated by a Secret)

SecretName: default-token-x5lmr

Optional: false

QoS Class: BestEffort

Node-Selectors: <none>

Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s

node.kubernetes.io/unreachable:NoExecute for 300s

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal Scheduled 2m50s default-scheduler Successfully assigned kube-system/mysql-7c8bc9c996-29dcf to kubenode2

Normal Pulling 2m47s kubelet, kubenode2 pulling image "mysql:5.7.14"

[root@kubemaster Wayne]#

[root@kubemaster Wayne]# kubectl get pods -n kube-system #再次查看pod，发现mysql的pod已经运行起来了

NAME READY STATUS RESTARTS AGE

coredns-86c58d9df4-5ztlc 1/1 Running 0 2d2h

coredns-86c58d9df4-dzwz9 1/1 Running 0 2d2h

etcd-kubemaster 1/1 Running 1 2d2h

kube-apiserver-kubemaster 1/1 Running 0 2d2h

kube-controller-manager-kubemaster 1/1 Running 0 2d2h

kube-flannel-ds-amd64-7ghpg 1/1 Running 0 41h

kube-flannel-ds-amd64-j2d65 1/1 Running 0 41h

kube-flannel-ds-amd64-jdvvq 1/1 Running 0 2d2h

kube-proxy-2lfnm 1/1 Running 0 41h

kube-proxy-9xqnh 1/1 Running 0 2d2h

kube-proxy-zbgxc 1/1 Running 0 41h

kube-scheduler-kubemaster 1/1 Running 0 2d2h

mysql-7c8bc9c996-29dcf 1/1 Running 0 4m18s

[root@kubemaster Wayne]#

测试一下mysql数据库是否可以正常使用

[root@kubemaster Wayne]# kubectl get svc -n kube-system

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 2d2h

mysql ClusterIP 10.107.79.214 <none> 3306/TCP 5m38s

[root@kubemaster Wayne]#

[root@kubemaster Wayne]# kubectl run client2 --image=busybox -it --restart=Never

If you don't see a command prompt, try pressing enter.

/ # mysql

sh: mysql: not found

/ # ping mysql

ping: bad address 'mysql'

/ # ping mysql.kube-system

PING mysql.kube-system (10.107.79.214): 56 data bytes

--- mysql.kube-system ping statistics ---

4 packets transmitted, 0 packets received, 100% packet loss

/ #

运行一个busybox的临时容器，ping一下mysql服务，由于mysql服务是在kube-system命名空间，所以ping mysql.kube-system 服务名.命名空间的方式

git clone 下载wayne的yaml文件配置

cd wayne/hack/kubernetes/wayne

sed -i 's#namespace: default#namespace: kube-system#g' `grep -rl 'namespace' ./` 将命名空间全部修改成 kube-system

由于我们这里是使用上面集群中部署的 MySQL 服务，所以这里需要对 configmap.yaml 文件进行简单的配置，而 360 文档上面的 ConfigMap 是不完整的，需要使用源码里面的 app.conf文件来进行创建，所以我们这里可以使用 --from-file关键字来创建 ConfigMap 对象，首先配置下 app.conf文件

vim /data/Wayne/wayne/src/backend/conf/app.conf

appname = wayne

httpport = 8080

runmode = prod

autorender = false

copyrequestbody = true

EnableDocs = true

EnableAdmin = true

StaticDir = public:static

# Custom config

ShowSql = false

## if enable username and password login

EnableDBLogin = true

# token, generate jwt token

RsaPrivateKey = "./apikey/rsa-private.pem"

RsaPublicKey = "./apikey/rsa-public.pem"

# token end time. second

TokenLifeTime=86400

# kubernetes labels config

AppLabelKey= wayne-app

NamespaceLabelKey = wayne-ns

PodAnnotationControllerKindLabelKey = wayne.cloud/controller-kind

# database configuration:

## mysql

DBName = "wayne"

DBTns = "tcp(mysql:3306)"

DBUser = "root"

DBPasswd = "rootPassw0rd"

DBLoc = "Asia%2FShanghai"

DBConnTTL = 30

# web shell auth

appKey = "860af247a91a19b2368d6425797921c6"

# Set demo namespace and group id

DemoGroupId = "1"

DemoNamespaceId = "1"

# Sentry

LogLevel = "7"

SentryEnable = false

# SentryDSN = ""

# SentryLogLevel = "4"

# Robin

EnableRobin = false

# api-keys

EnableApiKeys = false

# Bus

BusEnable = false

# BusRabbitMQURL = "amqp://guest:guest@rabbitmq:5672"

# Webhook

# EnableWebhook = true

# WebhookClientTimeout = 10

# WebhookClientWindowSize = 16

# other

# Use Canary/Production Update

# If set app metaData {"mode":"beta"}，the app will auto redirect to BetaUrl

# BetaUrl = ""

# AppUrl = ""

# oauth2

[auth.oauth2]

# redirect_url = "https://www.wayne.cloud"

enabled = false

client_id = client

client_secret = secret

auth_url = https://example.com/oauth2/v1/authorize

token_url = https://example.com/oauth2/v1/token

api_url = https://example.com/oauth2/v1/userinfo

# If your OAuth 2.0-based authorization service does not have email, name, and dispaly fields, use mapping criteria.

# api_mapping = name:name,email:email,display:display

# ldap config

# enable ldap login

[auth.ldap]

enabled = false

ldap_url = ldap://127.0.0.1

ldap_search_dn = "cn=admin,dc=example,dc=com"

ldap_search_password = admin

ldap_base_dn = "dc=example,dc=com"

ldap_filter =

ldap_uid = cn

ldap_scope = 2

ldap_connection_timeout = 30

创建一个配置文件configmap，通过app.conf文件来创建

[root@kubemaster src]# kubectl create configmap infra-wayne --namespace kube-system --from-file=/data/Wayne/wayne/src/backend/conf/app.conf

configmap/infra-wayne created

[root@kubemaster src]#

[root@kubemaster wayne]# kubectl create -f deployment.yaml 创建infra-wayne deployment

deployment.extensions/infra-wayne created

deployment.extensions/infra-wayne-woker created

deployment.extensions/infra-wayne-webhook created

[root@kubemaster wayne]# kubectl create -f service.yaml

创建infra-wayne service

service/infra-wayne created

[root@kubemaster wayne]# kubectl get pods -n kube-system

NAME READY STATUS RESTARTS AGE

coredns-86c58d9df4-5ztlc 1/1 Running 0 2d6h

coredns-86c58d9df4-dzwz9 1/1 Running 0 2d6h

etcd-kubemaster 1/1 Running 1 2d6h

infra-wayne-7ddd7f4b9c-dqqng 0/1 ContainerCreating 0 12s

infra-wayne-webhook-7f56c69675-xcjqd 0/1 ContainerCreating 0 12s

infra-wayne-woker-57685f749d-7mbb4 0/1 ContainerCreating 0 12s

kube-apiserver-kubemaster 1/1 Running 0 2d6h

kube-controller-manager-kubemaster 1/1 Running 1 2d6h

kube-flannel-ds-amd64-7ghpg 1/1 Running 0 45h

kube-flannel-ds-amd64-j2d65 1/1 Running 0 45h

kube-flannel-ds-amd64-jdvvq 1/1 Running 0 2d6h

kube-proxy-2lfnm 1/1 Running 0 45h

kube-proxy-9xqnh 1/1 Running 0 2d6h

kube-proxy-zbgxc 1/1 Running 0 45h

kube-scheduler-kubemaster 1/1 Running 1 2d6h

mysql-7c8bc9c996-29dcf 1/1 Running 0 4h18m

rabbitmq-wayne-79cdcbcf4-prs54 0/1 ContainerCreating 0 2m48s

[root@kubemaster wayne]#

这个时候发现

kubectl get pods -n kube-system

infra-wayne-webhook-7f56c69675-q2dpt 0/1 CrashLoopBackOff 3 73s

通过日志查看具体的原因，原来是Bus总线没有开启

[root@kubemaster wayne]# kubectl logs -f -n kube-system infra-wayne-webhook-7f56c69675-q2dpt

panic: Running workers requires BUS FEATURE enabled.

goroutine 1 [running]:

github.com/Qihoo360/wayne/src/backend/cmd/worker.run(0x1fd2b40, 0xc4208bafc0, 0x0, 0x4)

/go/src/github.com/Qihoo360/wayne/src/backend/cmd/worker/worker.go:60 +0x1e5

github.com/Qihoo360/wayne/src/vendor/github.com/spf13/cobra.(*Command).execute(0x1fd2b40, 0xc4208baf80, 0x4, 0x4, 0x1fd2b40, 0xc4208baf80)

/go/src/github.com/Qihoo360/wayne/src/vendor/github.com/spf13/cobra/command.go:766 +0x2c1

github.com/Qihoo360/wayne/src/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x1fd2420, 0x0, 0x1fd2680, 0x1fd2b40)

/go/src/github.com/Qihoo360/wayne/src/vendor/github.com/spf13/cobra/command.go:852 +0x30a

github.com/Qihoo360/wayne/src/vendor/github.com/spf13/cobra.(*Command).Execute(0x1fd2420, 0xc4201bbf78, 0xc420096058)

/go/src/github.com/Qihoo360/wayne/src/vendor/github.com/spf13/cobra/command.go:800 +0x2b

main.main()

/go/src/github.com/Qihoo360/wayne/src/backend/main.go:12 +0x50

[root@kubemaster wayne]# l

如果需要启动 infra-wayne-webhook 和 infra-wayne-woker 还需要开启rabbitMQ

infra-wayne-webhook-7f56c69675-vrl96 1/1 Running 0 16s

infra-wayne-woker-57685f749d-5d4p4 1/1 Running 0 16s

[root@kubemaster wayne]#

cat /data/Wayne/wayne/hack/kubernetes/dependency/rabbitmq.yaml

kind: Deployment

apiVersion: extensions/v1beta1

metadata:

namespace: kube-system

labels:

app: rabbitmq-wayne

spec:

replicas: 1

selector:

matchLabels:

app: rabbitmq-wayne

template:

metadata:

labels:

app: rabbitmq-wayne

spec:

containers:

- name: rabbitmq

image: 'rabbitmq:3.7.8-management'

resources:

limits:

cpu: '1'

memory: 1Gi

requests:

cpu: '1'

memory: 1Gi

---

apiVersion: v1

kind: Service

metadata:

labels:

app: rabbitmq-wayne

namespace: kube-system

spec:

ports:

- port: 5672

protocol: TCP

targetPort: 5672

selector:

app: rabbitmq-wayne

[root@kubemaster wayne]#

注意这里的rabbitMQ service的名字

然后修改

# Bus

BusEnable = true 需要把这个总线参数设置为true开启

BusRabbitMQURL = "amqp://guest:guest@rabbitmq-wayne:5672"

这里的mq服务器名称为rabbitmq service的名字

# Webhook 同时开启下面的webhook的三个参数，主要用于审计功能

EnableWebhook = true

WebhookClientTimeout = 10

WebhookClientWindowSize = 16

查看svc的地址，通过访问宿主机kubenode1的32308端口就可以访问infra-wayne

[root@kubemaster wayne]# kubectl get svc -n kube-system

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

infra-wayne NodePort 10.100.135.243 <none> 8080:32308/TCP 17m

kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 2d6h

mysql ClusterIP 10.107.79.214 <none> 3306/TCP 4h35m

rabbitmq-wayne ClusterIP 10.110.230.125 <none> 5672/TCP 19m

[root@kubemaster wayne]#

默认的用户名和密码都是 admin admin

[root@kubemaster wayne]# kubectl cluster-info

Kubernetes master is running at https://10.83.32.146:6443

KubeDNS is running at https://10.83.32.146:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

[root@kubemaster wayne]#

实例

推荐关注我的个人微信公众号 “云时代IT运维”,周期性更新最新的应用运维类技术文档。关注虚拟化和容器技术、CI/CD、自动化运维等最新前沿运维技术和趋势；