使用Filebeat和Logstash集中归档游戏日志

发表于 2018-01-13 | 阅读次数

阅读时长 7

背景说明

由于游戏项目日志目前不够规范,不太容易根据字段结构化数据,开发又有实时查看生产和测试环境服务运行日志需求;如果写入ES通过Kibana查看,对于非分析类查看还是不太友好,当然也可以通过LogTrail插件

方案

Filebeat->Logstash->Files
Filebeat->Redis->Logstash->Files
Nxlog(Rsyslog、Logstash)->Kafka->Flink(Logstash->ES-Kibana)
其他方案(可根据自己需求，选择合适的架构,作者选择了第二种方案)

注释: 由于Logstash无法处理输出到文件乱序的问题，可通过不同的文件使用不同的Logstash,单线程归档；或者直接写入ES(基于@timestamp)、通过Flink输出到文件

部署

系统环境

Debian8 x64
logstash-6.1.1
filebeat-6.1.1-amd64
Redis-3.2

Filebeat配置

/etc/filebeat/filebeat.yml
filebeat.prospectors:
- type: log
  paths:
    - /home/data/log/*
    - /home/data/*.log
  scan_frequency: 20s
  encoding: utf-8
  tail_files: true
  harvester_buffer_size: 5485760
fields:
  ip_address: 192.168.2.2
  env: qa
output.redis:
  hosts: ["192.168.1.1:6379"]
  password: "geekwolf"
  key: "filebeat"
  db: 0
  timeout: 5
  max_retires: 3
  worker: 2
  bulk_max_size: 4096

Logstash配置

input {
 #Filebeat
 # beats {
 #   port => 5044
 # }
 #Redis
  redis {
    batch_count => 4096
    data_type => "list"
    key => "filebeat"
    host => "127.0.0.1"
    port => 5044
    password => "geekwolf"
    db => 0
    threads => 2
   }
}
filter {
  ruby {
      code => 'event.set("filename",event.get("source").split("/")[-1])'
  }
}
output {
  if [filename] =~ "nohup" {
    file {
        path => "/data/logs/%{[fields][env]}/%{+YYYY-MM-dd}/%{[fields][ip_address]}/%{filename}"
        flush_interval => 3
        codec => line { format => "%{message}"}
    }
  } else {
    file {
         path => "/data/logs/%{[fields][env]}/%{+YYYY-MM-dd}/%{[fields][ip_address]}/logs/%{filename}"
        flush_interval => 3
        codec => line { format => "%{message}"}
   }
 }
 #stdout { codec => rubydebug }
}

生产日志目录

.
├── prod
│   └── 2018-01-13
│       └── 2.2.2.2
│           ├── logs
│           │   ├── rpg_slow_db_.27075
│           └── nohup_service.log
└── qa
    ├── 2018-01-12
    │   ├── 192.168.3.1
    └── 2018-01-13
        ├── 192.168.3.2

总结

笔者在测试Logstash单线程输出时，依然产生乱序问题(有知晓的可以留言),最终选择通过自己开发的daemon程序实现,参考Plogstash:

# -*- coding: utf-8 -*-
# @Author: Geekwolf
# @Date:   2018-01-29 14:23:04
# @Last Modified by:   Geekwolf
# @Last Modified time: 2018-01-31 10:55:01
#!/usr/bin/env python3
# daemon.py
import os
import sys
import time
import redis
import json
import re
import atexit
import signal
# import collections
class Base(object):
    def __init__(self, *args, **kwargs):
        self.pidfile = '/var/run/plogstash.pid'
        self.service_name = 'Plogstash'
        self.path = '/var/log/plogstash'
        os.makedirs(self.path, exist_ok=True)
        self.logfile = '%s/%s.log' % (self.path, self.service_name)
        self.redis_host = '127.0.0.1'
        self.redis_password = 'geekwolf'
        self.redis_port = 5044
        self.redis_db = 0
        self.redis_key = 'filebeat'
        self.batch_size = 5000
        self.expires = 5  # second
        self.archive_time = 1  # how long time to archive
        self.base_dir = '/data/logs'
        # self._tmp = '/tmp/.%s' % self.service_name
class Daemon(Base):
    def __init__(self, *args, **kwargs):
        super(Daemon, self).__init__(*args, **kwargs)
    def daemonize(self):
        # First fork (detaches from parent)
        try:
            if os.fork() > 0:
                raise SystemExit(0)   # Parent exit
        except OSError as e:
            raise RuntimeError('fork #1 failed.')
        os.chdir('/')
        # set this will 777
        # os.umask(0)
        os.setsid()
        # Second fork (relinquish session leadership)
        try:
            if os.fork() > 0:
                raise SystemExit(0)
        except OSError as e:
            raise RuntimeError('fork #2 failed.')
        # Flush I/O buffers
        sys.stdout.flush()
        sys.stderr.flush()
        # Replace file descriptors for stdin, stdout, and stderr
        with open(self.logfile, 'ab', 0) as f:
            os.dup2(f.fileno(), sys.stdout.fileno())
        with open(self.logfile, 'ab', 0) as f:
            os.dup2(f.fileno(), sys.stderr.fileno())
        with open(self.logfile, 'rb', 0) as f:
            os.dup2(f.fileno(), sys.stdin.fileno())
        # Write the PID file
        print(os.getpid())
        with open(self.pidfile, 'w') as f:
            print(os.getpid(), file=f)
        # Arrange to have the PID file removed on exit/signal
        atexit.register(lambda: os.remove(self.pidfile))
        # Signal handler for termination (required)
        def sigterm_handler(signo, frame):
            raise SystemExit(1)
        signal.signal(signal.SIGTERM, sigterm_handler)
    def get_now_date(self):
        return time.strftime('%Y-%m-%d', time.localtime(time.time()))
    def get_now_timestamp(self):
        return time.time()
    def get_now_time(self):
        return time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
    def logging(self, msg):
        with open(self.logfile) as f:
            print('%s  %s' % (self.get_now_time(), msg))
    def append_log(self):
        pass
    def start(self):
        if os.path.exists(self.pidfile):
            raise RuntimeError('Already running')
        else:
            try:
                self.daemonize()
                self.append_log()
                self.status()
            except RuntimeError as e:
                print(e, file=sys.stderr)
                raise SystemExit(1)
    def stop(self):
        # f = os.open(self.pipe_path, os.O_RDONLY | os.O_NONBLOCK)
        # ret = os.read(f, 1024).decode('utf-8')
        # print(ret.split('\n'))
        # os.close(f)
        if os.path.exists(self.pidfile):
            # with open(self._tmp) as f:
            #     _data = f.read()
            #     if _data is not None and len(eval(_data)) > 0:
            #         for k, v in eval(_data).items():
            #             v = v['fd'].rstrip('\n')
            #             v.close()
            with open(self.pidfile) as f:
                os.kill(int(f.read()), signal.SIGTERM)
            print('Plogstash is stopped')
        else:
            print('Not running', file=sys.stderr)
            raise SystemExit(1)
    def restart(self):
        self.stop()
        self.start()
    def status(self):
        try:
            with open(self.pidfile, 'r') as f:
                pid = int(f.read().strip())
        except:
            pid = None
        if pid:
            print('%s is running as pid:%s' % (self.service_name, pid))
        else:
            print('%s is not running' % self.service_name)
class Worker(Daemon):
    def __init__(self, *args, **kwargs):
        super(Worker, self).__init__(self, *args, **kwargs)
    def _redis(self):
        pool = redis.ConnectionPool(host=self.redis_host, password=self.redis_password, port=self.redis_port, db=self.redis_db, socket_timeout=10000)
        rc = redis.StrictRedis(connection_pool=pool)
        return rc
    def get_redis_data(self):
        _data = self._redis().lrange(self.redis_key, 0, self.batch_size - 1)
        # 删除数据(可考虑处理完再删除)
        return _data
    def del_redis_data(self):
        _data = self._redis().ltrim(self.redis_key, self.batch_size, -1)
    def append_log(self):
        file_meta = {}
        # file_handler = collections.defaultdict(dict)
        # try:
        #     os.mkfifo(self.pipe_path)
        # except Exception as e:
        #     print(str(e))
        # pipe_ins = os.open(self.pipe_path, os.O_SYNC | os.O_CREAT | os.O_RDWR)
        while True:
            time.sleep(self.archive_time)
            _data = self.get_redis_data()
            if _data:
                for _d in _data:
                    try:
                        _d = json.loads(_d.decode('utf-8'))
                        _path = '%s/%s/%s/%s' % (self.base_dir, _d['fields']['env'], self.get_now_date(), _d['fields']['ip_address'])
                        os.makedirs(_path + '/logs', exist_ok=True)
                        file_name = _d['source'].split('/')[-1]
                       # _path = '%s/%s/%s/%s' % (self.base_dir, _d['fields']['env'],self.get_now_date(), _d['fields']['ip_address'])
                        if re.match('nohup', file_name):
                            file_path = '%s/%s' % (_path, file_name)
                        else:
                            file_path = '%s/logs/%s' % (_path, file_name)
                        with open(file_path, 'a') as f:
                            f.write(_d['message'] + '\n')
                        # if 'fd' not in file_handler[file_path]:
                        #     f = open(file_path, 'a', buffering=1024000)
                        #     file_handler[file_path]['fd'] = str(f)
                        # file_handler[file_path]['time'] = self.get_now_timestamp()
                    except Exception as e:
                        self.logging(str(e))
                self.del_redis_data()
            # with open(self._tmp, 'w') as f:
            #     f.write(json.dumps(file_handler))
if __name__ == '__main__':
    if len(sys.argv) != 2:
        print('Usage: {} [start|stop|restart|status]'.format(sys.argv[0]), file=sys.stderr)
        raise SystemExit(1)
    daemon = Worker()
    if sys.argv[1] == 'start':
        daemon.start()
    elif sys.argv[1] == 'stop':
        daemon.stop()
    elif sys.argv[1] == 'restart':
        print("Restart ...")
        daemon.restart()
    elif sys.argv[1] == 'status':
        daemon.status()
    else:
        print('Unknown command {!r}'.format(sys.argv[1]), file=sys.stderr)
        raise SystemExit(1)

Zabbix3.4.5新特性：历史数据支持Elasticsearch

发表于 2018-01-09 | 分类于系统监控 | 阅读次数

阅读时长 2

特性功能

Zabbix自3.4.5rc1版本开始支持Elasticsearch作为历史数据存储，17年12月28日发布了3.4.5

部署Elasticsearch

安装Elasticsearch和Kibana:

echo "deb http://http.debian.net/debian jessie-backports main" >>/etc/apt/source.list
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" > /etc/apt/sources.list.d/elastic-6.x.list
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
apt-get -y update
apt-get install -t jessie-backports openjdk-8-jdk
update-java-alternatives -s java-1.8.0-openjdk-amd64
apt-get install -y elasticsearch kibana

配置Elasticsearch和Kibana(两者在同一台机):

vim /etc/elasticsearch/elasticsearch.yml
network.host: 0.0.0.0
vim  /etc/kibana/kibana.yml 
server.host: "0.0.0.0"
elasticsearch.url: "http://localhost:9200"

启动Elasticsearch和kibana服务

1 2	/etc/init.d/elasticsearch start /etc/init.d/kibana start

Zabbix3.4.0升级至3.4.5

注: 由于Zabbix3.4.5对libcurl库要求在7.20.0或者更高，Debian 8下面默认是7.38.0

/etc/init.d/zabbix_server stop
tar xf zabbix-3.4.5.tar.gz
cd zabbix-3.4.5
./configure --enable-server --enable-agent --with-mysql --enable-ipv6 --with-net-snmp --with-libcurl --with-libxml2
make -j8
make install

Zabbix_server配置支持Elasticsearch

1
2
3

vim /usr/local/etc/zabbix_server.conf
HistoryStorageURL=http://192.168.100.100:9200
HistoryStorageTypes=uint,dbl,str,log,text

说明:

Elasticsearch支持的监控项类型:uint,dbl,str,log,text
监控项数据类型|数据库表|对应Elasticsearch类型:

Zabbix Web配置历史数据读Elasticsearch

修改配置文件vim conf/zabbix.conf.php
1. 如果不同类型使用不同的ES集群，可以按如下进行配置
$HISTORY['url']   = [
    'uint' => 'http://localhost1:9200',
    'text' => 'http://localhost2:9200'
];
$HISTORY['types'] = ['uint', 'text'];
2. 所有类型使用相同ES集群
$HISTORY['url']   = 'http://192.168.100.100:9200';
$HISTORY['types'] = ['str', 'text', 'log', 'uint', 'dbl'];

注: 3.4.0升级到3.4.5后，请勿使用旧的zabbix.conf.php，根据新的zabbix.conf.php.example重新配置

重启Zabbix Server

1 2	/etc/init.d/zabbix_server start 此时可以通过观察日志，查看是否连接ES成功

测试

Zabbix配置ES成功后,通过Kibana可以看到：
创建索引
通过Zabbix Web访问是否正常显示数据

Django Channels实现Zabbix实时告警到页面

发表于 2018-01-09 | 阅读次数

阅读时长 10

什么是WebSocket

websocket是HTML5开始提供的一种新协议,用于浏览器和服务器之间实现全双工通讯的技术。本质上是基于tcp协议,先通过HTTP/HTTPS协议发起一条特殊的http请求进行握手后,创建一个用于双向数据交换的tcp连接,此后服务端与客户端通过此连接进行实时通信。在websocket之前实现全双工通讯一般使用轮训、SSE(Server-Sent Event,服务端推送事件)、Comet技术

HTTP与WebSocket的区别

由上面的示意图可知,在传统的http1.0,request和response是一对一的，每次都要发送header信息
http1.1 默认开启了keeplive也只是复用同一个tcp连接，但是服务器和客户端还要大量交换HTTP header，信息交换效率很低。
WebSocket是一种双向通信协议。在建立连接后，WebSocket服务器端和客户端都能主动向对方发送或接收数据，就像Socket一样。从而更好的节省服务器资源和带宽并达到实时通讯的目的
WebSocket需要像TCP一样，先建立连接，连接成功后才能相互通信

客户端通过WebSocket与服务端建立通信过程

在客户端，new WebSocket实例化一个新的WebSocket客户端对象，请求类似 ws://yourdomain:port/path 的服务端WebSocket URL，客户端WebSocket对象会自动解析并识别为WebSocket请求，并连接服务端端口，执行双方握手过程，客户端发送数据格式类似：
1
2
3
4
5
6
7
GET /ws/alert/ HTTP/1.1
Host: 127.0.0.1:8000
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: xqBt3ImNzJbYqRINxEFlkg==
Origin: http://127.0.0.1:8000
Sec-WebSocket-Version: 13
可以看到，客户端发起的WebSocket连接报文类似传统HTTP报文，Upgrade：websocket参数值表明这是WebSocket类型请求，Sec-WebSocket-Key是WebSocket客户端发送的一个 base64编码的密文，要求服务端必须返回一个对应加密的Sec-WebSocket-Accept应答，否则客户端会抛出Error during WebSocket handshake错误，并关闭连接。服务端收到报文后返回的数据格式类似：
1
2
3
4
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: K7DJLdLooIwIG/MOpvWFB3y3FE8=

Sec-WebSocket-Accept的值是服务端采用与客户端一致的密钥计算出来后返回客户端的，HTTP/1.1 101 Switching Protocols表示服务端接受WebSocket协议的客户端连接，经过这样的请求-响应处理后，两端的WebSocket连接握手成功, 后续就可以进行TCP通讯了

注释: WebSocket标识符是ws(如果加密,则是wss),如上图所示

WebSocket服务

Node(按热度排序): Socket.IO、uWebSockets、WebSocket-Node
Go: websocketd、websocket
Django: Channel

Django Channel

WSGI/ASGI

WSGI
大家都知道WSGI,即Web Server Gateway Interface,是服务器和客户端交互的接口规范，符合这种借口的application可以在所有符合该接口的server上运行，解耦了server和application;web组件被分成三类：client、server、middleware

如上图所示

Server/Gateway：处理HTTP协议，接受用户HTTP请求，调用application处理逻辑，将response返回给client;比如Apache、Nginx
Application：专注业务逻辑的python 应用或者框架，如Django；根据WSGI协议规范，Applicaiton需要定义http://wsgi.tutorial.codepoint.net/application-interface
Middleware：位于Server/Gateway 和 Application/Framework 之间，对 Server/Gateway 来说，它相当于 Application/Framework ；对 Application/Framework 来说，它相当于 Server/Gateway。每个 middleware 实现不同的功能，我们通常根据需求选择相应的 middleware 并组合起来，实现所需的功能。比如，可在 middleware 中实现以下功能：
1. 根据 url 把用户请求调度到不同的 application 中
2. 负载均衡，转发用户请求
3. 限制请求速率，设置白名单
  WSGI的middleware体现 unix 的哲学之一：do one thing and do it well

ASGI
由于WSGI协议支持HTTP,ASGI(Asynchronous Server Gateway Interface)在此基础上应运而生，对WSGI协议进行兼容和扩展，能够处理多种通用协议如HTTP、HTTP2、WebSocket，允许这些协议能通过网络或本地socket进行传输，以及让不同的协议被分配到不同的进程中

ASGI由三个不同的组件组成:协议服务、频道层(Channnel Layer)、应用层；其中Channel Layer是最重要的部分，同时对协议服务和应用提供接口；

频道和消息： ASGI规定所有通信都要通过在频道里发送消息进行，消息是一个dict,为了保证可序列化，只允许以下类型数据
string/Unicode/int(非long)/list/dict(Key是Unicode)/boolean/None
频道是一个先进先出队列，队列中的消息最多发送给一个消费者；频道中的消息超过设定时间会被清理，消息大小最大限定为1MB，超过需要分块
群组: 频道中消息只能被传送一次，不能广播；如果向任一组用户发送消息，就要用到群组

Channels

大概了解ASGI规范之后，看下django基于ASGI协议实现HTTP/HTTP2/WebSocket的模块Channels,安装好channels后，django将有原来的request-response模式，转换成worker工作模式；并没有运行单独的wsgi进程，而是分成了三层:

interface Server: 负责Django和Client通信，同时适配WSGI和WebSocket Server
Channel Layer: 可插拔的Python代码和数据存储，如Redis、或者内存，用于消息的传输
Workers: 监听频道，消息抵达时运行消费者代码

下面用例子来看下如何使用Channels: 实现Zabbix报警实时传送到客户端
描述:

Trigger触发时，根据Action设置通过脚本报警，并将报警信息发布到Redis的ALARM频道
Django Commands alert 订阅Redis的ALARM频道
调用channels的send方法，通过websocket实时推送到Client

目录结构:

monitor
├── monitor
│   ├── celery.py
│   ├── consumers.py
│   ├── __init__.py
│   ├── routing.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── commands
│   ├── admin.py
│   ├── apps.py
│   ├── __init__.py
│   ├── management
│   │   ├── commands
│   │   │   ├── alert.py
│   │   │   ├── __init__.py
│   │   │   └── __pycache__
│   │   ├── __init__.py
│   │   └── __pycache__
│   ├── migrations
│   │   ├── __init__.py
│   │   └── __pycache__
│   ├── models.py
│   ├── __pycache__
│   ├── tests.py
│   └── views.py
├── static
├── templates
│   ├── base
|        |—— base.html
├── README.md
├── requirements.txt

安装配置Channels

pip install channels asgi_redis
settings.py添加app和设置CHANNEL_LAYERS
#commands是后面定义Django命令的app
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'channels',
    'commands'
]
#Redis信息
REDIS_OPTIONS = {
    'HOST': '127.0.0.1',
    'PORT': 6379,
    'PASSWD': 'geekwolf',
    'DB': 0
}
#可以使用内存存储Channels消息
#CHANNEL_LAYERS = {
#    "default": {
#        "BACKEND": "asgiref.inmemory.ChannelLayer",
#        "ROUTING": "channels_example.routing.channel_routing",
#    },
#}
#使用Redis作为消息存储，需安装asgi_redis
CHANNEL_LAYERS = {
    'default': {
        'BACKEND': 'asgi_redis.RedisChannelLayer',
        'CONFIG': {
            'hosts': ['redis://:{0}@{1}:{2}/{3}'.format(REDIS_OPTIONS['PASSWD'], REDIS_OPTIONS['HOST'], REDIS_OPTIONS['PORT'], 1)]
        },
        'ROUTING': 'plonvol.routing.channel_routing'
    }
}
#Redis频道和Channels群组名
GROUP_NAME = 'alarm'

添加路由(routing.py)

# -*- coding: utf-8 -*-
from channels.routing import route
from .consumers import ws_connect, ws_disconnect, ws_receive
channel_routing = [
    route('websocket.connect', ws_connect),
    route('websocket.disconnect', ws_disconnect),
]

添加consumers文件(类似view)(consumers.py)

# -*- coding: utf-8 -*-
from channels import Group
from channels.handler import AsgiHandler
from django.conf import settings
def ws_connect(message):
    message.reply_channel.send({'accept': True})
    Group(settings.GROUP_NAME).add(message.reply_channel)
def ws_disconnect(message):
    Group(settings.GROUP_NAME).discard(message.reply_channel)

订阅Redis报警消息脚本(commands/management/commands/alert.py)

# -*- coding: utf-8 -*-
import json
import logging
from channels import Group
from django.core.management import BaseCommand
from django.conf import settings
import redis
logger = logging.getLogger(__name__)
class Command(BaseCommand):
    """
    Command to start zabbix alert worker from command line.
    """
    help = 'Subscribe the zabbix alerts channel'
    def handle(self, *args, **options):
        rc = redis.Redis(host=settings.REDIS_OPTIONS['HOST'],
                         password=settings.REDIS_OPTIONS['PASSWD'],
                         port=settings.REDIS_OPTIONS['PORT'],
                         db=settings.REDIS_OPTIONS['DB'])
        rc.delete(settings.GROUP_NAME)
        pubsub = rc.pubsub()
        pubsub.subscribe(settings.GROUP_NAME)
        for item in pubsub.listen():
            if item['type'] == 'message':
                Group(settings.GROUP_NAME).send({'text': bytes.decode(item['data'])})
                logger.debug('send a message %s ' % item)

前端页面base.html

<!-- 报警声音 -->
<audio id="notify"><source src="/static/notify.ogg" type="audio/ogg">></audio>
<script type="application/javascript">
    var ws_scheme = window.location.protocol == "https:" ? "wss" : "ws";
    var ws = new WebSocket(ws_scheme + '://' + window.location.host + window.location.pathname);
    console.log(ws);
    ws.onmessage = function (message) {
        var data = JSON.parse(message.data);
        if(data['当前状态'] == 'Problem')
        {
           var subject = '<br>故障！,服务器:' + data['告警主机'] + '发生:' + data['告警信息'] + '故障!<br>';
        }
        else{
           var subject = '<br>恢复！,服务器:' + data['告警主机'] + '发生:' + data['告警信息'] + '已经恢复!<br>';
        }
        content = new Array();
        $.each(data,function(k,v){
            content.push("<b>" + k + ':</b>   ' + v)
        })
        var data = subject + content.join("<br>")
        $('#notify')[0].play();
        notify('warning','glyphicon glyphicon-danger-sign',data);
    }
</script>

测试消息，用于发布消息到Redis

import redis
import json
rc = redis.Redis(host='127.0.0.1', password='geekwolf', port=6379, db=0)
msg = {"告警主机": "web-server-node1", "告警地址": "192.168.1133.11", "告警时间": "2017-11-11 05:05:22", "告警等级": "严重",
       "告警信息": "Web端口80监控", "问题详情": "80端口连接失败", "当前状态": "Problem", "事件ID": "12345"}
rc.publish('alarm', json.dumps(msg))

运行服务，测试

启动项目(http/websocket):
python manage.py runserver 0.0.0.0:8000
启动监听报警消息进程:
python manage.py alert

访问http://192.168.1.1:8000,运行test.py脚本

参考资料

利用Socket.IO实现消息实时推送 http://www.wukai.me/2017/08/27/push-message-with-socketio/

WebSocket相关资料 http://www.52im.net/thread-331-1-1.html

WSGI规范 http://wsgi.tutorial.codepoint.net/application-interface

Channels文档:http://channels.readthedocs.io

Chat Example： https://gearheart.io/blog/creating-a-chat-with-django-channels/

Websocket 5分钟从入门到精通 https://segmentfault.com/a/1190000012709475

运维故障管理系统FMS

发表于 2017-09-12 | 阅读次数

阅读时长 1

背景说明

考虑到日常运维中涉及到故障的记录、统计等需求，针对不同的产品线、故障类型、级别等开发了运维故障管理系统FMS

善于总结故障，刨根问底，方可预防，减少避免类似问题的出现

项目说明

1
2
3

Python: 3.6
Django: 1.11.0
项目地址: git@github.com:geekwolf/fms.git

功能说明

故障管理(类型、级别等)
项目管理(针对区分不同业务线)
用户管理(用户、用户组)
权限管理
统计Dashboard

部署配置

1	pip3 install -i https://pypi.douban.com/simple/ -r requirements.txt

MySQL配置修改settings.py:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'fms',
        'USER': 'root',
        'PASSWORD': 'xxxx',
        'HOST': '127.0.0.1',
        'PORT': '3306',
    }
}

修改故障通知邮箱settings.py:

EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_USE_TLS = False
EMAIL_HOST = 'service.simlinux.com'
EMAIL_PORT = 25
EMAIL_HOST_USER = 'admin@service.simlinux.com'
EMAIL_HOST_PASSWORD = 'xxx'
DEFAULT_FROM_EMAIL = 'geekwolf <admin@service.simlinux.com>'

初始化数据

python manage.py makemigrations
python manage.py migrate
python manage.py loaddata default_types
python manage.py loaddata default_user

1
2
3

python manage.py runserver
http://127.0.0.1:8000
admin admin

Demo

k8s部署之分布式KV存储Etcd

发表于 2017-09-08 | 分类于 docker | 阅读次数

阅读时长 13

Etcd是什么

Etcd是一个分布式、使用Raft算法维护一致性的kv存储系统，与其类似产品有Zookeeper(老牌经典)、Consul等，Etcd相对ZK，更加轻量、易运维。具体三者之间的对比可参考 https://luyiisme.github.io/2017/04/22/spring-cloud-service-discovery-products/

使用场景

和zk、consul等类似，使用场景多用于:

服务发现
消息发布与订阅
负载均衡
分布式锁
分布式队列

读写性能

压测数据参考官方：
https://coreos.com/etcd/docs/latest/op-guide/performance.html

本地集群部署

操作系统:Debian8 x64
Etcd v3.2.7

A. 安装

wget https://github.com/coreos/etcd/releases/download/v3.2.7/etcd-v3.2.7-linux-arm64.tar.gz
tar xf etcd-v3.2.7-linux-arm64.tar.gz
cd etcd-v3.2.7-linux-amd64
cp etc* /usr/local/bin/
etcd: Etcd服务端文件
etcdctl: 供用户使用的命令客户端

B. 启动服务

root@a4c8d490:/home/geekwolf# etcd
2017-09-07 15:42:23.957656 I | etcdmain: etcd Version: 3.2.7
2017-09-07 15:42:23.957699 I | etcdmain: Git SHA: bb66589
2017-09-07 15:42:23.957718 I | etcdmain: Go Version: go1.8.3
2017-09-07 15:42:23.957723 I | etcdmain: Go OS/Arch: linux/amd64
2017-09-07 15:42:23.957729 I | etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 8
2017-09-07 15:42:23.957739 W | etcdmain: no data-dir provided, using default data-dir ./default.etcd
2017-09-07 15:42:23.957764 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2017-09-07 15:42:23.957995 I | embed: listening for peers on http://localhost:2380
2017-09-07 15:42:23.958107 I | embed: listening for client requests on localhost:2379
2017-09-07 15:42:23.964607 I | etcdserver: name = default
2017-09-07 15:42:23.964633 I | etcdserver: data dir = default.etcd
2017-09-07 15:42:23.964652 I | etcdserver: member dir = default.etcd/member
2017-09-07 15:42:23.964657 I | etcdserver: heartbeat = 100ms
2017-09-07 15:42:23.964663 I | etcdserver: election = 1000ms
2017-09-07 15:42:23.964668 I | etcdserver: snapshot count = 100000
2017-09-07 15:42:23.964680 I | etcdserver: advertise client URLs = http://localhost:2379
2017-09-07 15:42:23.973007 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 14
2017-09-07 15:42:23.973041 I | raft: 8e9e05c52164694d became follower at term 2
2017-09-07 15:42:23.973065 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 2, commit: 14, applied: 0, lastindex: 14, lastterm: 2]
2017-09-07 15:42:23.984367 W | auth: simple token is not cryptographically signed
2017-09-07 15:42:23.993237 I | etcdserver: starting server... [version: 3.2.7, cluster version: to_be_decided]
2017-09-07 15:42:23.993659 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2017-09-07 15:42:23.993754 N | etcdserver/membership: set the initial cluster version to 3.2
2017-09-07 15:42:23.993796 I | etcdserver/api: enabled capabilities for version 3.2
2017-09-07 15:42:24.473288 I | raft: 8e9e05c52164694d is starting a new election at term 2
2017-09-07 15:42:24.473451 I | raft: 8e9e05c52164694d became candidate at term 3
2017-09-07 15:42:24.473519 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 3
2017-09-07 15:42:24.473568 I | raft: 8e9e05c52164694d became leader at term 3
2017-09-07 15:42:24.473605 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 3
2017-09-07 15:42:24.478746 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
2017-09-07 15:42:24.478824 I | embed: ready to serve client requests
2017-09-07 15:42:24.479116 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!

由上面的输出可知：

etcd服务之间通信端口是2380，暴露给客户端端口为2379
默认将数据存放到当前路径default.etcd/目录下
该节点的名称默认为default
集群和节点都会生成唯一的uuid
启动服务时，会根据raft算法，选举leader

C. 测试

查看api版本（默认api版本是v2）
root@a4c8d490:~/k8s/etcd-v3.2.7-linux-amd64# etcdctl  --version
etcdctl version: 3.2.7
API version: 2
使用API V3方法：
Etcd服务端和客户端添加变量 export ETCDCTL_API=3,重新启动etcd服务即可
以下操作在api v3版本:
写入key: etcdctl put foo bar
读取key: etcdctl get foo
多key范围读取: etcdctl get foo foo9(会将foo..foo8的key读取,不包括foo9)
读取过往版本key的值(Etcd键值对的修改都会增加全局修订版本号,--rev为版本号):
 etcdctl get --rev=4 foo foo9
删除key: etcdctl del foo
范围删除(foo-&gt;foo9):etcdctl del foo foo9
观察key变化:etcdctl watch foo
观察范围key变化: etcdctl watch foo foo9
从rev=2版本开始观察key变化: etcdctl watch --rev=2 foo
压缩版本5之前的修订版本(压缩后5之前的版本不可能访问): etcdctl compact 5
授予key有效期:
创建租约: 
$ etcdctl lease grant 10
lease 694d5e5b63a74f31 granted with TTL(10s)
附加key foo到租约694d5e5b63a74f31，该租约过期后，会删除附加的所有key
撤销租约(撤销后，附加改租约的所有key被删除): etcdctl lease revoke 32695410dcc0ca06
维持租约(执行后，会一直维持该租约): etcdctl lease keep-alive 32695410dcc0ca0
其他参数可参考 etcdctl --help
通过HTTP操作：
Etcd v2: https://coreos.com/etcd/docs/latest/v2/api.html
Etcd v3: https://coreos.com/etcd/docs/latest/dev-guide/api_grpc_gateway.html

多节点集群部署

静态模式部署

环境说明(三节点集群)

节点	地址	主机
etcd1	192.168.234.133	etcd1.simlinux.com
etcd2	192.168.234.134	etcd2.simlinux.com
etcd3	192.168.234.135	etcd3.simlinux.com

初始化环境

三个节点分别设置主机名:

1
2
3

hostnamectl --static  set-hostname etcd1.simlinux.com
hostnamectl --static  set-hostname etcd2.simlinux.com
hostnamectl --static  set-hostname etcd3.simlinux.com

三个节点hosts文件添加:

vim /etc/hosts
192.168.234.133 etcd1.simlinux.com 
192.168.234.134 etcd2.simlinux.com 
192.168.234.135 etcd3.simlinux.com

生成etcd证书(用于etcd间、客户端与etcd通信)

etcdctl等客户端与etcd服务通信证书

etcd服务之间通信证书

由上篇k8s部署之使用CFSSL创建证书的CA来生成

cat  etcd.json 
{
    "CN": "etcd",
    "hosts": [
        "127.0.0.1",
        "192.168.234.133",
        "192.168.234.134",
        "192.168.234.135"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CN",
            "L": "ShangHai",
            "ST": "ShangHai",
            "O": "K8s",
            "OU": "System"
        }
    ]
}
生成服务端证书,用于服务端认证客户端(hosts需要包括允许访问ETCD Cluster的IP或者FQDN):
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server-csr.json | cfssljson -bare server
server-key.pem
server.csr
server.pem
生成对等证书，用于etcd间通信(这里三个节点使用同一个证书,建议分别生成证书):
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd.json | cfssljson -bare etcd
etcd-key.pem
etcd.csr
etcd.pem
将CA和etcd证书拷贝到etcd所有节点:
cp ca.pem  etcd-key.pem etcd.pem server-key.pem server.pem /etc/etcd/ssl/

安装etcd节点(所有节点)

wget https://github.com/coreos/etcd/releases/download/v3.2.7/etcd-v3.2.7-linux-amd64.tar.gz
tar xf etcd-v3.2.7-linux-amd64.tar.gz
cd etcd-v3.2.7-linux-amd64
chmod +x etcd*
cp etcd* /bin

etcd配置

服务管理(所有节点相同):
vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/data/k8s/etcd/
EnvironmentFile=/etc/etcd/etcd.conf
ExecStart=/bin/etcd 
  --name=${NAME} 
  --cert-file=/etc/etcd/ssl/etcd.pem 
  --key-file=/etc/etcd/ssl/etcd-key.pem 
  --peer-cert-file=/etc/etcd/ssl/etcd.pem 
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem 
  --trusted-ca-file=/etc/etcd/ssl/ca.pem 
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem 
  --initial-advertise-peer-urls=${INITIAL_ADVERTISE_PEER_URLS} 
  --listen-peer-urls=${LISTEN_PEER_URLS} 
  --listen-client-urls=${LISTEN_CLIENT_URLS} 
  --advertise-client-urls=${ADVERTISE_CLIENT_URLS} 
  --initial-cluster-token=${INITIAL_CLUSTER_TOKEN} 
  --initial-cluster=${INITIAL_CLUSTER} 
  --initial-cluster-state=new 
  --data-dir=${DATA_DIR}
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

配置文件:

vim /etc/etcd/etcd.conf
#节点名称
NAME="etcd1"
#etcd数据存放目录
DATA_DIR="/data/k8s/etcd"
#etcd节点间通信监听地址
LISTEN_PEER_URLS="https://192.168.234.133:2380"
#对外提供服务的地址
LISTEN_CLIENT_URLS="https://192.168.234.133:2379,https://127.0.0.1:2379"
#通知其他etcd节点本实例地址
INITIAL_ADVERTISE_PEER_URLS="https://192.168.234.133:2380"
#初始化集群内节点地址
INITIAL_CLUSTER="etcd1=https://192.168.234.133:2380,etcd2=https://192.168.234.134:2380,etcd3=https://192.168.234.135:2380"
#初始化状态.new表示新建,已经存在的集群使用existing
INITIAL_CLUSTER_STATE="new"
#创建集群的token,每个集群唯一
INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
#告知其他集群本节点客户端监听地址
ADVERTISE_CLIENT_URLS="https://192.168.234.133:2379"
ETCDCTL_API=3
其中NAME/LISTEN_PEER_URLS/LISTEN_CLIENT_URLS/INITIAL_ADVERTISE_PEER_URLS/ADVERTISE_CLIENT_URLS替换成相应节点名称和地址

服务管理

yum -y install ntp 
systemctl start ntpd 
systemctl start etcd.service
systemctl stop etcd.service
systemctl status etcd.service(查看服务状态及日志)

测试验证

[root@etcd1 ~]# export etcd1=192.168.234.133
[root@etcd1 ~]# export etcd2=192.168.234.134
[root@etcd1 ~]# export etcd3=192.168.234.135
[root@etcd1 ~]# export ETCDCTL_API=3
[root@etcd1 ~]# export ENDPOINTS=$etcd1:2379,$etcd2:2379,$etcd3:2379
查看集群成员:
[root@etcd1 etcd]#  etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem member list
+------------------+---------+-------+------------------------------+------------------------------+
|        ID        | STATUS  | NAME  |          PEER ADDRS          |         CLIENT ADDRS         |
+------------------+---------+-------+------------------------------+------------------------------+
| 1a4a83ef243ff1c9 | started | etcd2 | https://192.168.234.134:2380 | https://192.168.234.134:2379 |
| 68243ef8797bd1ce | started | etcd1 | https://192.168.234.133:2380 | https://192.168.234.133:2379 |
| fa30209a63d949b0 | started | etcd3 | https://192.168.234.135:2380 | https://192.168.234.135:2379 |
+------------------+---------+-------+------------------------------+------------------------------+
查看集群状态:
[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem endpoint status
+----------------------+------------------+---------+---------+-----------+-----------+------------+
|       ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+----------------------+------------------+---------+---------+-----------+-----------+------------+
| 192.168.234.133:2379 | 68243ef8797bd1ce |   3.2.7 |   25 kB |     false |        10 |          9 |
| 192.168.234.134:2379 | 1a4a83ef243ff1c9 |   3.2.7 |   25 kB |     false |        10 |          9 |
| 192.168.234.135:2379 | fa30209a63d949b0 |   3.2.7 |   25 kB |      true |        10 |          9 |
+----------------------+------------------+---------+---------+-----------+-----------+------------+
[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem endpoint health
192.168.234.135:2379 is healthy: successfully committed proposal: took = 1.374345ms
192.168.234.134:2379 is healthy: successfully committed proposal: took = 2.217525ms
192.168.234.133:2379 is healthy: successfully committed proposal: took = 1.996245ms
保存快照:
[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem snapshot save my.db
Snapshot saved at my.db
查看快照状态:
[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem snapshot status my.db
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 9a496339 |        3 |          8 |      25 kB |
+----------+----------+------------+------------+
恢复数据(要先删除原来数据目录,所有节点操作)：
[root@etcd1 ~]# etcdctl  --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem snapshot restore my.db  --data-dir=/data/k8s/etcd/
2017-09-09 02:28:52.439616 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
删除节点:
[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem member remove 68243ef8797bd1ce
更新节点:
[root@etcd1 ~]# etcdctl --write-out=table --endpoints=$ENDPOINTS --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem member update 68243ef8797bd1ce https://192.168.234.133:1111(INITIAL_ADVERTISE_PEER_URLS)
添加节点(删除etcd3，添加etcd4):
export etcd4=192.168.234.136
[root@etcd1 ~]#  etcdctl --endpoints=${etcd1}:2379,${etcd2}:2379 --cacert=/etc/etcd/ssl/ca.pem  --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem  member add etcd4 --peer-urls=http://192.168.234.136:2380

Etcd:从应用场景到实现原理的全方位解读 http://www.infoq.com/cn/articles/etcd-interpretation-application-scenario-implement-principle
Eetcd集群管理 https://coreos.com/etcd/docs/latest/demo.html

k8s部署之使用CFSSL创建证书

发表于 2017-09-07 | 分类于 docker | 阅读次数

阅读时长 3

安装CFSSL

curl -s -L -o /bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
curl -s -L -o /bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
curl -s -L -o /bin/cfssl-certinfo https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x /bin/cfssl*

容器相关证书类型

client certificate：用于服务端认证客户端,例如etcdctl、etcd proxy、fleetctl、docker客户端
server certificate: 服务端使用，客户端以此验证服务端身份,例如docker服务端、kube-apiserver
peer certificate: 双向证书，用于etcd集群成员间通信

创建CA证书

生成默认CA配置

mkdir /opt/ssl
cd /opt/ssl
cfssl print-defaults config > ca-config.json
cfssl print-defaults csr > ca-csr.json

修改ca-config.json,分别配置针对三种不同证书类型的profile,其中有效期43800h为5年

{
"signing": {
    "default": {
        "expiry": "43800h"
    },
    "profiles": {
        "server": {
            "expiry": "43800h",
            "usages": [
                "signing",
                "key encipherment",
                "server auth"
            ]
        },
        "client": {
            "expiry": "43800h",
            "usages": [
                "signing",
                "key encipherment",
                "client auth"
            ]
        },
        "peer": {
            "expiry": "43800h",
            "usages": [
                "signing",
                "key encipherment",
                "server auth",
                "client auth"
            ]
        }
    }
}
}

修改ca-csr.json

{
"CN": "Self Signed Ca",
"key": {
    "algo": "rsa",
    "size": 2048
},
"names": [
    {
        "C": "CN",
        "L": "SH",
        "O": "Netease",
        "ST": "SH",            
        "OU": "OT"
    }    ]
}

生成CA证书和私钥

1 2	cfssl gencert -initca ca-csr.json \| cfssljson -bare ca - 生成ca.pem、ca.csr、ca-key.pem(CA私钥,需妥善保管)

签发Server Certificate

cfssl print-defaults csr &gt; server.json
vim server.json
{
    "CN": "Server",
    "hosts": [
        "192.168.1.1"
       ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CN",
            "L": "SH",
            "ST": "SH"
        }
    ]
}
生成服务端证书和私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server.json | cfssljson -bare server

签发Client Certificate

cfssl print-defaults csr &gt; client.json
vim client.json
{
    "CN": "Client",
    "hosts": [],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CN",
            "L": "SH",
            "ST": "SH"
        }
    ]
}
生成客户端证书和私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client.json | cfssljson -bare client

签发peer certificate

cfssl print-defaults csr &gt; member1.json
vim member1.json
{
    "CN": "member1",
    "hosts": [
        "192.168.1.1"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CN",
            "L": "SH",
            "ST": "SH"
        }
    ]
}
为节点member1生成证书和私钥:
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer member1.json | cfssljson -bare member1
针对etcd服务,每个etcd节点上按照上述方法生成相应的证书和私钥

最后校验证书

校验生成的证书是否和配置相符

1
2
3

openssl x509 -in ca.pem -text -noout
openssl x509 -in server.pem -text -noout
openssl x509 -in client.pem -text -noout

k8s集群所需证书

参考

https://coreos.com/os/docs/latest/generate-self-signed-certificates.html

Django模板无法使用perms变量问题

发表于 2017-09-06 | 分类于 Python | 阅读次数

阅读时长 2

首先,在使用Django内置权限管理系统时,settings.py文件要添加

INSTALLED_APPS添加:
'django.contrib.auth',
MIDDLEWARE添加:
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.auth.context_processors.auth',
TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(BASE_DIR, 'templates')],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.i18n',
                'django.template.context_processors.media',
                'django.template.context_processors.static',
                'django.template.context_processors.tz',
                'django.contrib.messages.context_processors.messages',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
            ],
        },
    },
]

如何在模板进行权限检查呢？
根据官网说明 https://docs.djangoproject.com/en/1.11/topics/auth/default/#permissions ,已登录用户权限保存在模板变量中,是权限模板代理django.contrib.auth.context_processors.PermWrapper的一个实例，
具体可以查看django/contrib/auth/context_processors.py源码

测试用例:

测试过程中,发现变量压根不存在,没有任何输出;好吧,只能取Debug Django的源码了

def auth(request):
    """
    Returns context variables required by apps that use Django's authentication
    system.
    If there is no 'user' attribute in the request, uses AnonymousUser (from
    django.contrib.auth).
    """
    if hasattr(request, 'user'):
        user = request.user
    else:
        from django.contrib.auth.models import AnonymousUser
        user = AnonymousUser()
    print(user, PermWrapper(user), '-----------------------')
    return {
        'user': user,
        'perms': PermWrapper(user),
    }

测试访问接口,发现有的接口有打印权限信息,有的没有，似乎恍然醒悟

可以打印权限信息的接口返回:
return render(request, 'fms/fms_add.html', {'request': request, 'form': form, 'error': error})
不能打印权限新的接口返回:
return render_to_response( 'fms/fms.html', data)

render和render_to_response区别
render是比render_to_reponse更便捷渲染模板的方法,会自动使用RequestContext,而后者需要手动添加:

1	return render_to_response(request, 'fms/fms_add.html', {'request': request, 'form': form, 'error': error},context_instance=RequestContext(request))

其中RequestContext是django.template.Context的子类.接受request和context_processors,从而将上下文填充渲染到模板
问题已经很明确，由于使用了render_to_response方法,没有手动添加context_instance=RequestContext(request)导致模板不能使用变量

企业级Docker私有仓库部署(https)

发表于 2017-08-05 | 分类于 docker | 阅读次数

阅读时长 1

部署环境

Centos7.3 x64
docker-ce-17.06.0
docker-compose-1.15.0
Python-2.7.5(系统默认)

部署目标

使用HTTPS协议
支持Clair(在Harbor1.2版本会支持)

支持HTTPS

生产环境最好由权威CA机构签发证书(免费的推荐StartSSL,可参考https://www.wosign.com/Support/Nginx.html),这里为了测试方便使用自签发的证书

创建CA证书

1	openssl req -newkey rsa:4096 -nodes -sha256 -keyout ca.key -x509 -days 365 -out ca.crt

生成CSR公钥

1	openssl req -newkey rsa:4096 -nodes -sha256 -keyout hub.wow.key -out hub.wow.csr

颁发证书

1	openssl x509 -req -days 365 -in hub.wow.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out hub.wow.crt

部署证书

cp hub.wow.crt hub.wow.key   /data/harbor/keys/
vim /data/harbor/harbor.cfg
  hostname = hub.wow
  ui_url_protocol = https
  ssl_cert = /data/harbor/keys/hub.wow.crt
  ssl_cert_key = /data/harbor/keys/hub.wow.key
    cd /data/harbor
    ./prepare  重新生成配置文件
    docker-compose down
    docker-compose up

通过HTTPS访问私有仓库

WebUI: https://how.wow
Docker Client:
[root@hub ~]# docker login -u admin -p Harbor12345 hub.wow
Login Succeeded

问题

docker login时提示x509: certificate signed by unknown authority
解决方法: 自签名的证书不被系统信任,需要cp ca.crt /etc/docker/certs.d/hub.wow/, 无需重启docker

企业级Docker私有仓库之Harbor部署(http)

发表于 2017-08-04 | 分类于 docker | 阅读次数

阅读时长 2

部署环境

Centos7.3 x64
docker-ce-17.06.0
docker-compose-1.15.0
Python-2.7.5(系统默认)

Docker及Docker-compose安装

 yum install -y yum-utils device-mapper-persistent-data lvm2
 yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo
 yum-config-manager --enable docker-ce-edge
 yum makecache fast
 systemctl start docker 
 systemctl enable docker
curl -L https://github.com/docker/compose/releases/download/1.15.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

Habor部署配置

wget https://github.com/vmware/harbor/releases/download/v1.1.2/harbor-offline-installer-v1.1.2.tgz
tar xf harbor-offline-installer-v1.1.2.tgz
cd harbor/
vim harbor.cfg
hostname = hub.wow
其他默认(http协议)
./install.sh
安装成功后，可以通过http://hub.wow/访问

Docker客户端使用

由于Harbor默认使用的http协议,故需要在Docker client上的Dockerd服务增加–insecure-registry hub.wow
Centos7修改方式为:

vim /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd  --insecure-registry hub.wow
systemctl daemon-reload
systemctl reload docker

[root@localhost harbor]# docker login -u admin -p Harbor12345 hub.wow
官方仓库下载busybox镜像
[root@localhost harbor]# docker pull busybox 
[root@localhost harbor]# docker images
REPOSITORY                  TAG                 IMAGE ID            CREATED             SIZE
busybox                     latest              efe10ee6727f        2 weeks ago         1.13MB
本地基于busybox:latest创建标记hub.wow/busybox:latest
[root@localhost harbor]# docker tag busybox:latest hub.wow/project_name/busybox:latest
推送本地镜像busybox:latest 到hub.wow私有仓库
[root@localhost harbor]# docker push hub.wow/project_name/busybox:latest

Harbor服务管理

1 2	cd harbor/ docker-compose -f ./docker-compose.yml [ up\|down\|ps\|stop\|start ]

使用Zabbix LLD实现进程数监控

发表于 2017-04-19 | 分类于系统监控 | 阅读次数

阅读时长 4

目的

针对特定进程数量做监控报警

思路

通过Zabbix LLD自动发现：每台机器都跑了什么服务、每个服务应该跑多少进程
Zabbix Agent 30s将当前机器跑了哪些服务、每个服务进程数上报Zabbix Server
开发给定配置文件proccessInfo.txt: IP 服务名称进程数量,此配置作为监控依据
proccessInfo.txt配置文件需在每次变更配置时，自动生成最新

配置流程

LLD自动发现脚本
数据采集脚本
Agent添加Key
Zabbix Server添加模板组
创建自动发现规则(监控项、报警触发器)
添加当前进程数监控项(通过Zabbix Trapper方式，由Agent端)
定义报警内容

具体步骤

LLD自动发现脚本

LLD自动发现,将进程名称及进程总数上报Zabbix Server：

/usr/bin/python services.py services_list
{
    "data": [
        {
            "{#SERVICENAME}": "192.168.1.2-p_q1_server", 
            "{#TRIGGER_VALUE}": 3
        }, 
        {
            "{#SERVICENAME}": "192.168.1.2-p_world_d2_server", 
            "{#TRIGGER_VALUE}": 1
        }, 
        {
            "{#SERVICENAME}": "192.168.1.2-p_gate_server", 
            "{#TRIGGER_VALUE}": 2
        }, 
        {
            "{#SERVICENAME}": "192.168.1.2-p_world_d1_server", 
            "{#TRIGGER_VALUE}": 1
        }
    ]
}
数据采集上报： /usr/bin/python services.py {HOST.HOST}

# -*- coding: utf-8 -*-
import json
import commands
import subprocess
import re
import sys
class services_monitor:
        def __init__(self):
            self.zabbix_server_ip = '192.168.1.1'
            self.info_path = '/home/proccessInfo.txt'
            self.data_path = '/tmp/.process_number_monitor.log'
        def ip(self):
            ipstr = '([0-9]{1,3}\.){3}[0-9]{1,3}'
            ipconfig_process = subprocess.Popen("ifconfig", stdout=subprocess.PIPE)
            output = ipconfig_process.stdout.read()
            ip_pattern = re.compile('(inet addr:%s)' % ipstr)
            pattern = re.compile(ipstr)
            iplist = []
            for ipaddr in re.finditer(ip_pattern, str(output)):
                ip = pattern.search(ipaddr.group())
                if ip.group() != "127.0.0.1":
                    iplist.append(ip.group())
            ip = '|'.join(iplist)
            return ip
        def check_proc(self,proc_name):
            cmd = 'ps -ef |grep  %s|grep -v grep|wc -l' % proc_name
            proccess_info = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE)
            # list=proccess_info.stdout.read().strip().split('\n')
            procss_num = proccess_info.communicate()[0]
            return procss_num
        def get_info(self,ip):
            service = []
            status, result = commands.getstatusoutput("grep -E '%s' %s" % (str(ip),self.info_path))
            result = result.split('\n')
            for i in result:
                i = list(i.split(' '))
                service.append({"{#SERVICENAME}": i[0].strip() + "-" + i[1].strip(), "{#TRIGGER_VALUE}":int(i[2].strip())})
            data = json.dumps({'data': service}, sort_keys=True, indent=4)
            return data
        def collect_data(self,data):
            data = json.loads(data)["data"]
            commands.getstatusoutput('cat /dev/null &gt;%s' % self.data_path)
            f = open(self.data_path,'a')
            for i  in data:
                name = i['{#SERVICENAME}'].split('-')
                ip = name[0]
                proc_name =  name[1]
                f.write('%s\tproc_num[%s]\t%s' %(ip,i['{#SERVICENAME}'],self.check_proc(proc_name)))
            f.close()
        def send_data(self,data_path):
            status,output = commands.getstatusoutput('/bin/bash -c "zabbix_sender -z  %s  -i  %s &amp;&gt;/dev/null"' % (self.zabbix_server_ip,self.data_path))
            print status,output
if __name__ == '__main__':
    services = services_monitor()
    ip = services.ip()
    data = services.get_info(ip)
    try:
        argv = sys.argv[1]
        if argv == "services_list":
            print data
        else:
            services.collect_data(data)
            services.send_data(services.data_path)
    except IndexError:
        print data

Agent添加Key

1
2
3

vim /usr/local/etc/zabbix_agentd.conf
UserParameter=dzpt.service.process.discovery,/usr/bin/python /home/opt/scripts/services.py services_list
UserParameter=dzpt.service.process.exec[*],/usr/bin/python /home/opt/scripts/services.py  $1

创建自动发现规则

(监控项Trapper方式、报警触发器)

添加当前进程数监控项

定义报警内容

Action中定义(此处略)

将定义好的模板链接到主机或者其他模板即可

最后

使用Zabbix LLD之后，可以设定多久更新一次监控项及监控阀值；当配置文件变更时，无需人为调整阀值和监控项

背景说明

方 案

部 署

系统环境

Filebeat配置

Logstash配置

生产日志目录

总结

特性功能

部署Elasticsearch

Zabbix3.4.0升级至3.4.5

Zabbix_server配置支持Elasticsearch

Zabbix Web配置历史数据读Elasticsearch

重启Zabbix Server

测试

什么是WebSocket

HTTP与WebSocket的区别

客户端通过WebSocket与服务端建立通信过程

WebSocket服务

Django Channel

WSGI/ASGI

Channels

安装配置Channels

添加路由(routing.py)

添加consumers文件(类似view)(consumers.py)

订阅Redis报警消息脚本(commands/management/commands/alert.py)

前端页面base.html

测试消息，用于发布消息到Redis

运行服务，测试

参考资料

背景说明

项目说明

功能说明

部署配置

初始化数据

Demo

Etcd是什么

使用场景

读写性能

本地集群部署

多节点集群部署

静态模式部署

环境说明(三节点集群)

初始化环境

安装etcd节点(所有节点)

etcd配置

服务管理

测试验证

安装CFSSL

容器相关证书类型

创建CA证书

生成默认CA配置

签发Server Certificate

签发Client Certificate

签发peer certificate

最后校验证书

k8s集群所需证书

参考

部署环境

部署目标

支持HTTPS

问题

部署环境

Docker及Docker-compose安装

Habor部署配置

Docker客户端使用

Harbor服务管理

目的

思路

配置流程

具体步骤

LLD自动发现脚本

Agent添加Key

创建自动发现规则

添加当前进程数监控项

定义报警内容

将定义好的模板链接到主机或者其他模板即可

最后

方案

部署