Skip to content

使用Flask、MongoDB、Redis、Celery與Docker,定時向Booking.com爬取住房資訊,同時撰寫相關API,進行住房查詢,並且導入Swagger進行API的測試,加入Celery Flower和Grafana監測隊列任務與API效能,以及使用Logstash蒐集Nginx日誌。

Notifications You must be signed in to change notification settings

LinYuMingBejing/booking.com

This branch is 1 commit ahead of master.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

4ba5aef · May 6, 2021

History

36 Commits
Dec 17, 2020
Dec 6, 2020
Aug 22, 2020
Aug 22, 2020
Aug 25, 2020
Dec 17, 2020
Jul 8, 2020
Jul 8, 2020
Sep 25, 2020
May 6, 2021
Jul 8, 2020
Dec 6, 2020
Aug 23, 2020
Sep 25, 2020

Repository files navigation

Booking.com

Enviroment:

  • Ubuntu: 16.04
  • Python: 3.6.2
  • Backend: Flask
  • Test: Swagger
  • Database: MongoDB, Redis
  • Asynchronous: Celery (Redis)
  • Monitoring Tool: Supervisor, Prometheus, Grafana
  • Collecting Logs: Logstash

Install Docker

$ sudo apt-get update

$ sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

$ sudo apt-key fingerprint 0EBFCD88

$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu
$(lsb_release -cs)
stable"

$ sudo apt-get update

$ sudo apt-get install docker-ce docker-ce-cli containerd.io

Install Docker Compose

$ sudo apt install docker-compose

Deploy Booking.com Application

  • Deploy
$ cd ./docker_stg
$ sudo docker-compose up --build -d
  • Check docker process
$ sudo docker ps
  • Get docker log
$ sudo docker-compose logs --tail=20 -f flask

Swagger

swagger

Monitor process

$ supervisorctl

supervisorctl

supervisor_web

Monitor uwsgi application

uwsgitop 127.0.0.1:5002

uwsgitop

Monitor celery worker

flower

Prometheus Monitor

Prometheus

Grafana Monitor

Grafana

Nginx Log Monitor

Note1: How to allow certain ips connect mongodb?

  • vim /etc/mongod.conf
    # bindIp: 127.0.0.1
    bindIp: 0.0.0.0  
  • restart mongodb
$ sudo service mongod restart
$ sudo service mongod status
  • firewall settings
$ sudo ufw enable
$ sudo ufw deny  from 192.168.18.0/24 to any port 27017
$ sudo ufw allow from 192.168.18.0/24 to any port 27017
$ sudo ufw status

Note2: How to create multiple user on mongodb?

  • create user
> use booking;
switched to db booking
> db.createUser(
  {
    user: "dbadmin",
    pwd: "StrongPassword",
    roles: [ { role: "readWrite", db: "booking" } ]
  }
)
> exit
bye
  • vim /etc/mongod.conf
security:
  authorization: enabled
  • restart mongodb
$ sudo service mongod restart
$ sudo service mongod status
Reference

About

使用Flask、MongoDB、Redis、Celery與Docker,定時向Booking.com爬取住房資訊,同時撰寫相關API,進行住房查詢,並且導入Swagger進行API的測試,加入Celery Flower和Grafana監測隊列任務與API效能,以及使用Logstash蒐集Nginx日誌。

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published