Highly Availabile Splash Cluster for crawling Javascript webpages using scrapy
- Create encrypted password for docker-flow-proxy (must escape the
$char like so:$$in the docker-compose file)
$ mkpasswd -m sha-512 testme
$6$WNNFqxU5G511h.Q$A/I6ciUHF7OhYvOWrFNCDZ1YbCbyxwkPbgtEHcZ9VnTG0abUq0Kbyn0VrLoQm8hLFB991ZVs254KnQnumJEQ50
- Provision manager/worker droplets using
docker-machine, add as many as required
sudo docker-machine create --driver=digitalocean --digitalocean-image=ubuntu-16-04-x64 \
--digitalocean-monitoring=true --digitalocean-private-networking=true \
--digitalocean-region=fra1 --digitalocean-size=s-4vcpu-8gb \
--digitalocean-tags=backend,dolphin \
--digitalocean-access-token=$DOTOKEN dolphin-staging-01
-
SSH into one of the prospective manager machines
sudo docker-machine ssh dolphin-staging-master -
Once inside
docker swarm init --advertise-addr node_ip_address -
Run the output from the below command to join the rest of the manager nodes to the swarm
docker swarm join-token manager -
Create user defined overlay network
docker network create --driver overlay proxy
- Deploy the images using the stack deploy command
docker stack deploy -c docker-compose-stack.yml proxy
docker stack deploy -c docker-compose-splash.yml dolphin
docker swarm init --advertise-addr <master-node-ip>
# run the output from this command on the other nodes to join as managers
docker swarm join-token manager
# run the output from this command on the other nodes to join as workers
docker swarm join-token worker
# create user-defined overlay network
docker network create --driver overlay proxy
# deploy the stack using the following docker-compose files
docker stack deploy -c docker-compose-stack.yml proxy
docker stack deploy -c docker-compose-splash.yml dolphin