Added auto inventory from vcenter. Replaced Murder Bittornado with more modern Bittorrent tools.
This commit is contained in:
parent
8beb1d5ced
commit
948045d47c
3
.gitignore
vendored
3
.gitignore
vendored
@ -35,9 +35,10 @@ nosetests.xml
|
|||||||
.project
|
.project
|
||||||
.pydevproject
|
.pydevproject
|
||||||
|
|
||||||
ansible.cfg
|
|
||||||
*.pem
|
*.pem
|
||||||
*.torrent
|
*.torrent
|
||||||
murder_dist.tar.gz
|
murder_dist.tar.gz
|
||||||
stage
|
stage
|
||||||
murder/
|
murder/
|
||||||
|
CMDB_share_credentials
|
||||||
|
vcsim_vars.yml
|
||||||
|
46
README.md
46
README.md
@ -1,20 +1,18 @@
|
|||||||
ansible-lssd
|
ansible-lssd
|
||||||
============
|
============
|
||||||
|
|
||||||
Large scale server deploys using BitTorrent and the BitTornado library by [Murder](https://github.com/lg/murder).
|
Large scale server deploys using BitTorrent. aria2c, NodeJS Bittorrent-tracker and mktorrent are used.
|
||||||
Mechanism is the same as Murder.
|
|
||||||
For an intro video, see: [Twitter - Murder Bittorrent Deploy System](http://vimeo.com/11280885)
|
|
||||||
|
|
||||||
|
|
||||||
DESCRIPTION
|
DESCRIPTION
|
||||||
-----------
|
-----------
|
||||||
|
|
||||||
ansible-lssd is a method of using Bittorrent (Powered by [Murder](https://github.com/lg/murder)) to distribute files to a large amount of servers within a production environment.
|
ansible-lssd is a method of using Bittorrent to distribute files to a large amount of servers within a production environment.
|
||||||
|
|
||||||
ansible-lssd operates as a playbook of [Ansible](https://github.com/ansible/ansible).
|
ansible-lssd operates as a playbook of [Ansible](https://github.com/ansible/ansible).
|
||||||
These playbooks require Ansible 1.4.
|
These playbooks require Ansible 1.4.
|
||||||
|
|
||||||
These playbooks were tested on CentOS 5.x so we recommend that you use CentOS to test these modules.
|
These playbooks were tested on CentOS 7.x (ansible 2.6) with peers on Centos 7.x and Centos 6.x.
|
||||||
|
|
||||||
Advantage
|
Advantage
|
||||||
---------
|
---------
|
||||||
@ -28,12 +26,8 @@ There is the following advantage as compared with Murder.
|
|||||||
Installation of software required minimum
|
Installation of software required minimum
|
||||||
-----------------------------------------
|
-----------------------------------------
|
||||||
|
|
||||||
`sudo pip install ansible` :
|
From EPEL:
|
||||||
```bash:
|
# yum -y install ansible
|
||||||
# install pip (Python package manager) and ansible
|
|
||||||
$ sudo easy_install pip
|
|
||||||
$ sudo pip install ansible
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
HOW IT WORKS
|
HOW IT WORKS
|
||||||
@ -47,15 +41,9 @@ CONFIGURATION
|
|||||||
|
|
||||||
You define `tracker`, `seeder` and `peer` server to inventory (./production) file.
|
You define `tracker`, `seeder` and `peer` server to inventory (./production) file.
|
||||||
|
|
||||||
All involved servers must have python installed and the related murder
|
All involved servers must have python installed.
|
||||||
support files (BitTornado, Murder lib, screen, pigz and etc.). To upload the support files to the tracker, seeder, and peers, run:
|
|
||||||
```bash:
|
|
||||||
$ ansible-playbook -i prodction setup.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
By default, these will go in `/usr/local/murder` in your apps deploy directory.
|
|
||||||
Override this by setting the variable `remote_murder_path`.
|
|
||||||
|
|
||||||
|
The file group_vars/all contains the configuration.
|
||||||
|
|
||||||
MANUAL USAGE
|
MANUAL USAGE
|
||||||
------------
|
------------
|
||||||
@ -100,7 +88,7 @@ Then manually run the ansible playbooks:
|
|||||||
1. Start the tracker:
|
1. Start the tracker:
|
||||||
|
|
||||||
```bash:
|
```bash:
|
||||||
$ ansible-playbook -i prodction start_tracker.yml
|
$ ansible-playbook -i production start_tracker.yml
|
||||||
```
|
```
|
||||||
|
|
||||||
2. Create a torrent from a directory of files on the seeder, and start seeding:
|
2. Create a torrent from a directory of files on the seeder, and start seeding:
|
||||||
@ -110,22 +98,22 @@ Then manually run the ansible playbooks:
|
|||||||
$ scp -r ./builds 10.0.0.1:~/builds
|
$ scp -r ./builds 10.0.0.1:~/builds
|
||||||
|
|
||||||
# create torrent file
|
# create torrent file
|
||||||
$ ansible-playbook -i prodction create_torrent.yml
|
$ ansible-playbook -i production create_torrent.yml
|
||||||
|
|
||||||
# start seeding
|
# start seeding
|
||||||
$ ansible-playbook -i prodction start_seeder.yml
|
$ ansible-playbook -i production start_seeder.yml
|
||||||
```
|
```
|
||||||
|
|
||||||
3. Distribute the torrent to all peers:
|
3. Distribute the torrent to all peers:
|
||||||
|
|
||||||
```bash:
|
```bash:
|
||||||
$ ansible-playbook -i prodction deploy.yml -f 1000
|
$ ansible-playbook -i production deploy.yml -f 1000
|
||||||
```
|
```
|
||||||
|
|
||||||
4. Stop the seeder and tracker:
|
4. Stop the seeder and tracker:
|
||||||
|
|
||||||
```bash:
|
```bash:
|
||||||
$ ansible-playbook -i prodction stop_seeder_and_tracker.yml
|
$ ansible-playbook -i production stop_seeder_and_tracker.yml
|
||||||
```
|
```
|
||||||
|
|
||||||
When this finishes, all peers will have the files in /opt/hoge/Deploy1
|
When this finishes, all peers will have the files in /opt/hoge/Deploy1
|
||||||
@ -138,11 +126,9 @@ MAIN PLAYBOOKS REFERENCE
|
|||||||
* Create torrent file on seeder node.
|
* Create torrent file on seeder node.
|
||||||
* `deploy.yml:`
|
* `deploy.yml:`
|
||||||
* Deploy files on peer nodes.
|
* Deploy files on peer nodes.
|
||||||
* `rm_tgz.yml:`
|
|
||||||
* Delete the file deployment of targz in all peer node.
|
|
||||||
* `setup.yml:`
|
* `setup.yml:`
|
||||||
* Install the software required for each node.
|
* Install the software required for each node.
|
||||||
* `site.yml:`
|
* `full_deploy.yml:`
|
||||||
* Run all Playbook deploy from the setup.
|
* Run all Playbook deploy from the setup.
|
||||||
* `start_seeder.yml:`
|
* `start_seeder.yml:`
|
||||||
* start seeding.
|
* start seeding.
|
||||||
@ -154,6 +140,10 @@ MAIN PLAYBOOKS REFERENCE
|
|||||||
* stop seeding and tracker.
|
* stop seeding and tracker.
|
||||||
|
|
||||||
|
|
||||||
|
NOTES
|
||||||
|
------------------------
|
||||||
|
|
||||||
|
You may hit the request length limit (fixed in newer aria2 releases) :
|
||||||
|
https://github.com/aria2/aria2/commit/e220c5384961f9261f19c28cd0b85f76f06d8993#diff-03671aebef9174610c96db97917b960a
|
||||||
|
|
||||||
|
Workaround is using "-l 22" as mktorrent option to make less pieces.
|
||||||
|
4
ansible.cfg
Normal file
4
ansible.cfg
Normal file
@ -0,0 +1,4 @@
|
|||||||
|
[defaults]
|
||||||
|
forks = 150
|
||||||
|
host_key_checking = False
|
||||||
|
callback_whitelist = profile_tasks
|
BIN
bittorrent-tracker.tar.gz
Executable file
BIN
bittorrent-tracker.tar.gz
Executable file
Binary file not shown.
23
create_inventory.sh
Executable file
23
create_inventory.sh
Executable file
@ -0,0 +1,23 @@
|
|||||||
|
#!/usr/bin/python
|
||||||
|
|
||||||
|
#relays={"HLZURT1EXE3M": {"Bat": "Bat Diderot", "Salle": "Salle 1"}, "HLZURT1EXE3N": {"Bat": "D'alembert", "Salle": "Salle 2"}}
|
||||||
|
relays={"HLZURT1EXE3M": {"Bat": "Bat C", "Salle": "Salle C31"}}
|
||||||
|
inventory_file="associations_vm-location"
|
||||||
|
|
||||||
|
for relay in relays.keys():
|
||||||
|
with open("inventory_" + relay, "w") as f:
|
||||||
|
f.write("# ansible host\n")
|
||||||
|
f.write("[ansible_host]\n")
|
||||||
|
f.write("localhost ansible_connection=local\n\n")
|
||||||
|
f.write("# tracker node\n")
|
||||||
|
f.write("[tracker]\n")
|
||||||
|
f.write(relay.lower() + " node_type=tracker\n\n")
|
||||||
|
f.write("# seeder node\n")
|
||||||
|
f.write("[seeder]\n")
|
||||||
|
f.write(relay.lower() + " node_type=seeder\n\n")
|
||||||
|
f.write("# peer nodes\n")
|
||||||
|
f.write("[peer]\n")
|
||||||
|
with open(inventory_file, "r") as inventory:
|
||||||
|
for line in inventory:
|
||||||
|
if relays[relay]["Bat"] in line and relays[relay]["Salle"] in line:
|
||||||
|
f.write(line.split(';')[0]+"\n")
|
104
create_inventory.yml
Normal file
104
create_inventory.yml
Normal file
@ -0,0 +1,104 @@
|
|||||||
|
- name: create_inventory
|
||||||
|
gather_facts: no
|
||||||
|
vars_files:
|
||||||
|
- vcsim_vars.yml
|
||||||
|
- vm_list.yml
|
||||||
|
hosts: localhost
|
||||||
|
|
||||||
|
tasks:
|
||||||
|
# - name: Gather VMware guest facts
|
||||||
|
# vmware_guest_facts:
|
||||||
|
# hostname: "{{ vcenter_server }}"
|
||||||
|
# username: "{{ vcenter_user }}"
|
||||||
|
# password: "{{ vcenter_pass }}"
|
||||||
|
# datacenter:
|
||||||
|
# name: "{{ item }}"
|
||||||
|
# validate_certs: no
|
||||||
|
# register: vm_result
|
||||||
|
# with_items: "{{ vm_list }}"
|
||||||
|
|
||||||
|
- name: Gather VMware guest facts
|
||||||
|
vmware_guest_facts:
|
||||||
|
hostname: "{{ item[1] }}"
|
||||||
|
username: "{{ vcenter_user }}"
|
||||||
|
password: "{{ vcenter_pass }}"
|
||||||
|
datacenter:
|
||||||
|
name: "{{ item[0] }}"
|
||||||
|
validate_certs: no
|
||||||
|
register: vm_result
|
||||||
|
ignore_errors: true
|
||||||
|
with_nested:
|
||||||
|
- "{{ vm_list }}"
|
||||||
|
- "{{ vcenter_servers }}"
|
||||||
|
|
||||||
|
- name: remove output file
|
||||||
|
file: name=./associations_vm-esx state=absent
|
||||||
|
|
||||||
|
- name: make output file
|
||||||
|
file: name=./associations_vm-esx state=touch
|
||||||
|
|
||||||
|
- name: Get ESXs of VMs
|
||||||
|
vars:
|
||||||
|
s_query: "[?instance.hw_name == '{{ item }}'].instance.hw_esxi_host"
|
||||||
|
with_items: "{{ vm_list }}"
|
||||||
|
lineinfile: dest=./associations_vm-esx line="{{ item }};{{ vm_result.results | json_query(s_query) | join }}"
|
||||||
|
|
||||||
|
- name: check results
|
||||||
|
shell: "[ $(grep {{ item }} ./associations_vm-esx| awk -F';' '{print $2}') != '' ] || { echo 'Could not find ESX for VM : {{ item }}'; exit 1; }"
|
||||||
|
with_items: "{{ vm_list }}"
|
||||||
|
|
||||||
|
- name: Get ESXs of VMs2
|
||||||
|
set_fact:
|
||||||
|
esx_list: "{{ esx_list|default([]) + [ vm_result.results | json_query(s_query) | join ] }}"
|
||||||
|
vars:
|
||||||
|
s_query: "[?instance.hw_name == '{{ item }}'].instance.hw_esxi_host"
|
||||||
|
with_items: "{{ vm_list }}"
|
||||||
|
|
||||||
|
- name: remove output file
|
||||||
|
file: name=./associations_vm-location state=absent
|
||||||
|
|
||||||
|
- name: make output file
|
||||||
|
file: name=./associations_vm-location state=touch
|
||||||
|
|
||||||
|
- name: create mount directory
|
||||||
|
file: name=/mnt/CMDB state=directory
|
||||||
|
become: yes
|
||||||
|
|
||||||
|
- name: mount CMDB share
|
||||||
|
shell: mountpoint -q /mnt/CMDB || mount -t cifs //CPMUPD2APPVQ.ZRES.ZTECH/mup10cmdb_s/Envoi/PUBLICATION /mnt/CMDB -o credentials={{playbook_dir}}/CMDB_share_credentials,domain=ZGIE
|
||||||
|
become: yes
|
||||||
|
|
||||||
|
- name: find CMDB file
|
||||||
|
shell: ls -t /mnt/CMDB/MUP10CMDBH_*LISTE_CS.CSV|head -n1
|
||||||
|
register: CMDB_file
|
||||||
|
|
||||||
|
- name: store CMDB file path
|
||||||
|
set_fact:
|
||||||
|
CMDB_file_path: "{{ CMDB_file.stdout }}"
|
||||||
|
|
||||||
|
- name: retrieve ESXs' room
|
||||||
|
shell: grep -i "{{ item.split('.')[0] }}" {{ CMDB_file_path }}|awk -F',' '{print $14}'|tr -d '\t\n'
|
||||||
|
register: esx_locations
|
||||||
|
with_items: "{{ esx_list }}"
|
||||||
|
|
||||||
|
- name: umount CMDB share
|
||||||
|
shell: mountpoint -q /mnt/CMDB && umount /mnt/CMDB
|
||||||
|
become: yes
|
||||||
|
|
||||||
|
- name: store location
|
||||||
|
set_fact:
|
||||||
|
location: "{{ esx_locations.results | map(attribute='stdout') | list }}"
|
||||||
|
|
||||||
|
- name: store associations
|
||||||
|
set_fact:
|
||||||
|
associations: "{{ (vm_list|zip([';'] * vm_list|length)|map('join')|list) | zip(location) | list | map('join') | list }}"
|
||||||
|
|
||||||
|
- name: save associations in file
|
||||||
|
#debug:
|
||||||
|
# msg: "{{ item }}"
|
||||||
|
lineinfile: dest=./associations_vm-location line="{{ item }}"
|
||||||
|
with_items: "{{ associations }}"
|
||||||
|
|
||||||
|
- name: create inventories
|
||||||
|
command: ./create_inventory.sh
|
||||||
|
|
@ -1,11 +1,9 @@
|
|||||||
---
|
---
|
||||||
- include: setup.yml
|
|
||||||
- include: stop_seeder_and_tracker.yml
|
- include: stop_seeder_and_tracker.yml
|
||||||
- include: stop_all_peers.yml
|
- include: stop_all_peers.yml
|
||||||
- include: rm_tgz.yml
|
- include: setup.yml
|
||||||
- include: start_tracker.yml
|
- include: start_tracker.yml
|
||||||
- include: create_torrent.yml
|
#- include: create_torrent.yml
|
||||||
- include: start_seeder.yml
|
- include: start_seeder.yml
|
||||||
- include: deploy.yml
|
- include: deploy.yml
|
||||||
- include: stop_seeder_and_tracker.yml
|
- include: stop_seeder_and_tracker.yml
|
||||||
|
|
@ -1,30 +1,36 @@
|
|||||||
# The global variable file
|
# The global variable file
|
||||||
|
|
||||||
# deploy tag
|
# deploy tag
|
||||||
tag: deploy_test2
|
tag: archive_clonage
|
||||||
|
|
||||||
|
# ADABO
|
||||||
|
seeder_ADABO: ZUR10
|
||||||
|
peer_ADABO: ZUR30
|
||||||
|
|
||||||
# path
|
# path
|
||||||
seeder_files_path: ~/builds
|
seeder_files_path: /data/{{ seeder_ADABO | upper }}/Sas/clonage/Envoi
|
||||||
destination_path: /product/hoge
|
destination_path: /opt/{{ peer_ADABO | upper }}/Logiciel/BDIG/aem-install/crx-quickstart
|
||||||
|
|
||||||
# path permission
|
# path permission
|
||||||
destination_owner: root
|
destination_owner: cdap{{ peer_ADABO | lower }}bdig
|
||||||
destination_group: root
|
destination_group: go_{{ peer_ADABO | lower }}_bdig
|
||||||
destination_mod: 755
|
destination_mod: "0755"
|
||||||
|
|
||||||
# default tracker port
|
# default tracker port
|
||||||
tracker_port: 8998
|
tracker_port: 6969
|
||||||
|
|
||||||
# tar options
|
# tar options
|
||||||
tar_options: --exclude ".git*" --exclude ".svn*"
|
tar_options: --exclude ".git*" --exclude ".svn*"
|
||||||
|
|
||||||
# options
|
# options
|
||||||
no_tag_directory: false
|
no_tag_directory: true
|
||||||
unsafe_please_delete: false
|
unsafe_please_delete: false
|
||||||
|
only_deploy_archive: true
|
||||||
|
remove_existing: false
|
||||||
|
|
||||||
# system path
|
# system path
|
||||||
torrent_path: .
|
src_path: "{{ playbook_dir }}"
|
||||||
remote_murder_path: /usr/local/murder
|
remote_bittorrent_tracker_path: /usr/local/bittorrent-tracker
|
||||||
ansible_cache_path: /var/cache/ansible
|
ansible_cache_path: /var/cache/ansible
|
||||||
default_temp_path: /tmp
|
default_temp_path: /tmp
|
||||||
murder_dist_tgz_file: ./murder_dist.tar.gz
|
bittorrent_tracker_tgz_file: bittorrent-tracker.tar.gz
|
||||||
|
18
inventory_HLZURT1EXE3M
Normal file
18
inventory_HLZURT1EXE3M
Normal file
@ -0,0 +1,18 @@
|
|||||||
|
# ansible host
|
||||||
|
[ansible_host]
|
||||||
|
localhost ansible_connection=local
|
||||||
|
|
||||||
|
# tracker node
|
||||||
|
[tracker]
|
||||||
|
hlzurt1exe3m node_type=tracker
|
||||||
|
|
||||||
|
# seeder node
|
||||||
|
[seeder]
|
||||||
|
hlzurt1exe3m node_type=seeder
|
||||||
|
|
||||||
|
# peer nodes
|
||||||
|
[peer]
|
||||||
|
HLZURT1EXE04
|
||||||
|
HLZURT1EXE05
|
||||||
|
HLZURT1EXE06
|
||||||
|
HLZURT1EXE08
|
BIN
packages/aria2-1.16.4-1.el6.rf.x86_64.rpm
Normal file
BIN
packages/aria2-1.16.4-1.el6.rf.x86_64.rpm
Normal file
Binary file not shown.
BIN
packages/aria2-1.18.10-2.el7.1.x86_64.rpm
Normal file
BIN
packages/aria2-1.18.10-2.el7.1.x86_64.rpm
Normal file
Binary file not shown.
BIN
packages/nettle-2.2-1.el6.rf.x86_64.rpm
Normal file
BIN
packages/nettle-2.2-1.el6.rf.x86_64.rpm
Normal file
Binary file not shown.
BIN
packages/pigz-2.3.4-1.el6.x86_64.rpm
Normal file
BIN
packages/pigz-2.3.4-1.el6.x86_64.rpm
Normal file
Binary file not shown.
BIN
packages/pigz-2.3.4-1.el7.x86_64.rpm
Normal file
BIN
packages/pigz-2.3.4-1.el7.x86_64.rpm
Normal file
Binary file not shown.
@ -1,9 +0,0 @@
|
|||||||
|
|
||||||
|
|
||||||
- name: rm_tgz
|
|
||||||
gather_facts: no
|
|
||||||
hosts: peer
|
|
||||||
sudo: yes
|
|
||||||
roles:
|
|
||||||
- murder/rm_tgz
|
|
||||||
|
|
@ -1,14 +0,0 @@
|
|||||||
|
|
||||||
- name: clone murder
|
|
||||||
git: repo=https://github.com/lg/murder.git dest=./murder
|
|
||||||
register: clone_status
|
|
||||||
tags: clone_murder_files
|
|
||||||
|
|
||||||
- name: rm old tgz
|
|
||||||
shell: rm -f {{ murder_dist_tgz_file }}
|
|
||||||
when: clone_status.changed
|
|
||||||
|
|
||||||
- name: tar -czf
|
|
||||||
shell: tar -czf {{ murder_dist_tgz_file }} -C ./murder/dist/ . creates={{ murder_dist_tgz_file }}
|
|
||||||
tags: clone_murder_files
|
|
||||||
|
|
31
roles/admin/deploy_bittorrent-tracker/tasks/main.yml
Normal file
31
roles/admin/deploy_bittorrent-tracker/tasks/main.yml
Normal file
@ -0,0 +1,31 @@
|
|||||||
|
# SCPs a compressed version of all files from ./dist (the python Bittorrent library and custom scripts) to all server.
|
||||||
|
# The entire directory is sent, regardless of the role of each individual server.
|
||||||
|
# The path on the server is specified by remote_bittorrent-tracker_path and will be cleared prior to transferring files over.
|
||||||
|
|
||||||
|
- name: upload
|
||||||
|
copy: src={{ bittorrent_tracker_tgz_file }} dest={{ ansible_cache_path }}/bittorrent-tracker.tar.gz
|
||||||
|
tags: deploy_bittorrent-tracker
|
||||||
|
register: copy_status
|
||||||
|
|
||||||
|
- name: clean up remote_bittorrent-tracker_path
|
||||||
|
shell: |
|
||||||
|
[ $(find {{ remote_bittorrent_tracker_path }}/* | wc -l ) -lt 1000 ] &&
|
||||||
|
rm -rf {{ remote_bittorrent_tracker_path }} ||
|
||||||
|
( echo 'Cowardly refusing to remove files! Check the remote_bittorrent-tracker_path.' ; exit 1 )
|
||||||
|
when: copy_status.changed
|
||||||
|
register: rm_status
|
||||||
|
tags: deploy_bittorrent-tracker
|
||||||
|
|
||||||
|
- name: dir stat
|
||||||
|
stat: path={{ remote_bittorrent_tracker_path }}/node_modules
|
||||||
|
register: st
|
||||||
|
|
||||||
|
- name: tar -xzf
|
||||||
|
shell: mkdir -p {{ remote_bittorrent_tracker_path }} && tar -xzf {{ ansible_cache_path }}/bittorrent-tracker.tar.gz -C {{ remote_bittorrent_tracker_path }}
|
||||||
|
when: st.stat.isdir is undefined
|
||||||
|
tags: deploy_bittorrent-tracker
|
||||||
|
|
||||||
|
- name: install NodeJS
|
||||||
|
yum: name=nodejs state=latest
|
||||||
|
tags: deploy_bittorrent-tracker
|
||||||
|
|
@ -1,32 +0,0 @@
|
|||||||
# SCPs a compressed version of all files from ./dist (the python Bittorrent library and custom scripts) to all server.
|
|
||||||
# The entire directory is sent, regardless of the role of each individual server.
|
|
||||||
# The path on the server is specified by remote_murder_path and will be cleared prior to transferring files over.
|
|
||||||
- name: mkdir ansible_cache_path
|
|
||||||
file: path={{ ansible_cache_path }} state=directory owner=root group=root mode=0755
|
|
||||||
tags: distribute_files
|
|
||||||
|
|
||||||
|
|
||||||
- name: upload
|
|
||||||
copy: src={{ murder_dist_tgz_file }} dest={{ ansible_cache_path }}/murder_dist.tar.gz
|
|
||||||
tags: distribute_files
|
|
||||||
register: copy_status
|
|
||||||
|
|
||||||
- name: clean up remote_murder_path
|
|
||||||
shell: |
|
|
||||||
[ $(find {{ remote_murder_path }}/* | wc -l ) -lt 1000 ] &&
|
|
||||||
rm -rf {{ remote_murder_path }} ||
|
|
||||||
( echo 'Cowardly refusing to remove files! Check the remote_murder_path.' ; exit 1 )
|
|
||||||
when: copy_status.changed
|
|
||||||
register: rm_status
|
|
||||||
tags: distribute_files
|
|
||||||
|
|
||||||
- name: dir stat
|
|
||||||
stat: path={{ remote_murder_path }}/BitTornado
|
|
||||||
register: st
|
|
||||||
|
|
||||||
- name: tar -xzf
|
|
||||||
shell: mkdir -p {{ remote_murder_path }} && tar -xzf {{ ansible_cache_path }}/murder_dist.tar.gz -C {{ remote_murder_path }}
|
|
||||||
when: st.stat.isdir is undefined
|
|
||||||
tags: distribute_files
|
|
||||||
|
|
||||||
|
|
BIN
roles/admin/pigz/tasks/.main.yml.swp
Normal file
BIN
roles/admin/pigz/tasks/.main.yml.swp
Normal file
Binary file not shown.
@ -1,5 +1,19 @@
|
|||||||
|
- name: copy pigz 6
|
||||||
|
copy:
|
||||||
|
src: packages/pigz-2.3.4-1.el6.x86_64.rpm
|
||||||
|
dest: "{{ ansible_cache_path }}"
|
||||||
|
when: ansible_distribution_major_version == "6"
|
||||||
|
|
||||||
- name: yum install pigz
|
- name: copy pigz 7
|
||||||
yum: name=pigz state=latest
|
copy:
|
||||||
tags: pigz
|
src: packages/pigz-2.3.4-1.el7.x86_64.rpm
|
||||||
|
dest: "{{ ansible_cache_path }}"
|
||||||
|
when: ansible_distribution_major_version == "7"
|
||||||
|
|
||||||
|
- name: yum localinstall pigz
|
||||||
|
shell: yum -y localinstall "{{ ansible_cache_path }}"/pigz*.rpm --nogpgcheck
|
||||||
|
|
||||||
|
#- name: yum install pigz
|
||||||
|
# yum: name=pigz state=latest
|
||||||
|
# tags: pigz
|
||||||
|
|
||||||
|
13
roles/admin/prepare/tasks/main.yml
Normal file
13
roles/admin/prepare/tasks/main.yml
Normal file
@ -0,0 +1,13 @@
|
|||||||
|
- name: check ansible_cache_path
|
||||||
|
stat: path={{ ansible_cache_path }}
|
||||||
|
register: test_path
|
||||||
|
tags: prepare
|
||||||
|
|
||||||
|
- name: remove ansible_cache_path if not a directory
|
||||||
|
file: path={{ ansible_cache_path }} state=absent
|
||||||
|
when: test_path.stat is defined and (test_path.stat.isdir is not defined or not test_path.stat.isdir)
|
||||||
|
tags: prepare
|
||||||
|
|
||||||
|
- name: mkdir ansible_cache_path
|
||||||
|
file: path={{ ansible_cache_path }} state=directory owner=root group=root mode=0755
|
||||||
|
tags: prepare
|
@ -1,5 +1,5 @@
|
|||||||
# Starts the Bittorrent tracker (essentially a mini-web-server) listening on port 8998.
|
# Starts the Bittorrent tracker (essentially a mini-web-server)
|
||||||
- name: start tracker
|
- name: start tracker
|
||||||
shell: SCREENRC=/dev/null SYSSCREENRC=/dev/null screen -dmS murder_tracker python {{ remote_murder_path }}/murder_tracker.py && sleep 0.2
|
shell: SCREENRC=/dev/null SYSSCREENRC=/dev/null screen -dmS bittorrent-tracker {{ remote_bittorrent_tracker_path }}/node_modules/bittorrent-tracker/bin/cmd.js --http -p 6969 && sleep 0.2
|
||||||
tags: start_tracker
|
tags: start_tracker
|
||||||
|
|
||||||
|
@ -1,7 +1,7 @@
|
|||||||
# Sometimes peers can go on forever (usually because of an error).
|
# Sometimes peers can go on forever (usually because of an error).
|
||||||
# This command will forcibly kill all "murder_client.py peer" commands that are running.
|
# This command will forcibly kill all "murder_client.py peer" commands that are running.
|
||||||
- name: stop all peering
|
- name: stop all peering
|
||||||
shell: pkill -f "murder_client.py peer"
|
shell: pkill -f "aria2c"
|
||||||
register: result
|
register: result
|
||||||
failed_when: result.rc not in [0, 1]
|
failed_when: result.rc not in [0, 1]
|
||||||
changed_when: result.rc not in [0, 1]
|
changed_when: result.rc not in [0, 1]
|
||||||
|
@ -1,8 +1,9 @@
|
|||||||
# If the Bittorrent tracker is running, this will kill the process. Note that if it is not running you will receive an error.
|
# If the Bittorrent tracker is running, this will kill the process. Note that if it is not running you will receive an error.
|
||||||
- name: stop tracker
|
- name: stop tracker
|
||||||
shell: pkill -f 'SCREEN.*murder_tracker.py'
|
shell: pkill -f 'SCREEN.*bittorrent-tracker*'
|
||||||
register: result
|
register: result
|
||||||
failed_when: result.rc not in [0, 1]
|
failed_when: result.rc not in [0, 1]
|
||||||
changed_when: result.rc not in [0, 1]
|
changed_when: result.rc not in [0, 1]
|
||||||
tags: stop_tracker
|
tags: stop_tracker
|
||||||
|
|
||||||
|
|
||||||
|
@ -1,7 +1,15 @@
|
|||||||
|
|
||||||
- name: set tagname to filename
|
- name: set tagname to filename
|
||||||
set_fact: filename={{ default_temp_path }}/{{ tag }}.tar.gz
|
set_fact: filename={{ destination_path }}/{{ tag }}.tar.gz
|
||||||
tags: common
|
tags: common
|
||||||
|
|
||||||
|
|
||||||
|
- name: set tagname to source filename
|
||||||
|
set_fact: source_filename={{ seeder_files_path }}/{{ tag }}.tar.gz
|
||||||
|
tags: common
|
||||||
|
|
||||||
|
|
||||||
|
- name: set tagname to base filename
|
||||||
|
set_fact: base_filename={{ tag }}.tar.gz
|
||||||
|
tags: common
|
||||||
|
|
||||||
|
@ -1,21 +1,47 @@
|
|||||||
|
|
||||||
- name: create tar.gz
|
#- name: create tar.gz
|
||||||
shell: tar -c -C {{ seeder_files_path }}/ {{ tar_options }} .|pigz > {{ filename }}
|
# shell: tar -c -C {{ seeder_files_path }}/ {{ tar_options }} .|pigz > {{ filename }}
|
||||||
|
# when: node_type == "seeder"
|
||||||
|
# tags: create_tgz
|
||||||
|
|
||||||
|
- name: remount cifs share
|
||||||
|
shell: systemctl restart SL{{ seeder_ADABO }}CIFSBDIG_S.service
|
||||||
when: node_type == "seeder"
|
when: node_type == "seeder"
|
||||||
tags: create_tgz
|
tags: create_torrent
|
||||||
|
become: yes
|
||||||
|
|
||||||
- name: set tracker ip
|
- name: set tracker ip
|
||||||
set_fact: tracker_host={{ hostvars[groups['tracker'][0]]['ansible_default_ipv4']['address'] }}
|
set_fact: tracker_host={{ hostvars[groups['tracker'][0]]['ansible_default_ipv4']['address'] }}
|
||||||
when: node_type == "seeder"
|
when: node_type == "seeder"
|
||||||
tags: create_torrent
|
tags: create_torrent
|
||||||
|
|
||||||
- name: create torrent
|
- name: remove torrent
|
||||||
shell: python {{ remote_murder_path }}/murder_make_torrent.py '{{ filename }}' {{ tracker_host }}:{{ tracker_port }} '{{ filename }}.torrent'
|
shell: rm '/tmp/{{ base_filename }}.torrent'
|
||||||
when: node_type == "seeder"
|
when: node_type == "seeder"
|
||||||
tags: create_torrent
|
tags: create_torrent
|
||||||
|
|
||||||
- name: fetch torrent
|
- name: install mktorrent
|
||||||
fetch: src={{ filename }}.torrent dest={{ torrent_path }}/ flat=yes fail_on_missing=yes
|
yum: name=mktorrent state=latest
|
||||||
when: node_type == "seeder"
|
when: node_type == "seeder"
|
||||||
tags: fetch_torrent
|
tags: create_torrent
|
||||||
|
|
||||||
|
- name: create torrent
|
||||||
|
shell: mktorrent -l 21 -a "http://{{ tracker_host }}:{{ tracker_port }}/announce" -o '/tmp/{{ base_filename }}.torrent' '{{ source_filename }}'
|
||||||
|
when: node_type == "seeder"
|
||||||
|
tags: create_torrent
|
||||||
|
|
||||||
|
- name: chmod torrent file
|
||||||
|
shell: chmod a+r '/tmp/{{ base_filename }}.torrent'
|
||||||
|
when: node_type == "seeder"
|
||||||
|
tags: create_torrent
|
||||||
|
|
||||||
|
#- name: create torrent
|
||||||
|
# shell: python {{ remote_murder_path }}/murder_make_torrent.py '{{ source_filename }}' {{ tracker_host }}:{{ tracker_port }} '/tmp/{{ base_filename }}.torrent'
|
||||||
|
# when: node_type == "seeder"
|
||||||
|
# tags: create_torrent
|
||||||
|
|
||||||
|
#- name: fetch torrent
|
||||||
|
#- name: fetch torrent
|
||||||
|
# fetch: src={{ base_filename }}.torrent dest={{ ansible_cache_path }}/ flat=yes fail_on_missing=yes
|
||||||
|
# when: node_type == "seeder"
|
||||||
|
# tags: fetch_torrent
|
||||||
|
@ -21,6 +21,11 @@
|
|||||||
when: unsafe_please_delete
|
when: unsafe_please_delete
|
||||||
tags: peer
|
tags: peer
|
||||||
|
|
||||||
|
- name: supprime
|
||||||
|
shell: /opt/{{ peer_ADABO }}/Logiciel/BDIG/Specifique_CAT/Scripts/clone_torrent_etape_supprime.sh
|
||||||
|
when: not only_deploy_archive
|
||||||
|
tags: peer
|
||||||
|
|
||||||
- name: check destination_path/*
|
- name: check destination_path/*
|
||||||
shell: |
|
shell: |
|
||||||
(
|
(
|
||||||
@ -33,14 +38,58 @@
|
|||||||
tags: peer
|
tags: peer
|
||||||
|
|
||||||
- name: upload torrent file
|
- name: upload torrent file
|
||||||
copy: src={{ torrent_path }}/{{ filename|basename }}.torrent dest={{ filename }}.torrent
|
copy: src=/tmp/{{ filename|basename }}.torrent dest={{ filename }}.torrent
|
||||||
tags: peer
|
tags: peer
|
||||||
|
|
||||||
- name: run murder_client.py
|
- name: copy aria2 RHEL6
|
||||||
shell: python {{ remote_murder_path }}/murder_client.py peer '{{ filename }}.torrent' '{{filename}}' {{ ansible_default_ipv4['address'] }}
|
copy: src={{ src_path }}/packages/aria2-1.16.4-1.el6.rf.x86_64.rpm dest="{{ ansible_cache_path }}/"
|
||||||
|
tags: peer
|
||||||
|
when: ansible_distribution_major_version == "6"
|
||||||
|
|
||||||
|
- name: copy nettle RHEL6
|
||||||
|
copy: src={{ src_path }}/packages/nettle-2.2-1.el6.rf.x86_64.rpm dest="{{ ansible_cache_path }}/"
|
||||||
|
tags: peer
|
||||||
|
when: ansible_distribution_major_version == "6"
|
||||||
|
|
||||||
|
- name: copy aria2 RHEL7
|
||||||
|
copy: src={{ src_path }}/packages/aria2-1.18.10-2.el7.1.x86_64.rpm dest="{{ ansible_cache_path }}/"
|
||||||
|
tags: peer
|
||||||
|
when: ansible_distribution_major_version == "7"
|
||||||
|
|
||||||
|
- name: install aria2 RHEL6
|
||||||
|
shell: yum -y localinstall "{{ ansible_cache_path }}"/aria2*.rpm "{{ ansible_cache_path }}"/nettle*.rpm --nogpgcheck
|
||||||
|
tags: peer
|
||||||
|
when: ansible_distribution_major_version == "6"
|
||||||
|
|
||||||
|
- name: install aria2 RHEL7
|
||||||
|
shell: yum -y localinstall "{{ ansible_cache_path }}"/aria2*.rpm --nogpgcheck
|
||||||
|
tags: peer
|
||||||
|
when: ansible_distribution_major_version == "7"
|
||||||
|
|
||||||
|
- name: clean existing files
|
||||||
|
shell: rm -rf {{ filename }} {{ filename }}.aria2
|
||||||
|
when: remove_existing
|
||||||
tags: peer
|
tags: peer
|
||||||
|
|
||||||
- name: tar xf
|
- name: run aria2
|
||||||
|
shell: aria2c --console-log-level=notice --peer-id-prefix={{ inventory_hostname[-8:] }} --disable-ipv6 --enable-dht=false --listen-port=6881 --allow-overwrite=true --bt-max-peers=4 --enable-peer-exchange=true --bt-seed-unverified=true --seed-time=1 -d '{{ destination_path }}' '{{ filename }}.torrent' &> /var/log/{{ peer_ADABO }}/BDIG/torrent.log
|
||||||
|
# shell: aria2c --console-log-level=debug --disable-ipv6 --enable-dht=false --listen-port=6881 --allow-overwrite=true --bt-max-peers=4 --enable-peer-exchange=true --bt-seed-unverified=true --seed-time=1 -d '{{ destination_path }}' '{{ filename }}.torrent' &> /var/log/{{ peer_ADABO }}/BDIG/torrent.log
|
||||||
|
tags: peer
|
||||||
|
|
||||||
|
#- name: run murder_client.py
|
||||||
|
# shell: python {{ remote_murder_path }}/murder_client.py peer '{{ filename }}.torrent' '{{filename}}' {{ ansible_default_ipv4['address'] }}
|
||||||
|
# tags: peer
|
||||||
|
|
||||||
|
- name: tar xf
|
||||||
shell: unpigz -c {{ filename }} | tar -x -C "{{ destination_path }}"
|
shell: unpigz -c {{ filename }} | tar -x -C "{{ destination_path }}"
|
||||||
|
when: not only_deploy_archive
|
||||||
tags: peer
|
tags: peer
|
||||||
|
|
||||||
|
- name: clean downloaded files
|
||||||
|
shell: rm -rf {{ filename }} {{ filename }}.torrent
|
||||||
|
tags: peer
|
||||||
|
|
||||||
|
- name: clone_torrent.sh
|
||||||
|
shell: /opt/{{ peer_ADABO }}/Logiciel/BDIG/Specifique_CAT/Scripts/clone_torrent.sh
|
||||||
|
when: not only_deploy_archive
|
||||||
|
tags: peer
|
||||||
|
@ -1,3 +0,0 @@
|
|||||||
---
|
|
||||||
dependencies:
|
|
||||||
- { role: murder/common }
|
|
@ -1,7 +0,0 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
- name: run {{ filename }}
|
|
||||||
file: path='{{ filename }}' state=absent
|
|
||||||
tags: rm_tgz
|
|
||||||
|
|
@ -3,6 +3,10 @@
|
|||||||
# The ip address returned by the 'host' bash command will be announced to the tracker.
|
# The ip address returned by the 'host' bash command will be announced to the tracker.
|
||||||
# The server will not stop seeding until the stop_seeding task is called.
|
# The server will not stop seeding until the stop_seeding task is called.
|
||||||
# You must specify a valid 'tag' argument (which identifies the .torrent in /tmp to use)
|
# You must specify a valid 'tag' argument (which identifies the .torrent in /tmp to use)
|
||||||
|
#- name: start seeding
|
||||||
|
# shell: SCREENRC=/dev/null SYSSCREENRC=/dev/null screen -dmS 'seeder-{{ tag }}' python {{ remote_murder_path }}/murder_client.py seeder '/tmp/{{ base_filename }}.torrent' '{{ seeder_files_path }}/{{ base_filename }}' {{ ansible_default_ipv4['address'] }}
|
||||||
|
# tags: start_seeding
|
||||||
|
|
||||||
- name: start seeding
|
- name: start seeding
|
||||||
shell: SCREENRC=/dev/null SYSSCREENRC=/dev/null screen -dmS 'seeder-{{ tag }}' python {{ remote_murder_path }}/murder_client.py seeder '{{ filename }}.torrent' '{{ filename }}' {{ ansible_default_ipv4['address'] }}
|
shell: SCREENRC=/dev/null SYSSCREENRC=/dev/null screen -dmS 'seeder-{{ tag }}' aria2c --console-log-level=notice --peer-id-prefix={{ inventory_hostname[-8:] }} --disable-ipv6 --enable-dht=false --listen-port=6881 --allow-overwrite=true --bt-max-peers=2 --bt-seed-unverified=true --seed-ratio=0.0 -d '{{ seeder_files_path }}' '/tmp/{{ base_filename }}.torrent'
|
||||||
tags: start_seeding
|
tags: start_seeding
|
||||||
|
27
setup.yml
27
setup.yml
@ -1,20 +1,29 @@
|
|||||||
|
- name: Prepare ansible cache
|
||||||
- name: get distribute_files
|
hosts:
|
||||||
hosts: ansible_host
|
- tracker
|
||||||
roles:
|
- seeder
|
||||||
- admin/clone_murder_files
|
- peer
|
||||||
|
sudo: yes
|
||||||
- name: Upload the support files and install pigz
|
roles:
|
||||||
|
- admin/prepare
|
||||||
|
|
||||||
|
- name: Install bittorrent-tracker
|
||||||
|
hosts:
|
||||||
|
- tracker
|
||||||
|
sudo: yes
|
||||||
|
roles:
|
||||||
|
- admin/deploy_bittorrent-tracker
|
||||||
|
|
||||||
|
- name: Install pigz
|
||||||
hosts:
|
hosts:
|
||||||
- tracker
|
- tracker
|
||||||
- seeder
|
- seeder
|
||||||
- peer
|
- peer
|
||||||
sudo: yes
|
sudo: yes
|
||||||
roles:
|
roles:
|
||||||
- admin/distribute_files
|
|
||||||
- admin/pigz
|
- admin/pigz
|
||||||
|
|
||||||
- name: install screen
|
- name: Install screen
|
||||||
hosts:
|
hosts:
|
||||||
- tracker
|
- tracker
|
||||||
- seeder
|
- seeder
|
||||||
|
9
vcsim_vars-example.yml
Normal file
9
vcsim_vars-example.yml
Normal file
@ -0,0 +1,9 @@
|
|||||||
|
vcenter_servers:
|
||||||
|
- SWTTSV1VCSSA.sres.stech
|
||||||
|
- SWTTYV1VCSYA.yres.ytech
|
||||||
|
- SWMUZV1VCSZA.zres.ztech
|
||||||
|
- SWMUZV1VCSZB.zres.ztech
|
||||||
|
- SWMUZV1VCSZC.zres.ztech
|
||||||
|
- SWMUZV1VCSZD.zres.ztech
|
||||||
|
vcenter_user: user
|
||||||
|
vcenter_pass: "password"
|
6
vm_list.yml
Normal file
6
vm_list.yml
Normal file
@ -0,0 +1,6 @@
|
|||||||
|
vm_list:
|
||||||
|
# - HLZURT1EXE02
|
||||||
|
- HLZURT1EXE04
|
||||||
|
- HLZURT1EXE05
|
||||||
|
- HLZURT1EXE06
|
||||||
|
- HLZURT1EXE08
|
Loading…
Reference in New Issue
Block a user