Forem: Sergey

Transible — represents cloud configuration as Ansible playbooks

Sergey — Mon, 02 Dec 2019 16:35:50 +0000

What is Transible

Short description

This tool takes your current cloud configuration and represents it as Ansible playbooks. Its repository on GitHub is: https://github.com/sshnaidm/transible

More details

It takes servers, security rules, images, volumes and all other things your cloud includes and defines their config as tasks in Ansible playbooks that are ready for deployment. By running these playbooks you actually can deploy or redeploy your current cloud.
Simple run:

git clone https://github.com/sshnaidm/transible
cd transible
./transible.py --os-cloud your_cloud_name

Why would anybody redeploy its existing cloud?

Use-cases of Transible

Freezing cloud config for further IaaC management

There are cases when you have a configured cloud with a lot of servers, networks,
images, etc. But sometimes we don't have this properly managed with
Infrastructure as Code principles and cloud maintenance turns to be a nightmare.
Hundreds of legacy servers with unknown configurations, complex network setups that nobody knows who made them and for what purpose. Just collecting information from the cloud can be a complex task when it's under load and things change quickly. Removing legacy and things that look unused can cause breakages in the work of others, while it's almost impossible to rollback when it turns out this deleted server is in usage and very important for other teams.
It's pretty difficult and sometimes even impossible manually to create IaaC configs for existing infrastructure that can include hundreds of servers or more, Transible comes to solve this problem.

Transible can take your cloud and convert everything in it to Ansible playbooks. That way you have all your servers, networks, images and others in one place, ready for maintenance and management. Since you have it in your deployment tool, all next changes can come via Git and keeping IaaC principles. By running ansible playbooks you can add, remove or change your cloud config.

Keeping current cloud configuration

There are various use-cases of testing on clouds and sometimes it's required to keep the most important resources untouched. Dumped configuration of desired state and all mandatory resources will help to ensure you have all you need before starting the next testing tasks. Just run Transible and get you config dumped in Ansible playbooks. You can edit them and leave there only important things. Executing these playbooks before every cloud work will restore any lacking resources.
Any time you want to "snapshot" your cloud in the current state, just run Transible and using generated playbooks you can return this config later.

Moving current infrastructure to another tenant, cloud, provider, whatever

For example, you need to move your current infrastructure to another cloud in a different hosting provider. In case you don’t have everything in Git or some un-tracked changes were made and you don’t want to lose them. Converting the current cloud to Ansible will freeze cloud config and you can run them with different cloud credentials to deploy everything in a new place.

Release vendor lock

In the next Transible versions it will be possible to convert one cloud configuration to another, for example, Openstack to AWS, or Azure to GCP. This will allow us to prevent hard vendor-lock and help Cloud Ops to maintain and move their infrastructure in more convenient ways.

How it works

Currently, it supports the Openstack cloud only, while others are under development. Using openstacksdk, it requests information from the cloud about current resources. After that using templates of Ansible Openstack modules it creates the playbooks configuration.

Although what if we have hundreds of resources or even more? Having 300 server configurations in a playbook that can end up with 10 kLOCs definitely won't help to maintenance and simplicity. For that Transible knows to optimize cloud resources using the loop cycles of Ansible.
All required data can be separated from playbooks and saved in vars/ folder where can be managed and changed separately, without touching the actual playbooks.
For example images playbook for Openstack can look like:

---
- os_image:
    name: '{{ name | default(omit) }}'
    owner: '{{ owner | default(omit) }}'
    filename: '{{ filename | default(omit) }}'
    min_disk: '{{ min_disk | default(omit) }}'
    disk_format: '{{ disk_format | default(omit) }}'
    properties: '{{ properties | default(omit) }}'
    cloud: my-cloud
    checksum: '{{ checksum | default(omit) }}'
    min_ram: '{{ min_ram | default(omit) }}'
    is_public: '{{ is_public | default(omit) }}'
    state: present
  loop: '{{ images }}'

While images config in vars/ folder includes actual images:

images:
  - name: new_image
    owner: ....
    min_disk: 20
    is_public: false
    filename: /v2/images/...hash.../file
  - name: cloud-image
    checksum: ....
    owner: ....
    min_disk: 8
    is_public: false
    filename: /v2/images/...hash.../file

Parameters that are not defined in variables config will be omitted. So that we'll have the same playbook if weren't optimizing it. But anyway vars optimizations is configurable for any resource type separately in a plugin configuration file:

# Variables optimization configuration
VARS_OPT_NETWORKS = False
VARS_OPT_SUBNETS = True
...

There various plugin configurations that will differ for different user scenarios like management of existing cloud, moving to other providers or deployment from scratch. Some of them can't be "guessed" from cloud configuration which doesn't allow to know what was the initial configuration for the deployment. You can tweak and tune these parameters according to your needs.
Let's take an example of moving to a different Openstack cloud or provider.

Use-case when moving to a new cloud or recreating from scratch

In case you want to recreate your boot volumes and boot servers from them, you will likely configure:

CREATE_NEW_BOOT_VOLUMES = True
USE_SERVER_IMAGES = False
SKIP_UNNAMED_VOLUMES = False
USE_EXISTING_BOOT_VOLUMES = False

This will recreate all required boot and non-boot volumes on a new cloud.
Most likely in the new cloud you'll have new IP pools from your provider, in this case you can't keep the same floating IPs you had before:

FIP_AUTO = True

But in case you'll have the same floating IP range and would like to keep all floating IPs as they are:

FIP_AUTO = False

For DHCP in subnets you can allow the server to have any IP they get from DHCP server:

NETWORK_AUTO = True

But if you want to keep the same IPs exactly in networks, configure:

NETWORK_AUTO = False

Use-case when moving to a new cloud or recreating from scratch

In case you want to rerun your playbook on the existing environment and continue to use it, most likely you don't want to recreate anything. Then your configuration will look like that:

CREATE_NEW_BOOT_VOLUMES = False
SKIP_UNNAMED_VOLUMES = True
USE_EXISTING_BOOT_VOLUMES = True
NETWORK_AUTO = False
FIP_AUTO = True

That way the playbook running will end not changing anything and keeping idempotency. (Be aware that not all Openstack modules are idempotent or support --check run.)
In network configuration you'll have the same IPs in internal networks and subnets are they are today. Although we set FIP_AUTO = True it doesn't mean that the server will get a different floating IP because they already have one, so it will change nothing for them.

Road map

Write tests!
Create playbooks for user roles, user groups, other openstack services.
Create a plugin for Kubernetes.
Create a plugin for AWS
Create a translator from Openstack to AWS and from AWS to Openstack.

More thoughts and ideas? Please post in comments on issues of the Transible project.

Speed up Ansible with Mitogen!

Sergey — Sun, 26 May 2019 13:25:17 +0000

Speed up Ansible with Mitogen!

Ansible is one of most popular Configuration Management Systems nowadays, after it was acquired by Red Hat in 2015 Ansible has reached numbers of thousands of contributors and became maybe one of most used deployment and orchestration tools. Its use-cases are quite impressive.
Ansible works by SSH connections to remote hosts. It opens SSH session, logs in to the shell, copy python code via network and create a temporary file on remote hosts with this code. In the next step, it executes the current file with python interpreter. All this workflow is pretty heavy and there are multiple ways to make it faster and lighter.

One of these ways is using SSH pipelines which reuses one SSH session for copying python code of multiple tasks and prevent opening multiple sessions, which saves a lot of time. (Just don’t forget to disable requiretty settings for sudo on the remote side in /etc/sudoers)

The new way to speed up Ansible is a great python library called Mitogen. If somebody like me was not familiar with it — this library allows fast execution of python code on a remote host and Ansible is only one of its cases. Mitogen uses UNIX pipes on remote machines while passing "pickled" python code compressed with zlib. This allows to run it fast and without much traffic. If you're interested you can read details about how it works in its "How it works" page. But we'll focus today on Ansible related part of it.
Mitogen in specific circumstances can speed up your Ansible in a few times and significantly lower your bandwidth. Let's check the most popular use cases and figure out if it's helpful for us.

The most popular use cases for me to run Ansible are: creating configuration files on a remote host, packages installation, downloading and uploading files from and to a remote host. Maybe you want to check other use cases, please leave comments to this article.

Let's start rolling!
Configuring Mitogen for Ansible is pretty simple:
Install the Mitogen module:

pip install mitogen

Then either configure environment variables or set configuration options in ansible.cfg file, both options are fine:
Let's assume /usr/lib/python2.7/site-packages/ansible_mitogen/plugins/strategy is your path to installed Mitogen library.

export ANSIBLE_STRATEGY_PLUGINS=/usr/lib/python2.7/site-packages/ansible_mitogen/plugins/strategy
export ANSIBLE_STRATEGY=mitogen_linear

[defaults]
strategy = mitogen_linear
strategy_plugins = /usr/lib/python2.7/site-packages/ansible_mitogen/plugins/strategy

Prepare Ansible in virtualenv, with and without Mitogen enabled:

virtualenv mitogen_ansible
./mitogen_ansible/bin/pip install ansible==2.7.10 mitogen
virtualenv pure_ansible
./pure_ansible/bin/pip install ansible==2.7.10

Please pay attention that Mitogen 0.2.7 doesn't work with Ansible 2.8 (for May 2019)

Create aliases:

alias pure-ansible-playbook='$(pwd)/pure_ansible/bin/ansible-playbook'
alias mitogen-ansible-playbook='ANSIBLE_STRATEGY_PLUGINS=/usr/lib/python2.7/site-packages/ansible_mitogen/plugins/strategy:$(pwd)/mitogen_ansible/lib/python3.7/site-packages/ansible_mitogen/plugins/strategy ANSIBLE_STRATEGY=mitogen_linear $(pwd)/mitogen_ansible/bin/ansible-playbook'

Now let's try the playbook that creates file on remote:

---
- hosts: all
  gather_facts: false
  tasks:
    - name: Create files with copy content module
      copy:
        content: |
          test file {{ item }}
        dest: ~/file_{{item}}
      with_sequence: start=1 end={{ n }}

And run it with Mitogen and without while creating 10 files:

time mitogen-ansible-playbook file_creation.yml -i hosts -e n=10 &>/dev/null

real    0m2.603s
user    0m1.152s
sys     0m0.096s

time pure-ansible-playbook file_creation.yml -i hosts -e n=10 &>/dev/null

real    0m5.908s
user    0m1.745s
sys     0m0.643s

Right now we see improvement in x2 times. Let's check it for 20, 30, ..., 100 files:

time pure-ansible-playbook file_creation.yml -i hosts -e n=100 &>/dev/null

real    0m51.775s
user    0m8.039s
sys     0m6.305s

time mitogen-ansible-playbook file_creation.yml -i hosts -e n=100 &>/dev/null

real    0m4.331s
user    0m1.903s
sys     0m0.197s

Eventually, we improved execution time in more than 10 times!

Now let's try different scenarios and see how it improves:

Scenario of uploading files from the local host to remote (with copy module):
Scenario of creating files on the remote host with copy module:
Scenario with fetching files from the remote host to local:

Let's try the last scenario on a few (3) remote hosts, for example uploading files scenario:

As we can see the Mitogen saves us both time and bandwidth in these scenarios. But if the bottleneck is not Ansible,
but for example I/O of disk or network, or somewhere else, then it's hard to expect from Mitogen to help of course.
Let's run for example packages installation with yum/dnf and python modules installation with pip.
Packages were pre-cached to avoid dependencies on network glitches:

---
- hosts: all
  gather_facts: false
  tasks:
    - name: Install packages
      become: true
      package:
        name:
          - samba
          - httpd
          - nano
          - ruby
        state: present

    - name: Install pip modules
      become: true
      pip:
        name:
          - pytest-split-tests
          - bottle
          - pep8
          - flask
        state: present

With Mitogen it takes 12 seconds, as well as with pure Ansible.
In Mitogen for Ansible page you can see additional benchmarks
and measurements. As the page declares:

Mitogen cannot improve a module once it is executing, it can only ensure the module executes as quickly as possible

That's why it's important to find where your bottlenecks are and if they are related to Ansible operations, Mitogen will
help you to solve it and speed your playbooks up significantly.

Docker container for HP servers management with ILO

Sergey — Tue, 21 May 2019 14:35:29 +0000

Well, you can wonder — why would I use docker container for such a purpose? What’s the problem to enter web-interface of ILO and manage server as usual?
The same thought I had when I’ve got a few old servers that required a reprovision. The servers are located in different continent and the only interface I had it was just a web interface of ILO. And when I had to enter a few manual commands via Virtual Console I discovered that it’s hardly possible.

For various sorts of Virtual Console of servers (both HP and Dells) usually Java web applets are used. But Firefox and Chrome don’t support them anymore and the newest IcedTea doesn’t work with those old system anyway. So I had a few options:

To install in parallel old versions of browsers and Java, trying to find a required combination, on my system. This option was filtered out, since I don’t want to pollute my system just because of few console commands.
Create a virtual machine with old systems, install there Java 6 and use Virtual Console as before.
The same as in point 2, but with container, not a virtual machine. Since a few my colleagues hit the same problem, I’d prefer to pass them one bash command to run Virtual Console instead of sharing Virtual Machine disk, passwords for it, etc etc. (To be honest, point 3 I made only after point 2). Point 3 is what we are going to implement today.

I’ve been inspired mostly by these two projects:

Actually, the first project docker-baseimage-gui contains already all needed configs and tools to start desktop apps in browser within a container. Usually you define specific environment variables and your app will become accessible via browser (websocket) or VNC. In our case we start with Firefox and VNC, websocket didn’t work well.

Firstly, let’s install required packages: Java 6 and IcedTea:



RUN echo “deb http://archive.ubuntu.com/ubuntu precise main universe” > /etc/apt/sources.list && \
    apt-get update && \
    apt-get -y upgrade && \
    apt-get -y install firefox \
    nano curl \
    icedtea-6-plugin \
    icedtea-netx \
    openjdk-6-jre \
    openjdk-6-jre-headless \
    tzdata-java

Now let’s open the web page of ILO interface in Firefox and enter credentials there. Start Firefox:



RUN bash -c ‘echo “exec openbox-session &” >> ~/.xinitrc’ && \
    bash -c ‘echo “firefox \${HILO_HOST}”>> ~/.xinitrc’ && \
    bash -c ‘chmod 755 ~/.xinitrc’

Variable HILO_HOST is URL of our ILO interface, for example https://myhp.example.com.

For automation let’s add authentication. ILO login is executed via simple POST request, in response you get session_key value and then pass this value in GET request. Let’s discover session_key with curl if environment variables HILO_USER and HILO_PASS are defined:



export HOME=/config
export HILO_HOST=${HILO_HOST%%/}
SESSION_KEY=””
data=”{\”method\”:\”login\”,\”user_login\”:\”${HILO_USER}\”,\”password\”:\”${HILO_PASS}\”}”
if [[ -n “${HILO_USER}” && -n “${HILO_PASS}” ]]; then
 SESSION_KEY=$(curl -k -X POST “${HILO_HOST}/json/login_session” -d “$data” 2>/dev/null | grep -Eo ‘“session_key”:”[^”]+’ | sed ‘s/”session_key”:”//’)
fi
echo “SESSION_KEY=$SESSION_KEY”
echo $SESSION_KEY > /session_key

After we wrote session_key in containers we can start VNC server:



exec x11vnc -forever -create

Now just connect with VNC client to port 5900 (or what you defined in your choice) to localhost and enter the Virtual Console of HP server.
The code is located in git repository docker-ilo-client.

Full one line command to connect to ILO Virtual Console:



docker run -d --rm --name ilo-client \
    -p 5900:5900 \
    -e HILO_HOST=https://ADDRESS_OF_YOUR_HOST \
    -e HILO_USER=SOME_USERNAME \
    -e HILO_PASS=SOME_PASSWORD \
    sshnaidm/docker-ilo-client

where ADDRESS_OF_YOUR_HOST is hostname of ILO, SOME_USERNAME is login and SOME_PASSWORD is password for ILO.

Next just go with any VNC client to address vnc://localhost:5900.
Pull requests and comments are more than welcome.

The similar project for connection to Dell IDRAC servers is here: docker-idrac6.