designetwork(EN)

IT technical memo of networking

Install BOSH on OpenStack

Platform Platform-OpenStack Platform-BOSH

Basically install BOSH on OpenStack as described here.

https://bosh.io/docs/init-openstack.html

Although it is possible to work with CentOS that is used as a host of OpenStack, in my environment using PackStack, did not work well because of OpenSSL dependency problem. Therefore, CentOS 7 Minimal is separately created and working.

* Please note that this procedure is implemented in the local environment, and issues such as security remain.

Installing OpenStack

Create standalone OpenStack environment using PackStack.

See here for OpenStack installation instructions.

en-designetwork.hatenablog.com

Since the resources are insufficient, components to be installed are reduced as follows.

# cat ./answers.cfg | grep INSTALL
CONFIG_MARIADB_INSTALL=y
CONFIG_GLANCE_INSTALL=y
CONFIG_CINDER_INSTALL=y
CONFIG_MANILA_INSTALL=n
CONFIG_NOVA_INSTALL=y
CONFIG_NEUTRON_INSTALL=y
CONFIG_HORIZON_INSTALL=y
CONFIG_SWIFT_INSTALL=n
CONFIG_CEILOMETER_INSTALL=n
CONFIG_AODH_INSTALL=n
CONFIG_PANKO_INSTALL=n
CONFIG_SAHARA_INSTALL=n
CONFIG_HEAT_INSTALL=n
CONFIG_MAGNUM_INSTALL=n
CONFIG_TROVE_INSTALL=n
CONFIG_IRONIC_INSTALL=n
CONFIG_CLIENT_INSTALL=y
# installation was not specified in CONFIG_MARIADB_INSTALL, specify
CONFIG_LBAAS_INSTALL=n
CONFIG_NEUTRON_METERING_AGENT_INSTALL=n
CONFIG_HEAT_CFN_INSTALL=n

Preparation

Execution environment, use CentOS 7 Minimal.

Download BOSH CLI

https://bosh.io/docs/cli-v2.html

* Please download the latest version

$ curl -OL https://s3.amazonaws.com/bosh-cli-artifacts/bosh-cli-3.0.1-linux-amd64
$ chmod +x ./bosh-cli-3.0.1-linux-amd64
$ sudo mv ./bosh-cli-3.0.1-linux-amd64 /usr/local/bin/bosh
$ bosh -v
version 3.0.1-712bfd7-2018-03-13T23:26:43Z

Succeeded

Yum package installation

Install packages required for compiling various packages. If the package is insufficient, the following error occurs.

Dependent packages are also listed here.
https://bosh.io/docs/cli-env-deps.html

(BOSH Docs)
$ sudo yum install -y gcc gcc-c++ ruby ruby-devel mysql-devel postgresql-devel postgresql-libs sqlite-devel libxslt-devel libxml2-devel patch openssl
$ gem install yajl-ruby

(Option)
$ sudo yum install -y git libtool zlib-devel openssl-devel

Also install the OpenStack CLI.

$ sudo yum install -y epel-release python-devel python-pip
$ sudo pip install pip --upgrade setuptools
$ sudo pip install python-openstackclient

Deploying BOSH Director

Deploy BOSH Director as follows.

Network diagram

Create the following network. Detailed content is as described in the BOSH document.

Step 1: Prepare an OpenStack environment

Security rules are as follows. ICMP and others are added according to the environment.

$ cat ./create_security_group.sh
#!/bin/sh
openstack security group create bosh
openstack security group rule create bosh --ingress --protocol tcp --dst-port 22:22 --remote-ip 0.0.0.0/0
openstack security group rule create bosh --ingress --protocol tcp --dst-port 6868:6868 --remote-ip 0.0.0.0/0
openstack security group rule create bosh --ingress --protocol tcp --dst-port 25555:25555 --remote-ip 0.0.0.0/0
openstack security group rule create bosh --egress --ethertype IPv4 --remote-ip 0.0.0.0/0
openstack security group rule create bosh --egress --ethertype IPv6 --remote-ip ::/0
openstack security group rule create bosh --ingress --protocol tcp --dst-port 1:65535 --remote-group bosh

Download bosh-deployment

$ mkdir bosh && cd bosh
$ git init
$ git submodule add https://github.com/cloudfoundry/bosh-deployment

Downloading bosh-deployment as a Git submodule makes it easier to keep track of official updates.

Check auth_url & authentication information

OpenStack credentials (authentication information) are generated in the home directory of the server that installed OpenStack with PackStack.

[root@openstack1 ~]# cat keystonerc_admin
unset OS_SERVICE_TOKEN
    export OS_USERNAME=admin
    export OS_PASSWORD='PASSWORD'
    export OS_AUTH_URL=http://192.168.1.202:5000/v3
    export PS1='[\u@\h \W(keystone_admin)]\$ '

export OS_PROJECT_NAME=admin
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3

For keystone API v2 check here
https://bosh.io/docs/openstack-keystonev2.html

Each parameter varies depending on the environment, but it is described as it is to ensure consistency within the article.

Deploy command to shell

Commands including arguments are convenient to be stored in a shell script file so that changes can be tracked.

$ touch deploy_bosh.sh
$ chmod +x ./deploy_bosh.sh

Edit Deploy Shell

$ vi deploy_bosh.sh
bosh create-env bosh-deployment/bosh.yml \
    --state=state.json \
    --vars-store=creds.yml \
    -o bosh-deployment/openstack/cpi.yml \
    -o bosh-deployment/external-ip-with-registry-not-recommended.yml \
    -v director_name=bosh-1 \
    -v internal_cidr=100.64.0.0/24 \
    -v internal_gw=100.64.0.1 \
    -v internal_ip=100.64.0.5 \
    -v external_ip=192.168.1.214 \
    -v auth_url=http://192.168.1.202:5000/v3 \
    -v az=nova \
    -v default_key_name=bosh \
    -v default_security_groups=[bosh] \
    -v net_id=1cdfa900-dfbb-4f95-8571-854bee863e69 \
    --vars-file=openstack_creds.yml \
    -v openstack_domain=Default \
    -v openstack_project=admin \
    -v private_key=../id_rsa_bosh.pem \
    -v region=RegionOne \
    -o operations_file/vm_size.yml \  #option
    -o operations_file/worker_instances.yml  #option

net_id is not a network name, it must be specified by ID.

By write OpenStack's authentication information in a separate file, it can be excluded and prevented from flowing out when managing Git.

$ cat openstack_creds.yml
openstack_username: admin
openstack_password: 'PASSWORD'

(Optional) If it is the default, the instance's Flavor is m1.xlarge, but my environment overwrites the setting in the operations file because of lack of resources. This overwrites the contents described in bosh-deployment/openstack/cpi.yml.

$ cat operations_file/vm_size.yml
- type: replace
  path: /resource_pools/name=vms/cloud_properties?
  value:
    instance_type: m1.medium
    availability_zone: ((az))

Also reduce the number of instances of Worker from the default of 4 to 1 to save resources.

$ cat ./operations_file/worker_instances.yml
- type: replace
  path: /instance_groups/name=bosh/properties/director/workers
  value: 1

Deploy

Run Deploy Shell. It took me more than 90 minutes in my environment .... It takes time to compile ruby_openstack_cpi if the VM is small ....

$ ./deploy_bosh.sh
Deployment manifest: '/root/bosh/bosh-deployment/bosh.yml'
Deployment state: 'state.json'

Started validating
  Downloading release 'bosh'... Finished (00:07:40)
  Validating release 'bosh'... Finished (00:00:01)
  Downloading release 'bosh-openstack-cpi'... Finished (00:00:05)
  Validating release 'bosh-openstack-cpi'... Finished (00:00:00)
  Validating cpi release... Finished (00:00:00)
  Validating deployment manifest... Finished (00:00:00)
  Downloading stemcell... Finished (00:30:56)
  Validating stemcell... Finished (00:00:04)
Finished validating (00:38:51)

Started installing CPI
  Compiling package 'ruby_openstack_cpi/ddb0f4a3013923fb1b074454d4314264c47d33c5'... Finished (00:00:00)
  Compiling package 'bosh_openstack_cpi/f4e7f49b87ef90a8c0186602cad189c00bd508ca'... Finished (00:00:00)
  Installing packages... Finished (00:00:03)
  Rendering job templates... Finished (00:00:00)
  Installing job 'openstack_cpi'... Finished (00:00:00)
Finished installing CPI (00:00:03)

Starting registry... Finished (00:00:00)
Uploading stemcell 'bosh-openstack-kvm-ubuntu-trusty-go_agent/3468.21'... Skipped [Stemcell already uploaded] (00:00:00)

Started deploying
  Creating VM for instance 'bosh/0' from stemcell '7c81c288-593b-4b3c-8d62-89037a5de635'... Finished (00:02:04)
  Waiting for the agent on VM '47382065-ac7e-4703-a7f6-edae677f46cf' to be ready... Finished (00:04:23)
  Creating disk... Finished (00:00:08)
  Attaching disk '29f04e0c-0f13-45d8-809f-525272023100' to VM '47382065-ac7e-4703-a7f6-edae677f46cf'... Finished (00:00:59)
  Rendering job templates... Finished (00:00:03)
  Compiling package 'ruby_openstack_cpi/ddb0f4a3013923fb1b074454d4314264c47d33c5'... Finished (01:37:04)
  Compiling package 'ruby-2.4-r3/8471dec5da9ecc321686b8990a5ad2cc84529254'... Skipped [Package already compiled] (00:00:02)
  Compiling package 'mysql/b7e73acc0bfe05f1c6cbfd97bf92d39b0d3155d5'... Skipped [Package already compiled] (00:00:02)
  Compiling package 'libpq/3afb51e921e950abb31e5d039d2144591a41482d'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'postgres/3b1089109c074984577a0bac1b38018d7a2890ef'... Skipped [Package already compiled] (00:00:03)
  Compiling package 'bosh_openstack_cpi/f4e7f49b87ef90a8c0186602cad189c00bd508ca'... Finished (00:01:41)
  Compiling package 'registry/2231e6d61fb5a36afafad82ea6ff49f83334c9bb'... Skipped [Package already compiled] (00:00:05)
  Compiling package 'nginx/3518a530de39c41ec65abf1194c27aadae23b711'... Skipped [Package already compiled] (00:00:01)
  Compiling package 'bosh-gcscli/fce60f2d82653ea7e08c768f077c9c4a738d0c39'... Skipped [Package already compiled] (00:00:01)
  Compiling package 'postgres-9.4/52b3a31d7b0282d342aa7a0d62d8b419358c6b6b'... Skipped [Package already compiled] (00:00:02)
  Compiling package 'davcli/2672d0a96a775f5252fef6ac7bbab3928aa41599'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'verify_multidigest/8fc5d654cebad7725c34bb08b3f60b912db7094a'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'director/06635593362c742ed6027270d6fbe0ddd8439650'... Skipped [Package already compiled] (00:00:07)
  Compiling package 'gonats/866cdc573ac10dd85929abb531923197486ffa95'... Skipped [Package already compiled] (00:00:01)
  Compiling package 's3cli/b6e38c619dd5575e16ea9fcabc4b7c500effdd26'... Skipped [Package already compiled] (00:00:01)
  Compiling package 'health_monitor/81dd0f6b874d009696027d43282893df4c18b2c8'... Skipped [Package already compiled] (00:00:04)
  Updating instance 'bosh/0'... Finished (00:02:07)
  Waiting for instance 'bosh/0' to be running... Failed (01:07:13)
Failed deploying (02:56:28)

Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs... Finished (00:00:00)

Deploying:
  Received non-running job state: 'failing'

Exit code 1

Although it finally became an error, but it was able to confirm the connection as follows.

Connection confirmation

Make sure you can access BOSH Director with bosh command. If the result comes back as follows OK.

$ bosh alias-env bosh -e 192.168.1.214 --ca-cert <(bosh int creds.yml --path /director_ssl/ca)
Using environment '192.168.1.214' as client 'admin'

Name      bosh-1
UUID      ca43266f-fee1-4627-a96c-f080cc2eab91
Version   264.7.0 (00000000)
CPI       openstack_cpi
Features  compiled_package_cache: disabled
          config_server: disabled
          dns: disabled
          snapshots: disabled
User      admin

Succeeded

Deploy with BOSH

I will continue to validate ...

Conclusion - Install BOSH on OpenStack

I built a BOSH environment in OpenStack installed with PackStack. Resource shortage is remarkable, but we will continue to verify how far we can use it.

Trouble shooting

private_key no such file or directory

private_key needs to be specified starting from the yml file to be deployed. Since it is stored in the bosh directory this time, it is necessary to designate the file as one level above.

Deployment manifest: '/home/dev/bosh/bosh-deployment/bosh.yml'
Deployment state: 'state.json'

Started validating
Failed validating (00:00:02)

Parsing installation manifest '/home/dev/bosh/bosh-deployment/bosh.yml':
  Reading private key from /home/dev/bosh/bosh-deployment/bosh:
    Opening file /home/dev/bosh/bosh-deployment/bosh:
      open /home/dev/bosh/bosh-deployment/bosh: no such file or directory

Exit code 1

Not enough DISK capacity

$ ./deploy_bosh.sh
Deployment manifest: '/home/dev/bosh/bosh-deployment/bosh.yml'
Deployment state: 'state.json'

Started validating
  Downloading release 'bosh'... Finished (00:07:51)
  Validating release 'bosh'... Finished (00:00:06)
  Downloading release 'bosh-openstack-cpi'... Finished (00:00:09)
  Validating release 'bosh-openstack-cpi'... Finished (00:00:02)
  Validating cpi release... Finished (00:00:00)
  Validating deployment manifest... Finished (00:00:02)
  Downloading stemcell...
 Finished (00:31:36)
  Validating stemcell... Failed (00:00:14)
Failed validating (00:40:03)

Extracting stemcell from '/home/dev/.bosh/downloads/301bb47087fe66403788c852a401f42006426be9-f08706560b67b50654998e46038b6adacc8a4b46':
  reading extracted stemcell manifest in '/home/dev/.bosh/installations/0a6fbed5-08af-4e7b-7e1f-27965a2522ac/tmp/stemcell-manager107784699':
    Extracting stemcell from '/home/dev/.bosh/downloads/301bb47087fe66403788c852a401f42006426be9-f08706560b67b50654998e46038b6adacc8a4b46' to '/home/dev/.bosh/installations/0a6fbed5-08af-4e7b-7e1f-27965a2522ac/tmp/stemcell-manager107784699':
      Shelling out to tar:
        Running command: 'tar --no-same-owner -xzf /home/dev/.bosh/downloads/301bb47087fe66403788c852a401f42006426be9-f08706560b67b50654998e46038b6adacc8a4b46 -C /home/dev/.bosh/installations/0a6fbed5-08af-4e7b-7e1f-27965a2522ac/tmp/stemcell-manager107784699', stdout: '', stderr: 'tar: image: 10240 バイトのうち、9216 バイトのみ書き込みました
tar: stemcell.MF: open 不能: デバイスに空き領域がありません
tar: stemcell_dpkg_l.txt: open 不能: デバイスに空き領域がありません
tar: 前のエラーにより失敗ステータスで終了します
':
          exit status 2

Exit code 1

Workaround: Increase the DISK of the OpenStack host. Even if there is room in the Cinder volume, since the limit of the capacity of the root becomes the limit, also increase the root.

Package requirement

If the following error occurs, check the Yum installation in the article. In addition, there are cases where reading can not be done well even if Yum is installed on a server running OpenStack. As a workaround, prepare a server like BOSH Deploy separately.

Basically install additional packages that are checking xxx ... no .

[root@openstack1 bosh]# ./deploy_bosh.sh
Deployment manifest: '/root/bosh/bosh-deployment/bosh.yml'
Deployment state: 'state.json'

Started validating
  Downloading release 'bosh'... Finished (00:07:40)
  Validating release 'bosh'... Finished (00:00:01)
  Downloading release 'bosh-openstack-cpi'... Finished (00:00:05)
  Validating release 'bosh-openstack-cpi'... Finished (00:00:00)
  Validating cpi release... Finished (00:00:00)
  Validating deployment manifest... Finished (00:00:00)
  Downloading stemcell... Finished (00:30:56)
  Validating stemcell... Finished (00:00:04)
Finished validating (00:38:51)

Started installing CPI
  Compiling package 'ruby_openstack_cpi/ddb0f4a3013923fb1b074454d4314264c47d33c5'... Failed (00:00:01)
Failed installing CPI (00:00:01)

Installing CPI:
  Compiling job package dependencies for installation:
    Compiling job package dependencies:
      Compiling package:
        Running command: 'bash -x packaging', stdout: 'Installing yaml
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for gcc... no
checking for cc... no
checking for cl.exe... no
', stderr: '+ set -e -x
...
+ ./configure --prefix=/root/.bosh/installations/05c8688b-3c66-4365-7a4a-d0d7eee3902c/packages/ruby_openstack_cpi --disable-shared
configure: error: in `/root/.bosh/installations/05c8688b-3c66-4365-7a4a-d0d7eee3902c/tmp/bosh-release-pkg026788788/yaml-0.1.7':
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details
':
          exit status 1

Exit code 1
[root@openstack1 bosh]#
+ echo 'Installing rubygems'
+ tar zxvf ruby_openstack_cpi/rubygems-2.7.3.tar.gz
+ pushd rubygems-2.7.3
+ /root/.bosh/installations/05c8688b-3c66-4365-7a4a-d0d7eee3902c/packages/ruby_openstack_cpi/bin/ruby setup.rb
/root/.bosh/installations/05c8688b-3c66-4365-7a4a-d0d7eee3902c/tmp/bosh-release-pkg077630227/rubygems-2.7.3/lib/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- zlib (LoadError)
creating stemcell (bosh-openstack-kvm-ubuntu-trusty-go_agent 3468.21):
  Executing external CPI command: '/root/.bosh/installations/05c8688b-3c66-4365-7a4a-d0d7eee3902c/jobs/openstack_cpi/bin/cpi':
    Running command: '/root/.bosh/installations/05c8688b-3c66-4365-7a4a-d0d7eee3902c/jobs/openstack_cpi/bin/cpi', stdout: 'bundler: failed to load command: /root/.bosh/installations/05c8688b-3c66-4365-7a4a-d0d7eee3902c/packages/bosh_openstack_cpi/bin/openstack_cpi (/root/.bosh/installations/05c8688b-3c66-4365-7a4a-d0d7eee3902c/packages/bosh_openstack_cpi/bin/openstack_cpi)
', stderr: 'LoadError: cannot load such file -- openssl

Process multiple activation?

Waiting for instance 'bosh/0' to be running... did not complete, so I confirmed SSH to the instance and it seemed that the same process started multiple times. CPU processing conflicted and seemed to be inefficient, so I left one and killed. The influence is unknown.

$ ssh vcap@192.168.1.214 -i ./id_rsa_bosh.pem
bosh/0:~$ ps auxf
<snip>
root     21166  0.0  0.0  19616  3308 ?        S<   06:12   0:00 /bin/bash /var/vcap/jobs/director/bin/director_ctl start
vcap     21251 11.3  1.1 142288 47684 ?        R<l  06:14   2:06  \_ ruby /var/vcap/packages/director/bin/bosh-director-migrate -c /var/vcap/jobs/director/config/director.yml
root     21238  0.0  0.0  19616  3316 ?        S<   06:14   0:00 /bin/bash /var/vcap/jobs/director/bin/director_ctl start
vcap     21318 11.3  1.1 140184 45452 ?        R<l  06:15   1:57  \_ ruby /var/vcap/packages/director/bin/bosh-director-migrate -c /var/vcap/jobs/director/config/director.yml
root     21335  0.0  0.0  19616  3372 ?        S<   06:15   0:00 /bin/bash /var/vcap/jobs/director/bin/director_ctl start
vcap     21402 11.3  1.0 136236 43620 ?        R<l  06:17   1:47  \_ ruby /var/vcap/packages/director/bin/bosh-director-migrate -c /var/vcap/jobs/director/config/director.yml
vcap     21357 10.5  0.9 132160 39820 ?        R<l  06:16   1:44 ruby /var/vcap/packages/director/bin/bosh-director-worker -c /var/vcap/jobs/director/config/worker_1.yml -i 1

bosh/0:~$ kill -9 21402 21318

daichi703n

Related Entry
follow us in feedly
Write a comment

This Blog is English Version of my JP's.

Sorry if my English sentences are incorrect.

designetwork