designetwork(EN)

IT technical memo of networking

BOSH-Lite in VirtualBox with Bridged Adapter for remote access

On the 23rd day of Cloud Foundry Advent Calender 2018

Procedures for building BOSH-Lite in the local environment with VirtualBox are explained in some articles. In these, as described in the BOSH Official, you have to manage and access from the host OS. However, running BOSH-Lite on the main PC is resource intensive, it affects other work, or it is troublesome to start the BOSH again when stopping the PC. Also, even if Cloud Foundry etc is built, environment can not be shared and applications can not be published.

Therefore, I changed the network configuration of VirtualBox and built an environment remote access to BOSH-Lite Director on the remote server.

Hardware

Physical server: DELL PowerEdge T110II, Xeon E3 1220, RAM 24GB
Host OS: CentOS 7.5 (vSphere VM), 1vCPU (4 cores), RAM 8GB
PC: MacBookAir 2018, RAM 8GB

Prerequisite environment

Basic flow and prerequisite environment are as described in the official.

Quick Start - Cloud Foundry BOSH

VirtualBox - Cloud Foundry BOSH

Change the VirtualBox network to Bridge Adapter

In the official procedure, build BOSH Director with the configuration of host-only adapter (vboxnet 0) + NAT network (NatNetwork) as follows.

The problem is how to connect from the remote device to the VM behind the host only adapter. As described in the later-described NG example, this problem can not be solved by any method of port-forwarding and routing.

Therefore, I changed the network configuration, share the NIC of the host OS and connect directly to the external NW by the connection method of Bridged Adapter of VirtualBox.

Official compatibility

It is not supported by the official as of 2018/Dec. It will be usable if this branch is merged, but it looks like it is left unattended...

github.com

Some people prepared documents.

github.com

However, it has not been merged, the release file has not been updated...

Clone and create Release

Since the function has already been added in the bridged branch, simply checkout and bosh create release.

git clone > git checkout -b bridged remote/origin/bridged > update blob > fix config/final.yml,config/private.yml > bosh create release --final ...

How to create BOSH Release is as below. Creating a Release - Cloud Foundry BOSH

Deploy BOSH-Lite Director

Deploy BOSH-Lite Director using the created Release file. Deployment of BOSH-Lite Director is need to operate in the host OS which VirtualBox is running.

Replace CPI with Operations-file and use bridged adapter.

* The following URL can be used as it is, but please be noted that it may stop publishing in the future.

ops-files/virtualbox-bridged-network.yml

- type: remove
  path: /releases/name=bosh-virtualbox-cpi

- type: replace
  path: /releases/-
  value:
    name: bosh-virtualbox-cpi
    version: "0.0.13"
    url: https://s3-ap-northeast-1.amazonaws.com/d703n-bosh-virtualbox-blobs/bosh-virtualbox-cpi-release-0.0.13-bridged.tgz
    sha1: beaf2ff627663f7df6a8730c96caf6fd7c0c57bd

- type: replace
  path: /networks/name=default/subnets/0/cloud_properties?
  value:
    type: bridged
    name: ((network_device))

- type: replace
  path: /networks/name=default/subnets/0/dns
  value: [((dns_servers))]

Please change network_device, IP address according to the environment.

deploy-bosh.sh

bosh create-env ~/workspace/bosh-deployment/bosh.yml \
  --state ./state.json \
  -o ~/workspace/bosh-deployment/virtualbox/cpi.yml \
  -o ~/workspace/bosh-deployment/bosh-lite.yml \
  -o ~/workspace/bosh-deployment/uaa.yml \
  -o ~/workspace/bosh-deployment/credhub.yml \
  -o ~/workspace/bosh-deployment/jumpbox-user.yml \
  --vars-store ./creds.yml \
  -o ops-files/virtualbox-bridged-network.yml \
  -v director_name=bosh-lite \
  -v network_device=ens192 \
  -v internal_ip=192.168.1.222 \
  -v internal_gw=192.168.1.5 \
  -v internal_cidr=192.168.1.0/24 \
  -v dns_servers=192.168.1.77

Deploy.

$ ./deploy-bosh.sh
Deployment manifest: '/home/dev/workspace/bosh-deployment/bosh.yml'
Deployment state: './state.json'

Started validating
  Downloading release 'bosh'... 

Started installing CPI
  Compiling package 'golang-1.8-linux/c97f9a00c26b34a3f59ca15b0f5a079d7f7e27c334cc8100248143c5dc0d4c0a'... 

Starting registry... Finished (00:00:00)
Uploading stemcell 'bosh-vsphere-esxi-ubuntu-xenial-go_agent/170.9'... Finished (00:00:24)

Started deploying
  Creating VM for instance 'bosh/0' from stemcell 'sc-261d9c63-8431-41e5-6396-0068ab24cfdf'... 

Finished deploying (00:10:06)

Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs... Finished (00:00:00)

Succeeded

Connect to BOSH Director from remote

From here remote work is possible. Copy creds.yml needed for login to local and load it as an environment variable as follows.

#BOSH-Lite
export BOSH_ENVIRONMENT=192.168.1.222
export BOSH_CA_CERT=`bosh int ~/project/bosh-deployment/creds.yml --path /director_ssl/ca`
export BOSH_CLIENT=admin
export BOSH_CLIENT_SECRET=`bosh int ~/project/bosh-deployment/creds.yml --path /admin_password`

If environment variable is read and a command such as bosh vms responds, connection succeeded.

$ bosh vms
Using environment '192.168.1.222' as client 'admin'

Succeeded

Deploy the Nginx BOSH Release

Since environment was built, try to deploy BOSH Release. Deploy a simple Nginx. (https://github.com/cloudfoundry-community/nginx-release)

Upload Stemcell and Cloud Config as preparation, and download Nginx BOSH Release.

bosh upload-stemcell https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-trusty-go_agent

bosh update-cloud-config ~/project/bosh-deployment/warden/cloud-config.yml

git submodule add https://github.com/cloudfoundry-community/nginx-release

Create an Ops-file for environment tuning.

ops-files / nginx-options.yml

- type: replace
  path: /releases/name=nginx/url?
  value: https://github.com/cloudfoundry-community/nginx-release/releases/download/1.13.12/nginx-release-1.13.12.tgz

- type: replace
  path: /releases/name=nginx/sha1?
  value: c5d767f43a68b89d9603bccac2a60b6a5dc29836

- type: remove
  path: /instance_groups/name=nginx/networks/name=default/static_ips

Deploy Command

deploy-nginx.sh

bosh deploy -d nginx nginx-release/manifests/nginx-lite.yml \
  -o ops-files/nginx-options.yml

Deploy.

$ ./deploy-nginx.sh
Using environment '192.168.1.222' as client 'admin'

Using deployment 'nginx'

Task 2

Task 2 | 17:03:23 | Downloading remote release: Downloading remote release (00:00:04)
Task 2 | ...
Task 2 done
+ azs:
+ - name: z1
...

Continue? [yN]: y


Task 3

Task 3 | 17:03:47 | Preparing deployment: Preparing deployment (00:00:01)
Task 3 | 17:03:48 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 3 | 17:03:48 | Compiling packages: nginx/23c0391f6bb6630cf68ba02c99f93eabdd65839d (00:02:27)
Task 3 | 17:06:17 | Creating missing vms: nginx/bd0e9a7f-6aeb-45e0-b5cf-8647c55bd7c7 (0) (00:00:13)
Task 3 | 17:06:31 | Updating instance nginx: nginx/bd0e9a7f-6aeb-45e0-b5cf-8647c55bd7c7 (0) (canary) (00:00:24)

Task 3 Started  Sun Dec 23 17:03:47 UTC 2018
Task 3 Finished Sun Dec 23 17:06:55 UTC 2018
Task 3 Duration 00:03:08
Task 3 done

Succeeded

$ bosh vms
Using environment '192.168.1.222' as client 'admin'

Task 4. Done

Deployment 'nginx'

Instance                                    Process State  AZ  IPs         VM CID                                VM Type  Active
nginx/bd0e9a7f-6aeb-45e0-b5cf-8647c55bd7c7  running        z1  10.244.0.2  b290ee7a-fa28-4232-500c-54422c6f3eab  default  true

1 vms

Succeeded

Check access

You need to add routing to access the BOSH environment from the remote environment. Mac add the following route as follows. (Note that it will be reset when rebooting)

When sharing with multiple people, it is good to add a static route to the router which is the default gateway of the PCs.

sudo route add -net 10.244.0.0/16 192.168.1.222
$ curl http://10.244.0.2
    <html><head><title>BOSH on IPv6</title>
    </head><body>
    <h2>Welcome to BOSH's nginx Release</h2>
    <h2>
    My hostname/IP: <b>10.244.0.2</b><br />
    Your IP: <b>192.168.1.126</b>
    </h2>
    </body></html>

I got access without problems.

Conclusion - BOSH-Lite in VirtualBox with Bridged Adapter for remote access

Enhanced the bosh-virtualbox-cpi-release to access the BOSH-Lite environment remotely by using the VirtualBox Bridged Adapter. By running BOSH-Lite on a remote server, it is possible to work development of the BOSH Release and the confirmation of operation without influence of stopping the PC and resources etc.

In addition, it is limited to evaluation, it can also be used for public environment to the team.

NG method

I tried the following method, but it did not become expected behavior.

VirtualBox Port-Forwarding

Host OS: 192.168.1.83
Guest OS (BOSH Director): 192.168.50.6
Port-Forwarding: 192.168.1.83:25555 -> 192.168.50.6:25555

Just port forwarding with VirtualBox can not connect with a certificate error as follows.

$ bosh vms
Fetching info:
  Performing request GET 'https://192.168.1.83:25555/info':
    Performing GET request:
      Retry: Get https://192.168.1.83:25555/info: x509: certificate is valid for 192.168.50.6, not 192.168.1.83

Exit code 1

Even though I think that it is sufficient to invalidate the certificate verification, unfortunately verification of the certificate is indispensable in BOSH CLIv2.

stackoverflow.com

Routing with host OS

Set a static route to the subnet (host only adapter) to which BOSH Director belongs on remote PC.

$ sudo route add -net 192.168.50.6/32 192.168.1.83
Password:
add net 192.168.50.6: gateway 192.168.1.83

The route added.

$ netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
default            192.168.1.5        UGSc          165        0     en0
127                127.0.0.1          UCS             0        0     lo0
127.0.0.1          127.0.0.1          UH              1     1750     lo0
...
192.168.1.83       x:x:x:xx:xx:xx     UHLWIi          2      899     en0   1074
...
192.168.50.6/32    192.168.1.83       UGSc            0        0     en0
...
255.255.255.255/32 link#9             UCS             0        0     en0      !

Also add routes to BOSH Director so that return packets will not be NatNetwork.

$ bosh int creds.yml --path /jumpbox_ssh/private_key > jumpbox.key
$ chmod 600 jumpbox.key
$ ssh jumpbox@192.168.50.6 -i jumpbox.key
bosh/0:~# ip r
default via 10.0.2.1 dev eth0
10.0.2.0/24 dev eth0  proto kernel  scope link  src 10.0.2.4
192.168.50.0/24 dev eth1  proto kernel  scope link  src 192.168.50.6

bosh/0:~# route add -net 192.168.1.0/24 dev eth1 gw 192.168.50.1

bosh/0:~# ip r
default via 10.0.2.1 dev eth0
10.0.2.0/24 dev eth0  proto kernel  scope link  src 10.0.2.4
192.168.1.0/24 via 192.168.50.1 dev eth1
192.168.50.0/24 dev eth1  proto kernel  scope link  src 192.168.50.6

In the case of the above setting, although Ping passes, but it can not communicate with TCP. As discussed here, it seems that the host-only adapter is not routed. (It is strange that Ping goes through...)

superuser.com

BOSH guest can not connect to bridge

In BOSH-Lite, guest OS is created with Warden VM, guest network can not extend to VBox bridged adapter.

If the setting of the network in Cloud Config is incorrect, VM communication can not be performed and a deployment error occurs.

Task 6

Task 6 | 18:30:53 | Preparing deployment: Preparing deployment (00:00:01)
Task 6 | 18:30:54 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 6 | 18:30:54 | Compiling packages: nginx/23c0391f6bb6630cf68ba02c99f93eabdd65839d (00:10:48)
                   L Error: Timed out pinging to 38b8f90b-395b-40d0-8610-ca3de241a0bd after 600 seconds
Task 6 | 18:41:42 | Error: Timed out pinging to 38b8f90b-395b-40d0-8610-ca3de241a0bd after 600 seconds

Task 6 Started  Fri Dec 21 18:30:53 UTC 2018
Task 6 Finished Fri Dec 21 18:41:42 UTC 2018
Task 6 Duration 00:10:49
Task 6 error

Updating deployment:
  Expected task '6' to succeed but state is 'error'

Exit code 1

This Blog is English Version of my JP's.

Sorry if my English sentences are incorrect.

designetwork