On the 23rd day of Cloud Foundry Advent Calender 2018
Procedures for building BOSH-Lite in the local environment with VirtualBox are explained in some articles. In these, as described in the BOSH Official, you have to manage and access from the host OS. However, running BOSH-Lite on the main PC is resource intensive, it affects other work, or it is troublesome to start the BOSH again when stopping the PC. Also, even if Cloud Foundry etc is built, environment can not be shared and applications can not be published.
Therefore, I changed the network configuration of VirtualBox and built an environment remote access to BOSH-Lite Director on the remote server.
- Hardware
- Prerequisite environment
- Change the VirtualBox network to Bridge Adapter
- Conclusion - BOSH-Lite in VirtualBox with Bridged Adapter for remote access
- NG method
Hardware
Physical server: DELL PowerEdge T110II, Xeon E3 1220, RAM 24GB
Host OS: CentOS 7.5 (vSphere VM), 1vCPU (4 cores), RAM 8GB
PC: MacBookAir 2018, RAM 8GB
Prerequisite environment
Basic flow and prerequisite environment are as described in the official.
Quick Start - Cloud Foundry BOSH
VirtualBox - Cloud Foundry BOSH
Change the VirtualBox network to Bridge Adapter
In the official procedure, build BOSH Director with the configuration of host-only adapter (vboxnet 0) + NAT network (NatNetwork) as follows.
The problem is how to connect from the remote device to the VM behind the host only adapter. As described in the later-described NG example, this problem can not be solved by any method of port-forwarding and routing.
Therefore, I changed the network configuration, share the NIC of the host OS and connect directly to the external NW by the connection method of Bridged Adapter of VirtualBox.
Official compatibility
It is not supported by the official as of 2018/Dec. It will be usable if this branch is merged, but it looks like it is left unattended...
Some people prepared documents.
However, it has not been merged, the release file has not been updated...
Clone and create Release
Since the function has already been added in the bridged branch, simply checkout and bosh create release.
git clone > git checkout -b bridged remote/origin/bridged > update blob > fix config/final.yml,config/private.yml > bosh create release --final ...
How to create BOSH Release is as below. Creating a Release - Cloud Foundry BOSH
Deploy BOSH-Lite Director
Deploy BOSH-Lite Director using the created Release file. Deployment of BOSH-Lite Director is need to operate in the host OS which VirtualBox is running.
Replace CPI with Operations-file and use bridged adapter.
* The following URL can be used as it is, but please be noted that it may stop publishing in the future.
ops-files/virtualbox-bridged-network.yml
- type: remove
path: /releases/name=bosh-virtualbox-cpi
- type: replace
path: /releases/-
value:
name: bosh-virtualbox-cpi
version: "0.0.13"
url: https://s3-ap-northeast-1.amazonaws.com/d703n-bosh-virtualbox-blobs/bosh-virtualbox-cpi-release-0.0.13-bridged.tgz
sha1: beaf2ff627663f7df6a8730c96caf6fd7c0c57bd
- type: replace
path: /networks/name=default/subnets/0/cloud_properties?
value:
type: bridged
name: ((network_device))
- type: replace
path: /networks/name=default/subnets/0/dns
value: [((dns_servers))]
Please change network_device, IP address according to the environment.
deploy-bosh.sh
bosh create-env ~/workspace/bosh-deployment/bosh.yml \ --state ./state.json \ -o ~/workspace/bosh-deployment/virtualbox/cpi.yml \ -o ~/workspace/bosh-deployment/bosh-lite.yml \ -o ~/workspace/bosh-deployment/uaa.yml \ -o ~/workspace/bosh-deployment/credhub.yml \ -o ~/workspace/bosh-deployment/jumpbox-user.yml \ --vars-store ./creds.yml \ -o ops-files/virtualbox-bridged-network.yml \ -v director_name=bosh-lite \ -v network_device=ens192 \ -v internal_ip=192.168.1.222 \ -v internal_gw=192.168.1.5 \ -v internal_cidr=192.168.1.0/24 \ -v dns_servers=192.168.1.77
Deploy.
$ ./deploy-bosh.sh Deployment manifest: '/home/dev/workspace/bosh-deployment/bosh.yml' Deployment state: './state.json' Started validating Downloading release 'bosh'... Started installing CPI Compiling package 'golang-1.8-linux/c97f9a00c26b34a3f59ca15b0f5a079d7f7e27c334cc8100248143c5dc0d4c0a'... Starting registry... Finished (00:00:00) Uploading stemcell 'bosh-vsphere-esxi-ubuntu-xenial-go_agent/170.9'... Finished (00:00:24) Started deploying Creating VM for instance 'bosh/0' from stemcell 'sc-261d9c63-8431-41e5-6396-0068ab24cfdf'... Finished deploying (00:10:06) Stopping registry... Finished (00:00:00) Cleaning up rendered CPI jobs... Finished (00:00:00) Succeeded
Connect to BOSH Director from remote
From here remote work is possible. Copy creds.yml needed for login to local and load it as an environment variable as follows.
#BOSH-Lite export BOSH_ENVIRONMENT=192.168.1.222 export BOSH_CA_CERT=`bosh int ~/project/bosh-deployment/creds.yml --path /director_ssl/ca` export BOSH_CLIENT=admin export BOSH_CLIENT_SECRET=`bosh int ~/project/bosh-deployment/creds.yml --path /admin_password`
If environment variable is read and a command such as bosh vms responds, connection succeeded.
$ bosh vms Using environment '192.168.1.222' as client 'admin' Succeeded
Deploy the Nginx BOSH Release
Since environment was built, try to deploy BOSH Release. Deploy a simple Nginx. (https://github.com/cloudfoundry-community/nginx-release)
Upload Stemcell and Cloud Config as preparation, and download Nginx BOSH Release.
bosh upload-stemcell https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-trusty-go_agent bosh update-cloud-config ~/project/bosh-deployment/warden/cloud-config.yml git submodule add https://github.com/cloudfoundry-community/nginx-release
Create an Ops-file for environment tuning.
ops-files / nginx-options.yml
- type: replace path: /releases/name=nginx/url? value: https://github.com/cloudfoundry-community/nginx-release/releases/download/1.13.12/nginx-release-1.13.12.tgz - type: replace path: /releases/name=nginx/sha1? value: c5d767f43a68b89d9603bccac2a60b6a5dc29836 - type: remove path: /instance_groups/name=nginx/networks/name=default/static_ips
Deploy Command
deploy-nginx.sh
bosh deploy -d nginx nginx-release/manifests/nginx-lite.yml \ -o ops-files/nginx-options.yml
Deploy.
$ ./deploy-nginx.sh Using environment '192.168.1.222' as client 'admin' Using deployment 'nginx' Task 2 Task 2 | 17:03:23 | Downloading remote release: Downloading remote release (00:00:04) Task 2 | ... Task 2 done + azs: + - name: z1 ... Continue? [yN]: y Task 3 Task 3 | 17:03:47 | Preparing deployment: Preparing deployment (00:00:01) Task 3 | 17:03:48 | Preparing package compilation: Finding packages to compile (00:00:00) Task 3 | 17:03:48 | Compiling packages: nginx/23c0391f6bb6630cf68ba02c99f93eabdd65839d (00:02:27) Task 3 | 17:06:17 | Creating missing vms: nginx/bd0e9a7f-6aeb-45e0-b5cf-8647c55bd7c7 (0) (00:00:13) Task 3 | 17:06:31 | Updating instance nginx: nginx/bd0e9a7f-6aeb-45e0-b5cf-8647c55bd7c7 (0) (canary) (00:00:24) Task 3 Started Sun Dec 23 17:03:47 UTC 2018 Task 3 Finished Sun Dec 23 17:06:55 UTC 2018 Task 3 Duration 00:03:08 Task 3 done Succeeded $ bosh vms Using environment '192.168.1.222' as client 'admin' Task 4. Done Deployment 'nginx' Instance Process State AZ IPs VM CID VM Type Active nginx/bd0e9a7f-6aeb-45e0-b5cf-8647c55bd7c7 running z1 10.244.0.2 b290ee7a-fa28-4232-500c-54422c6f3eab default true 1 vms Succeeded
Check access
You need to add routing to access the BOSH environment from the remote environment. Mac add the following route as follows. (Note that it will be reset when rebooting)
When sharing with multiple people, it is good to add a static route to the router which is the default gateway of the PCs.
sudo route add -net 10.244.0.0/16 192.168.1.222
$ curl http://10.244.0.2
<html><head><title>BOSH on IPv6</title>
</head><body>
<h2>Welcome to BOSH's nginx Release</h2>
<h2>
My hostname/IP: <b>10.244.0.2</b><br />
Your IP: <b>192.168.1.126</b>
</h2>
</body></html>
I got access without problems.
Conclusion - BOSH-Lite in VirtualBox with Bridged Adapter for remote access
Enhanced the bosh-virtualbox-cpi-release to access the BOSH-Lite environment remotely by using the VirtualBox Bridged Adapter. By running BOSH-Lite on a remote server, it is possible to work development of the BOSH Release and the confirmation of operation without influence of stopping the PC and resources etc.
In addition, it is limited to evaluation, it can also be used for public environment to the team.
NG method
I tried the following method, but it did not become expected behavior.
VirtualBox Port-Forwarding
Host OS: 192.168.1.83
Guest OS (BOSH Director): 192.168.50.6
Port-Forwarding: 192.168.1.83:25555 -> 192.168.50.6:25555
Just port forwarding with VirtualBox can not connect with a certificate error as follows.
$ bosh vms
Fetching info:
Performing request GET 'https://192.168.1.83:25555/info':
Performing GET request:
Retry: Get https://192.168.1.83:25555/info: x509: certificate is valid for 192.168.50.6, not 192.168.1.83
Exit code 1
Even though I think that it is sufficient to invalidate the certificate verification, unfortunately verification of the certificate is indispensable in BOSH CLIv2.
Routing with host OS
Set a static route to the subnet (host only adapter) to which BOSH Director belongs on remote PC.
$ sudo route add -net 192.168.50.6/32 192.168.1.83 Password: add net 192.168.50.6: gateway 192.168.1.83
The route added.
$ netstat -nr Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 192.168.1.5 UGSc 165 0 en0 127 127.0.0.1 UCS 0 0 lo0 127.0.0.1 127.0.0.1 UH 1 1750 lo0 ... 192.168.1.83 x:x:x:xx:xx:xx UHLWIi 2 899 en0 1074 ... 192.168.50.6/32 192.168.1.83 UGSc 0 0 en0 ... 255.255.255.255/32 link#9 UCS 0 0 en0 !
Also add routes to BOSH Director so that return packets will not be NatNetwork.
$ bosh int creds.yml --path /jumpbox_ssh/private_key > jumpbox.key $ chmod 600 jumpbox.key $ ssh jumpbox@192.168.50.6 -i jumpbox.key bosh/0:~# ip r default via 10.0.2.1 dev eth0 10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.4 192.168.50.0/24 dev eth1 proto kernel scope link src 192.168.50.6 bosh/0:~# route add -net 192.168.1.0/24 dev eth1 gw 192.168.50.1 bosh/0:~# ip r default via 10.0.2.1 dev eth0 10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.4 192.168.1.0/24 via 192.168.50.1 dev eth1 192.168.50.0/24 dev eth1 proto kernel scope link src 192.168.50.6
In the case of the above setting, although Ping passes, but it can not communicate with TCP. As discussed here, it seems that the host-only adapter is not routed. (It is strange that Ping goes through...)
BOSH guest can not connect to bridge
In BOSH-Lite, guest OS is created with Warden VM, guest network can not extend to VBox bridged adapter.
If the setting of the network in Cloud Config is incorrect, VM communication can not be performed and a deployment error occurs.
Task 6
Task 6 | 18:30:53 | Preparing deployment: Preparing deployment (00:00:01)
Task 6 | 18:30:54 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 6 | 18:30:54 | Compiling packages: nginx/23c0391f6bb6630cf68ba02c99f93eabdd65839d (00:10:48)
L Error: Timed out pinging to 38b8f90b-395b-40d0-8610-ca3de241a0bd after 600 seconds
Task 6 | 18:41:42 | Error: Timed out pinging to 38b8f90b-395b-40d0-8610-ca3de241a0bd after 600 seconds
Task 6 Started Fri Dec 21 18:30:53 UTC 2018
Task 6 Finished Fri Dec 21 18:41:42 UTC 2018
Task 6 Duration 00:10:49
Task 6 error
Updating deployment:
Expected task '6' to succeed but state is 'error'
Exit code 1
