Boxes

This repository is a clone of https://github.com/chef/bento/. It's highly recommend to keep changes minimal so one can fetch upstream improvements easily from time to time. As for usage, here's an example quickstart to create a RHEL 7.4 vagrant box.

Automated usage

According to the pipeline defined in this reposity .gitlab-ci.yml file, those boxes are built weekly on sunday and pushed to https://nexus.local/content/sites/packer. You shouldn't need to build them on your own. Just add the box_download_insecure and box_url parameters in your Vagrantfile as shown in this example

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|
  config.vm.box = "company/rhel-6"
  config.vm.box_download_insecure = true
  config.vm.box_url = 'https://nexus.local/content/sites/packer/rhel-6-x86_64.virtualbox.box'
  config.vm.provider "virtualbox" do |vb|
    vb.memory = "2048"
    vb.cpus = "2"
  end
  config.vm.provider :vmware_esxi do |esxi, override|
    esxi.esxi_hostname = '10.35.1.40'
    esxi.esxi_hostport = 22
    esxi.esxi_username = 'root'    
    esxi.esxi_disk_store = 'datastore2'
    esxi.guest_memsize = '4096'
    esxi.guest_numvcpus = '2'    
  end
end

Then vagrant up for first provisionner (virtualbox), will default to second (vmware_esxi) if first one failed, or enforce a specific provider via vagrant up --provider=vmware_esxi

Note, due to company's firewall you might have to add company's certificate in the file C:/HashiCorp/Vagrant/embedded/cacert.pem

Manual build for VirtualBox

  • Install Git
  • Install VirtualBox
  • Install Vagrant
  • Install Packer : download the zip, and extract the content in a folder in you PATH. For Linux users, /usr/bin would do, for Windows users C:\windows.
  • git clone this repository, or download it as an archive then extract its content, as you see fit
  • Open a command prompt (console for Linux users, CMD or Powershell for Windows users) in the repository folder

Now to build a RedHat 6.9 x86_64 vagrant box for VirtualBox

cd rhel
packer build -var "headless=true" -only=virtualbox-iso rhel-6.9-x86_64.json  

It takes time, like 10 minutes, a VirtualBox windows will appear, you'll see the OS installation going on, the VM rebooting... Known issue : sometimes on my slow workstation without SSD, the last reboot is stuck in a prompt, asking for root password or Ctrl-D. Click in the windows to capture mouse & keyboard, Ctrl-D so boot resume and Packer can finish the build, hit Ctrl to free your mouse & keyboard from VirtualBox's Windows.

Once finished, the newly packaged box will be in builds folder, it can be added to you local vagrant environment with

vagrant box add company/rhel-6 ../builds/rhel-6.9.virtualbox.box --provider virtualbox

Type vagrant box list to see what's already available. The newly built box can be used in a Vagrantfile looking like

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|
  config.vm.box = "company/rhel-6" # <================ value specified as "vagrant box add ORG/BOXNAME"
  config.vm.provider "virtualbox" do |vb|
    vb.memory = "2048"
    vb.cpus = "2"
  end
end

If the box isn't available locally, vagrant will look for it on https://app.vagrantup.com (for example, centos/7)

Building for VMWare

There are two possibilities to build a box / template for VMWare. Either Packer uses a local VMWare Workstation or Player, or it uses a remote ESX(i). Despites many attempt on a Windows 7 workstation or CentOS 7 VM (see the Vagrantfile shipped in this repository), the first possibility never worked. So we used the second, whose only downside is to have to enable SSH on an ESX. To avoid this security concern, using a dedicated ESX would be a good solution. Or maybe install it on a VM, could work. Here's the ESXi 6 ISO, 2 CPU otherwise installer won't accept it, 4Go RAM minimum otherwise vmkernel panic, 80 Go HDD otherwise Packer can't create the 65Gb disk for the build...

To build on remote ESXi:

  • Install packer
  • Install VMWare OVF Tool
  • If running on Windows, install the required Visual C++ Redistribuable if running on Windows, and add C:\Program Files\VMware\VMware OVF Tool to your PATH
  • Enable SSH on the ESXi
  • Enable GuestIPHack on the ESXi running the command esxcli system settings advanced set -o /Net/GuestIPHack -i 1
  • Download this repository

Manually building RHEL 6 with tons of CPU & RAM to speed up the build will look like

cd centos
packer build -var "cpus=8" -var "memory=16384" -var "headless=true" -only=vmware-iso -var "esx_host=10.35.1.40" -var "esx_username=root" -var "esx_password=XXXXXXXXXXXX" -var "esx_datastore=datastore2" -var "esx_network=VM Network" -var "http_proxy=http://10.10.10.10:8000" -var "https_proxy=http://10.10.10.10:8000" -var "no_proxy=localhost,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16" rhel-6.9-x86_64.json

WARNING : THIS DOESN'T WORK ! The resulting boxes won't boot on company's ESXi, disk is being ignored somehow and the VM loop on PXE / TFTP boot lookup. Most of the Packer available settings were tried, nothing fixed it. Tests have been made (see vagrant repository, upstream_*), official centos/6 and centos/7 works. A notable difference is they build their images over VMWare Workstation, most likely with a license. Some reading suggest switching to ext4 but rhel6 shouldn't use XFS anyway... I asked questions here for insight. Eventually testing VMWare tools built from linux.iso, or open-vm-tools. Or try other kickstart like CentOS's official kickstart. According to Evolution & TrevorH on IRC freenode #centos, CentOS build image via qemu-kvm & koji (their forge) and they just works for VMWare : another lead, try packer's qemu builder ?

How it works

As explained before this repository is a clone of Chef's Bento. All the major Linux distribution plus Windows are available, we only use RedHat and CentOS related stuff now, Windows someday maybe. So the relevant directories and files for our usage are

  • _common contains some generic scripts for most of Linux OSes  Amongst which vmware.sh was customized to replace VMWare tools installation from ISO by some scripting found there and forked on our Gitlab  
  • centos & rhel for Packer's json configuration files related to those OSes, plus OSes specific scripts.  Compared to upstream, network interface name was fixed in http/7/ks.cfg Json files were adapted, mirror and mirror_directory targeting company's, disk_size reduced, vmware-iso boot's command updated to use floppy and configuration added for ESXi's build, from remote_type to format  
  • company for our scripts, at least a repository script for redthat is required, otherwise no license mean no package during build, which is a problem
  • builds is the work directory for packer builds, as defined in the json configurations files. This is where the boxes are written when building locally.
  • .gitlab-ci.yml define the Gitlab Pipeline building the boxes weekly, the frequency, variables like passwords and jobs timeout being defined in Gitlab repository settings
  • Jenkinsfile is unused and most likely outdated, which can be fixed by c/cing from .gitlab-ci.yml if this job need to be migrated to Jenkins
  • Vagrantfile is just here for thinkering, wanted to be try some packer command under linux to see if they worked better there. Might still be useful !
  • Rest of the files like Rakefile, .travis.yml, etc are from upstream. It's better to keep them to stick closer to the original sources, so once in a while one can more easily compare with upstream and import the improvements.

Pipelines

.gitlab-ci.yml looks like this

---
# Timeout updated to 1 hour * build & upload job in https://gitlab.local/.../packer/settings/ci_cd, general pipelines settings, timeout

stages:
  - vmware
  - virtualbox

variables:
  PACKER_VERSION: "1.3.1"
  # Password are set in https://astgitlab.local/rg/packer/settings/ci_cd
  # ESX_HOST: "10.10.10.35"
  # ESX_USERNAME: "root"
  # ESX_PASSWORD: ""
  # ESX_DATASTORE: "datastore2"
  # ESX_NETWORK: "Admin"
  # NEXUS_URL: "https://nexus.local/content/sites/packer/"
  # NEXUS_LOGIN: "packer_rw"
  # NEXUS_PWD: ""
  PROXY: "http://10.10.10.10:8000"
  PACKER_LOG: 0

before_script:
  - export HTTP_PROXY=${PROXY}
  - export HTTPS_PROXY=${PROXY}
  - |
    if [ ! -x ~/bin/packer ]
    then
      curl -O https://releases.hashicorp.com/packer/${PACKER_VERSION}/packer_${PACKER_VERSION}_linux_amd64.zip
      mkdir -p ~/bin
      unzip -n -d ~/bin packer_${PACKER_VERSION}_linux_amd64.zip
      chmod +x ~/bin/packer
      rm packer_${PACKER_VERSION}_linux_amd64.zip
    fi
  - ~/bin/packer --version
  - |
    if [ ! $(which VBoxManage) ]
    then
      sudo curl -o /etc/yum.repos.d/virtualbox.repo http://download.virtualbox.org/virtualbox/rpm/rhel/virtualbox.repo
      sudo yum install -y VirtualBox-5.2 kernel kernel-devel kernel-headers
      sudo vboxconfig
      # reboot # if kernel got updated
    fi
  - VBoxManage --version
  - |
    if [ ! $(which unzip) ]
    then
      sudo yum install -y unzip
    fi



# https://docs.gitlab.com/ce/ci/yaml/#anchors
.job_template: &packerbuild
  stage: virtualbox
  tags:
  - ansible
  - docker
  # retry: 2

centos-7-x86_64-vmware:
  <<: *packerbuild
  stage: vmware
  script:
    - export http_proxy=${PROXY}
    - export https_proxy=${PROXY}
    - mkdir -p builds/packer-centos-7-x86_64-vmware
    - cd centos && ~/bin/packer build -var "cpus=4" -var "memory=8192" -var "headless=true" -var "http_proxy=${PROXY}" -var "https_proxy=${PROXY}" -only=vmware-iso -var "esx_host=${ESX_HOST}" -var "esx_username=${ESX_USERNAME}" -var "esx_password=${ESX_PASSWORD}" -var "esx_datastore=${ESX_DATASTORE}" -var "esx_network=${ESX_NETWORK}" centos-7.5-x86_64.json
    - cd ../builds/ && curl -k -u ${NEXUS_LOGIN}:${NEXUS_PWD} --upload-file centos-7.5.vmware.box ${NEXUS_URL}/centos-7-x86_64.vmware.box


centos-7-x86_64-virtualbox:
  <<: *packerbuild
  script:
    - export http_proxy=${PROXY}
    - export https_proxy=${PROXY}
    - cd centos && ~/bin/packer build -var "cpus=4" -var "memory=8192" -var "headless=true" -var "http_proxy=${PROXY}" -var "https_proxy=${PROXY}" -only=virtualbox-iso  centos-7.5-x86_64.json
    - cd ../builds/ && curl -k -u ${NEXUS_LOGIN}:${NEXUS_PWD} --upload-file centos-7.5.virtualbox.box ${NEXUS_URL}/centos-7-x86_64.virtualbox.box

Known issues

Those should be fixed, but still can come back I guess

  • SSH not connecting, no IP Address on interface : open tty1, login as vagrant/vagrant, sudo su, nmcli d co ens160. It's fixed here by network --bootproto=dhcp --onboot=on --device=ens160 instead of --device=eth0 in centos/http/7/ks.cfg, but name ens160 might change someday/when.
  • Kickstart file : for VMWare the boot command was updated to use a floppy. Reason is, default behavior is to query http://host_running_packer:8000~9000 to get the kickstart. Which can fail due to network firewall, routing & co. Note for floppy boot command, there are two syntax, the RedHat < 7 and RedHat >= 7, see this part of the doc.
  • VNC : documentation say you've to disable password for ESXi 6.5 and 6.7. Update "vnc_disable_password" in json files if needed.
  • Secrets & passwords shouldn't be hardcoded, especially with git or svn versionning. For the weekly build, passwords are stored in Gitlab (see CI/CD vars panel), passed down via the pipeline (.gitlab-ci.yml file) and the command with --var "option=value"
  • Build fail with something like Build 'vmware-iso' errored: FindFirstFile ../builds/packer-rhel-6.9-x86_64-vmware: Le fichier spécifié est introuvable.. It's most likely a packer bug, mkdir -p ../builds/packer-rhel-6.9-x86_64-vmware then run the build again

If something doesn't work, export PACKER_LOG=1 and run the command again to see what's wrong. Forget about -debug flag.

For more information

... about the way this work, read the README.md from the bento upstream repository and Packer documentation