I haven't posted in a little while, so here's a nice juicy tutorial that I've been working on. There's some more news surrounding this, but I think that I'll start off with a Homemade Hypervisor for you to sink into.
A clean Gentoo server using systemd. I can provide stage3 tarballs if you need them to install from. SSH access and PAM accounts are setup (local if you need them, NIS or LDAP if you can). You have a root login (or can sudo). Host has static IP address (v4 and v6), DNS resolves and a hostname is set for your convenience (but nothing in this tutorial depends on correct resolvers). Host is not a router/DHCP/DNS server or running any other network infrastructure services. Host interfaces are attached to an internal bridge, "br0".
/etc/systemd/network/bridge.netdev
[NetDev]
Name=br0
Kind=bridge
/etc/systemd/network/ethernet.network
[Match]
Name=enp*
[Network]
Bridge=br0
/etc/systemd/network/host.Network
[Match]
Name=br0
[Network]
Address=192.0.2.X/24
DNS=192.0.2.1
Gateway=192.0.2.1
Address=2001:db8::X/64
DNS=2001:db8::1
Gateway=2001:db8::1
Host has a volume group named "vg"
# pvs
PV VG Fmt Attr PSize PFree
/dev/sda1 vg lvm2 a-- 2.73t 2.53t
/dev/sdb1 vg lvm2 a-- 2.73t 2.53t
/dev/sdc1 vg lvm2 a-- 2.73t 2.53t
/dev/sdd1 vg lvm2 a-- 2.73t 2.53t
Install xen and libvirt Reboot into Xen Start xen and libvirtd services In gentoo, you need to do some configuration, so here are some salt states for you.
/srv/salt/xen/init.sls
# These unit files are taken from Arch, which took them from Fedora.
# Some additions have been made based on Gentoo's init.d scripts.
{% for unitfile in [
'proc-xen.mount',
'var-lib-xenstored.mount',
'xenconsoled.service',
'xenstored.service',
'xen-watchdog.service'] %}
/etc/systemd/system/{{ unitfile }}:
file:
- managed
- source: salt://xen/files/{{ unitfile }}
{% endfor %}
/etc/xen/xl.conf:
file:
- managed
- contents: |
vif.default.bridge="br0"
# Some systems use /run, but the default configuration values use /var.
/var/lock:
file:
- directory
xen-packages:
pkg:
- installed
- names:
- app-emulation/xen
- app-emulation/xen-tools
- app-emulation/xen-pvgrub
xen-services:
service:
- running
- enable: True
- names:
- xenstored
- xenconsoled
- xen-watchdog
- provider: systemd
/srv/salt/xen/files/proc-xen.mount
[Unit]
Description=Mount /proc/xen filesystem
ConditionPathExists=/proc/xenconsoled
RefuseManualStop=True
[Mount]
What=xenfs
Where=/proc/xenconsoled
Type=xenfs
/srv/salt/xen/files/var-lib-xenstored.mount
[Unit]
Description=mount xenstore file system
What=tmpfs
Where=/var/lib/xenstored
Type=tmpfs
/srv/salt/xen/files/xenstored.service
[Unit]
Description=Xenstored - daemon managing xenstore filesystem
Requires=proc-xen.mount var-lib-xenstored.mount
After=proc-xen.mount var-lib-xenstored.mount
Before=libvirtd.service libvirt-guests.service xendomains.service xend.service
RefuseManualStop=true
ConditionPathExists=/proc/xen
[Service]
Type=forking
Environment=XENSTORED_ARGS=
Environment=XENSTORED_ROOTDIR=/var/lib/xenstored
EnvironmentFile=-/etc/conf.d/xenstored
PIDFile=/var/run/xenstored.pid
ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities
ExecStartPre=-/bin/rm -f ""/tdb*
ExecStartPre=/bin/mkdir -p /var/run/xen
ExecStart=/usr/sbin/xenstored --pid-file /var/run/xenstored.pid
ExecStartPost=/usr/bin/xenstore-write "/local/domain/0/name" "Domain-0"
ExecStartPost=/usr/bin/xenstore-write "/local/domain/0/domid" "0"
[Install]
WantedBy=multi-user.target
/srv/salt/xen/files/xenconsoled.service
[Unit]
Description=Xenconsoled - handles logging from guest consoles and hypervisor
Requires=proc-xen.mount
After=proc-xen.mount xenstored.service
ConditionPathExists=/proc/xen
[Service]
Type=simple
Environment=XENCONSOLED_ARGS=
Environment=XENCONSOLED_LOG=none
Environment=XENCONSOLED_LOG_DIR=/var/log/xen/console
EnvironmentFile=-/etc/conf.d/xenconsoled
PIDFile=/var/run/xenconsoled.pid
ExecStartPre=/bin/grep -q control_d /proc/xen/capabilities
ExecStart=/xen/sbin/xenconsoled --log= --log-dir=
[Install]
WantedBy=multi-user.target
/srv/salt/xen/files/xen-watchdog.service
[Unit]
Description=Xen watchdog daemon
Requires=proc-xen.mount
After=proc-xen.mount
ConditionPathIsDirectory=/proc/xen
[Service]
Type=forking
ExecStart=/usr/sbin/xenwatchdogd 30 15
KillSignal=USR1
[Install]
WantedBy=multi-user.target
/srv/salt/libvirt/init.sls
app-emulation/libvirt:
pkg:
- installed
/etc/libvirt/libvirtd.conf:
file:
- managed
- contents: |
unix_sock_group = "qemu"
log_level = 1
log_outputs="1:stderr"
libvirt-services:
service:
- running
- enable: True
- names:
- libvirtd
- virtlockd.socket
- provider: systemd
- watch:
- file: /etc/libvirt/libvirtd.conf
There is no salt module to do this (yet?), but it only needs to be done once.
cat << EOF > pool-default.xml
<pool type='dir'>
<name>default</name>
<source>
</source>
<target>
<path>/var/lib/libvirt/images</path>
</target>
</pool>
EOF
virsh pool-define pool-default.xml
cat << EOF > pool-vg.xml
<pool type='logical'>
<name>vg</name>
<source>
<name>vg</name>
<format type='lvm2'/>
</source>
<target>
<path>/dev/vg</path>
</target>
</pool>
EOF
virsh pool-define pool-vg.xml
You may need to undefine the preconfigured virbr0
virsh net-destroy default
virsh net-undefine default
Define a bridge to attach VM networking to
cat << EOF > net-default.xml
<network ipv6='yes'>
<name>default</name>
<forward mode='bridge'/>
<bridge name='br0'/>
</network>
EOF
virsh net-define net-default.xml
virsh pool-start default
virsh pool-autostart default
virsh pool-start vg
virsh pool-autostart vg
virsh net-start default
virsh net-autostart default
virsh pool-list
virsh net-list
Download a VM image to run. I'm using an all-in-one kernel (with integrated initramfs) which boots into a live environment. You can build one using the instructions on GitHub. Also, take the time to create a unique ID for this VM and provision an LV.
wget https://dl.condi.me/gentoo-systemd/latest/vmlinuz
cp vmlinuz /var/lib/libvirt/images/gentoo-systemd
UUID=$(uuidgen)
virsh vol-create-as vg $UUID 50G
Define a new VM. I deploy a lot of servers and throwing VNC around my network isn't desirable, so I'm only using a serial console.
cat << EOF > vm-$UUID.xml
<domain type='xen'>
<name>vm-$UUID</name>
<uuid>$UUID</uuid>
<memory unit='GiB'>1</memory>
<os>
<type arch='x86_64' machine='xenpv'>linux</type>
<kernel>/var/lib/libvirt/images/gentoo-systemd</kernel>
</os>
<devices>
<disk type='block' device='disk'>
<source dev='/dev/vg/$UUID'/>
<target dev='xvda' bus='xen'/>
</disk>
<interface type='bridge'>
<mac address='00:00:00:00:00:00'/>
<source bridge='br0'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='xen' port='0'/>
</console>
</devices>
</domain>
EOF
virsh define vm-$UUID.xml
Start the vm
virsh start vm-$UUID
Enter the console
virsh console vm-$UUID
You can "ctrl + ]" to exit the console
Destroy and cleanup
virsh destroy vm-$UUID
virsh undefine vm-$UUID
virsh vol-delete $UUID --pool vg
rm vm-$UUID.xml