The other day I was trying out Linux Containers (LXC) on my Debian Jessie server and found that Debian's support for controlling it with libvirt is somewhat inconclusive. I already had a basic libvirt setup for controlling a handful of virtual machines so I really wanted it to work also with LXC if possible.
The basic steps to create a container and control it with libvirt are the following (ref):
- Create a container using the LXC userspace tools.
- Create a libvirt XML domain definition.
- "Define" the libvirt container domain using the libvirt userspace tools.
- Start the libvirt container domain using the libvirt userspace tools.
See for example Scott Lowe's blog post for information on how to create the actual container. Once that is done you need to write a basic libvirt domain definition.
The following is a minimal domain definition.
Most elements are self-explanatory but I want to point out a few of them:
<memory>is mandatory even if you don't want to set a memory limit. This is problematic; see the next section.
<vcpu>defaults to 1 if not set, but you probably want to allow the container access to more than one CPU core.
- I am using an existing network bridge (br0). libvirt supports many types of
virtualized networks (NAT, host-only, PCI device passthrough etc.) and each
has a slightly different syntax for the
<interface>element. See the official documentation for further information.
- You need some sort of console for the container to start. I am not sure if
this is a restriction imposed by libvirt or if the Linux kernel or init system
simply refuse to boot without one. Easiest seems to be a pseudo terminal (PTY)
which you can then later attach to (for example using the
The cgroups memory controller
As mentioned, it is mandatory to set a memory limit (even if it is a really high
limit) for libvirt to accept the domain definition. The limit is enforced by the
cgroups memory controller. The Linux kernel in Debian Jessie is compiled with
support for the memory controller (kernel config parameter
However, it is operationally disabled by default (kernel config paramenter
CONFIG_MEMCG_DISABLED=y). Raymond P. Burkholder researched this
and found that this is for performance reasons and links to relevant Debian bug
reports, because apparently enforcing the memory limit has a certain amount of
I'll deal with that performance problem when it makes itself apparent and for
now I want to enable the cgroups memory controller. This is easily done by
cgroup_enable=memory kernel boot parameter. Append it to
GRUB_CMDLINE_LINUX in /etc/default/grub,
update-grub and reboot.
Now the container starts properly in libvirt. However, attaching to its console and attempting to login yields output similar to the following:
[ OK ] Started Permit User Sessions. Starting Getty on tty3... [ OK ] Started Getty on tty3. Starting Getty on tty2... [ OK ] Started Getty on tty2. Starting Getty on tty4... [ OK ] Started Getty on tty4. Starting Getty on tty1... [ OK ] Started Getty on tty1. Starting Console Getty... [ OK ] Started Console Getty. [ OK ] Reached target Login Prompts. [ OK ] Reached target Multi-User System. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Debian GNU/Linux 8 debian console debian login: Debian GNU/Linux 8 debian tty1 debian login: Debian GNU/Linux 8 debian console debian login: Debian GNU/Linux 8 debian tty1 debian login: root Password: p Login incorrect
Two getty instances are started, one on the "console" and one on tty1 and both are competing for your input. Parts of the login credentials are read by each getty instance, causing the login to fail and the password to be partially printed to the terminal. This has been fixed by LXC upstream in the Debian container template recently, so that getty is now only started on one of the consoles.
Debian being Debian, we can probably look forward to this fix downstream in 2017 or 2018 or so. Until then, it can be worked around by logging into the container (using LXC tools) and disabling getty on tty1-6 so that it only runs on console. Console is a "pseudo TTY" which by default is not declared to be a "secure TTY" (allowing root to login using it). To allow root login it must also be added to /etc/securetty. The following shell script performs those two actions:
Now the container can be started and controlled in libvirt.