kubernetes Featured

Spacewalk--Containerized And Running In Kubernetes

Jacob Salmela

Feb 18, 2019 • 8 min read

Spacewalk is a decent open source project for patch management but there is no native support for running it in a container. I’m a fan of Spacewalk and use it daily but it falls short in a few areas–namely, it’s a large monolithic application, which I’m not a fan of. I thought it could benefit from running in a container and on Kubernetes.

This blog post explains how you can run Spacewalk 2.8, 2.9, or 2.10 in Kubernetes and how I accomplished it. Whether or not you should, is a question you need to answer yourself after reading through this. The work I’ve done is all in a repo and I have images created on DockerHub (for Spacewalk v2.8, v2.9, and v2.10) or you can generate them yourself from the provided Dockerfiles:

Update 2020-04:

Spacewalk is being discontinued at the end of May. I hope this project will help it live on a bit. When I make enough time, I’ll fork my own version of Spacewalk. Until then, you can try Uyuni–but it’s a paid option so far less appealing.

You can find lots of articles on Spacewalk and even some about containerizing it, but most are very outdated or relate to ancient versions of Spacewalk. I used some of those posts as a starting point and then used my own knowledge and skills to get the rest of the way.

I initially got this working on Spacewalk version 2.8 many months ago. Since then, 2.9 was released and I finally got around to writing this blog post. I think 2.10 will be the last version they release (see the update above). I tagged releases in my repo if you want to check out specific versions and build the images yourself.

Deploying Spacewalk On Kubernetes

I’ll first show how you can deploy Spacewalk on Kubernetes if you just want to test it out quickly without knowing any details. Then, I’ll explain how I went about making Spacewalk into a container and why things are the way they currently are. This was tested using Docker for Mac but I’ve also actually deployed it to a real Kubernetes cluster, so you may need to adjust things for your environment. Any feedback and pull requests are welcome.

I’ll make the following assumptions:

You’ve worked with or installed Spacewalk before
You care about using Spacewalk for patches, and not much else
You have Docker for Mac or another Kubernetes cluster to test with
You’re familiar with Dockerfiles
You’re familiar with Kubernetes
You have some Linux experience (hacking your way through if the guide is not perfectly accurate to your system)
You understand that database is also running in a container with a persistent volume and what this means for any data you put in it
This isn’t for a production system

Quick and dirty deployment

Here is the quick and dirty of how to get it running:

git clone https://github.com/jacobsalmela/spacewalk-k8s.git
cd spacewalk-k8s
docker build -t spacewalk:2.9 -f Dockerfile-centos .# Docker pull will also work here since I pushed them to my repo
cd postgres/
docker build -t spacewalk-postgres:2.9 .# Docker pull will also work here since I pushed them to my repocd .. kubectl create -f spacewalk-pv/ kubectl create -f spacewalk/ kubectl get po -wkubectl exec -it spacewalk-web-something /bin/bash[root@spacewalk-web-84db75f4fb-zjgc7 /]# supervisorctl status spacewalk-initspacewalk-init RUNNING pid 20, uptime 0:00:10# about a minute later...[root@spacewalk-web-84db75f4fb-zjgc7 /]# supervisorctl status spacewalk-initspacewalk-init EXITED Feb 18 05:53 PM kubectl port-forward <pod> 8000:80 8443:443 8080:8080

If It Doesn’t Seem To Be Working…

The above should give you a working deployment. Although, I occasionally needed to adjust the sleep command for a longer duration. I think this has to do with certain parts of the deployment that are not fully ready yet. There’s probably some advanced Kubernetes tricks I haven’t learned yet to account for this stuff, but if it fails or is just endlessly running, you can just exec into the container and run this command (when first installing):

supervisorctl stop spacewalk-init

You can also run the command manually:

spacewalk-setup --skip-fqdn-test --skip-db-install --skip-db-population --skip-services-restart --answer-file=/spacewalk-init/answerfile.txt && supervisorctl restart httpd && supervisorctl restart tomcat

The important thing to note here is that the above command is only run during the initial installation. Once installed, you should modify your deployment (spacewalk/spacewalk-supervisord-config.yml) to run this command instead (instructions for this are in the config map file):

spacewalk-setup --skip-ssl-ca-generation --skip-ssl-cert-generation --skip-fqdn-test --skip-db-install --skip-db-population --skip-services-restart --answer-file=/spacewalk-init/answerfile.txt && supervisorctl restart httpd && supervisorctl restart tomcat

This just passes some different flags to avoid messing with the database and the certs. More on the certs later, because they are important with Spacewalk if you want to be able to successfully patch something.

Once it’s up and running…

After the containers are all running and you have forwarded the ports (I’m assuming local development here–you may have an ingress or something, so adjust accordingly), you can navigate to:

https://127.0.0.1:8443/rhn/newlogin/CreateFirstUser.do

This will let you create the first super user and populate the database.

So now you have the latest version of Spacewalk (2.9 at the time of this writing) running in Kubernetes. Yay! There is more to do, however–and more to explain.

Registering Clients

For development’s sake, I used a CentOS container to demonstrate how you’d register a client but the process would be the same for a regular server (because you don’t really need to patch a container…). I’m also using an activation key here, so you’ll need to make one of those for the last command to work without modification.

rpm -Uvh https://copr-be.cloud.fedoraproject.org/results/%40spacewalkproject/spacewalk-2.9-client/epel-7-x86_64/00830558-spacewalk-repo/spacewalk-client-repo-2.9-4.el7.noarch.rpm
rpm -Uvh http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum -y install rhn-client-tools rhn-check rhn-setup rhnsd m2crypto yum-rhn-plugin
rpm -Uvh http://<SPACEWALK WEB CONTAINER>/pub/rhn-org-trusted-ssl-cert-1.0-1.noarch.rpm
rhnreg_ks --serverUrl=https://<SPACEWALK WEB CONTAINER>/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT --activationkey=1-centos7

Keeping Registered Clients Registered

If you happen to re-run the installer by deploying a new container without adjusting the supervisord config map, you’ll have regenerated the client certs that Spacewalk uses to verify the clients and vice versa. This is why you need to modify the deployment to pass the extra flags to not generate the certs on install . This is obviously a bit janky but we’re not in production and this is just the first step towards something cool.

Why Make Spacewalk Into A Container?

I have used Spacewalk for a couple of years and it’s a total workhorse for pushing our patches and keeping a local sync of repositories on your LAN. It definitely has some quirks, which I won’t go into detail on, but for the most part, Spacewalk is a great tool for an enterprise environment. Others may argue because of some of it shortfalls, but I’ve used it to support 500 machines for a large enterprise and it works. In light of other competition in the free tier, Spacewalk is currently the winner.

One large problem with the software is that is monolithic. This means upgrades are difficult and there’s not a lot of redundancy built into it. It depends on a lot of different services and if they go do down, it can be difficult to troubleshoot. This is, of course, where containers and microservices show their value–you could update each component of the software independent of each other. So if you want to update the database software to a new version, but leave the Web front end alone, containers can help with that.

How I “Containerized” Spacewalk

I spent many days with one failure after another, but it was a good learning experience. Fortunately, Spacewalk’s installer has the flag for an external database, which simplified things. Here’s a basic summary of what I did:

I tried to run the official installer inside a CentOS container
This failed miserably
I tried the installer again, but with the flag to use an external database
This showed as successful but nothing worked
I tried injecting the .sql from a clean install instead of relying on the installer to do it
I had to dig around the source code to find this
I modified some services to start using supervisord
Some works, some failed
I modified the install command with additional flags and used two containers (one for Spacewalk and one for the database)
I converted config files and secrets into ConfigMaps
This resulted in an easier deployment and less manual intervention
New certs were generated each time I deployed
I added persistent storage
I added a hostname flag the deployment
I injected pre-made certs as a secret

Of course, all of the above things took several days per bullet point to actually accomplish. In addition to all that, I ran into lots of other roadblocks:

Containers don’t have systemd, which Spacewalk depends on
I had new error messages every step of the way
I had to exec into containers and debug a lot of issues
I did a lot of reading through Spacewalk’s “interesting” source code
I experimented with the Redhat container, which has some systemd stuff, but the route I took seems to work

What Is Happening With This Hot Mess Of A Deployment?

There are two main containers that make up this deployment, the postgres one and the Spacewalk one, which is the front end.

The Postgres Container

The postgres container is a custom build that installs main.sql, which can be found on a normal install at /etc/sysconfig/rhn/postgres/main.sql. This pre-populates postgres with the same thing that the interactive installer would, I just inject it beforehand.

# Install the main.sql file from a clean install of Spacewalk 2.9 so it populates the database correctly

ADD main.sql /docker-entrypoint-initdb.d/

The only modification I made to the main.sql file was to add these two lines, which it fails without them:

CREATE LANGUAGE pltclu;
CREATE USER spaceuser SUPERUSER;

The Spacewalk Container

This container basically follows the same install instructions with a few additions. First, it installs the Spacewalk repos.

...
RUN yum install -y https://copr-be.cloud.fedoraproject.org/results/@spacewalkproject/spacewalk-2.9/epel-7-x86_64/00830557-spacewalk-repo/spacewalk-repo-2.9-4.el7.noarch.rpm \https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm && \yum clean all
...

Then it installs Spacewalk. But I also install supervisord, which is essentially used to replace systemd in this case. Spacewalk also needed a couple of perl utilities, which I have included.

RUN yum -y install \
  spacewalk-postgresql \
  supervisor \
  osad \
  wget \
  perl-Text-Unidecode \
  perl-Frontier-RPC-Client && \
  yum clean all

Then I EXPOSE the necessary ports

# Web interface, Jabber, DHCP/TFTP
EXPOSE 80 443 5222 68 69

I add in some custom shell scripts I wrote for syncing CentOS repos and errata.

# Copy scripts needed for syncing errata. Perhaps better served as a PV, but this works for now
COPY centos-clone-errata-full.sh centos-clone-errata.sh epel-clone-errata.sh ya-errata-import.pl repo_sync.sh /spacewalk-scripts/

Then I set the entrypoint to use supervisord, which is running and loading all the services Spacewalk needs.

ENTRYPOINT supervisord -c /etc/supervisord.d/supervisord.conf

The supervisord.conf file is stored as a configmap and basically runs the essential components needed for Spacewalk (Jabber, Tomcat, Apache, etc.) This file also controls what command is run when the container boots up, so basically every time you deploy the container it re-runs the spacewalk install command.

And that’s it in a nutshell. There are a lot of other things happening in the deployment, but I’ll leave that to future edits of this blog post.