sampo Featured

Introducing Sampo: A Bash API Server For Running Shell Scripts

Jacob Salmela

Sep 15, 2020 • 8 min read

v1.0 of Sampo is an MVP. Written in bash, it is a shell script API server you can run directly in your terminal or via container on Kubernetes. It's open source and I hope you'll consider contributing.

It looks like this if you run it directly in your shell:

jacobsalmela:~ while true; do { echo -e 'HTTP/1.1 200 OK\r\n'; bash ./sampo.sh; } | nc -l 1042; done &
jacobsalmela:~ curl http://localhost/echo/jacob
jacob
jacobsalmela:~

In this example, I start the shell script in unison with netcat and send that process to the background, and then I test it again using curl.

or this on Kubernetes:

jacobsalmela:~ kubectl create -f sampo/
jacobsalmela:~ kubectl port-forward --address 0.0.0.0 "$POD" 1042:1042 &
jacobsalmela:~ curl -i http://localhost:1042/echo/jacob
HTTP/1.1 200 OK
Date: Tue, 01 Sep 2020 20:12:57 UTC
Version: HTTP/1.1
Accept: text/plain
Accept-Language: en-US
Server: sampo/1.0.0
Content-Type: text/plain
Content-Length: 54

jacob
jacobsalmela:~

In this example, I create the Kubernetes deployment, and then test the endpoint using curl

With sampo, you can:

configure an endpoint in shell code ( /echo/ in the example above)
link the endpoint to some shell code (the example endpoint just returns the value you sent)
and then call it from somewhere--often using shell code (I'm just using a local curl call in the example above)

The sampo container and Kubernetes deployment manifest offer the most flexibility, but you can also run it directly from a shell if you prefer.

The majority of testing I did was in Kubernetes since it's more resilient and easy to test and re-deploy (see the build.sh script if you're developing locally).

Why Develop This?

There's a few reasons, but mainly it comes down to my love for shell scripts and I want others to be able to learn it as well.

The Enterprise Uses Shell Code

If an enterprise uses Linux, it's highly-likely the engineers and admins have shell scripts in place that keep things running. I have seen people write shell scripts to fix a small issue quickly, but then that same shell script grows until it's difficult to remove without completely breaking the infrastructure. This is why I often say:

shell scripts have an unbelievably long half-life.

A script might start out as a few lines of code written by an admin to fix something and can be passed around to others to fix it on their system, which a developer then finds and then expands the shell code to do even more. Before you know it, this humble shell script is now keeping the enterprise operation going. I have seen this happen at every place I've ever worked at. And often the admin and developer are the same person. In either case, it shows that shell scripts are used and needed in the enterprise.

Software Ebbs and Flow But Shell Scripts Are...Forever?

I mentioned that shell scripts have a long half-life because I've seen it in every job I've had--someone will write a shell script to do x, y, or z, and years later that shell script is still keeping the lights on by supporting a production system.

I have also seen paid-for, enterprise software get swapped out for a competitive product that does the same thing without a second thought. So while these expensive products are expendable and replaceable, the free, hand-made shell scripts persist and keep the business running.

The example above also always reminds me of this:

This reminds me of how enterprise software sometimes stacks on top of existing open source projects

Software Delivery

Many software products also use a shell script to help users install their software. Take Docker for example: https://get.docker.com/ links you directly to a shell script. Or Homebrew, which naturally uses a shell script for their installer.

No Dependencies

This leads me into my next point, which is why shell code is so pervasive. Writing a shell script needs very few--if any--external dependencies.

That's not always the case. Even in sampo, I depend on netcat and/or socat, which isn't always included by default in some operating systems. And many of the shell scripts out there will have the software that is needed already installed. But the point I mean to get across is that if you have a Linux system, you have a shell; which means you write a considerable amount of code without having to add anything extra, and it can perform the same across systems because you don't need to install dependencies to run it.

Even if you do need some extra dependencies, you can write simple checks for those into your shell script and fail if it doesn't have them.

APIs Are Critical

Now that you know how important shell scripts are and how they can outlive their initial purpose, I'll discuss the next import piece, which are APIs. Application Programing Interfaces allow applications to talk to one another.

Instead of clicking a link or a checkbox; typing something into an application and pressing enter; or moving data from one application to another, an API takes away the graphical interface and replaces it with code that does the same thing. This code is easy for a computer to understand and easy for a human to keep in source control. These graphical interfaces are often just calling an API in the background.

So if APIs are very useful and so are shell scripts, why not combine them so your shell scripts can be available to others to use via an API call?

Shell Code Is A Gateway Language

I see shell scripts as a gateway for higher-level languages because there is a gradual and continuous learning curve:

you start out learning the basic commands ( ls, cd, rm, mkdir, etc.)
you begin learning other commands to manipulate services ( systemctl, service, etc.) or do other things with the OS
you put commonly used commands into a shell script in a simple manner: just calling one after another to save yourself from entering them manually
you start manipulating input, output, and errors
you start modifying your shell scripts to do simple error checking
you add logic to your scripts to account for different situations
you remove hardcodes and replace them with variables
you make reusable functions
you source different files to logically organize your code
you start deploying software or fixing issues with your shell scripts
you continue to learn new commands and new ways to incorporate them into your scripts
you write less and less code as you learn more about bash because you can condense and consolidate older snippets

It's this gradual and continuous learning curve that makes bash so wonderful. Your skills are kept up-to-date because as you use the command line on a regular basis, things that you do throughout the day re-enforce what you do in your shell scripts.

Shell Scripts Are Very Fast Humans

Shell scripts have such a close tie to the OS that they often just:

emulate what a human would do if they were sitting at the terminal, but they can do it in a matter of seconds.

Computers still need humans to do things. But shell scripts enable humans to do it faster. Thus, this is one of the reasons why I wrote the whole thing in bash.

Details

I'll explain most of this in the context of Kubernetes since that's where I run it, but the shell code will be all the same, you'll just go about deploying it differently.

sampo is built off of the following technologies:

bash
bashttpd
netcat / socat
Docker
Kubernetes
bats-core (for simple testing during development)

There are two scripts that do it all:

sampo.sh is the API server, which needs to be run with socat or a while true loop using netcat
sampo.conf is the config file for defining your endpoints (this is just a shell script that gets sourced by sampo.sh) but you can technically define your own functions here as well

In it's default state, you get a four basic endpoints as examples:

/ shows a list of all the configured endpoints and the function that it executes
/echo/ shows an echo of a string passed to it
/root/ shows the files in / using ls
/issue/ shows the content of /etc/issue
/example/ calls a shell script

With a little bit of shell scripting know-how, you can extrapolate from these endpoints how to start creating your own.

The most useful and extendable endpoint is one that can run any arbitrary shell script. I have included an example of this using the function run_external_script, which can call a script elsewhere on your server. In the example script, I just echo some output and it shows when you hit the example /example endpoint.

jacobsalmela:~$ curl -i http://localhost:1042/example
HTTP/1.1 200 OK
Date: Tue, 08 Sep 2020 11:17:47 UTC
Version: HTTP/1.1
Accept: text/plain
Accept-Language: en-US
Server: sampo/1.0.0

This is an example of an external script.
It's running a script with three echo commands in it.
This could be any arbitrary code you want.

This script is running in the container and could be any shell code you want:

---
apiVersion: v1
data:
  external.sh: |
    echo -e "This is an example of an external script."
    echo -e "It's running a script with three echo commands in it."
    echo -e "This could be any arbitrary code you want."
kind: ConfigMap
metadata:
  name: external-example

This simply demonstrates that you can have any shell script you want and have it callable by an API endpoint. It will return whatever prints out from your shell script. This could be a simple echo like I showed, or it could be a complex shell script that runs something important.

Caveats And What It's Not

This is an MVP and the main purpose is to prove out that you can run a shell script from an API endpoint. It does that, but not much else...yet.

There's no authentication at all, no encryption, and it's not compliant with HTTP standards. It's also not really RESTful right now but is what it's goal is.

Theme

In the Kalevala, the sampo is a magical artifact of indeterminate type, constructed by Ilmarinen, that brought good fortune to its holder. It was stolen and lost at sea during a battle.

The sampo is described as a compass, astrolabe, a Vendel period shield, or a box-like device.

There's some nautical theme in those descriptions above, which tie in well with Kuberenetes. I'm also Finnish, so the name sampo seemed like a good fit. Plus, 🐚 shell scripts also fit in with the nautical theme in a punny sort of way...

Contribute

sampo is open source and I'm hoping you'll contribute. I have left detailed comments in the shell scripts to help make it easy to learn and understand. I'll continue to work on it myself, but it's always more fun when people can find improvements I might never even think of. I also have two full-time jobs, so any progress I make individually will be slow, so I'm launching this to see if it takes off.

Feel free to open any issues or PRs that you want.