Build a bash builtin

2020-01-15

Anybody who has used the bash shell has used the help command at one point or another. This command lists all the commands that built into the shell and a few other help topics but since it dumps almost one hundred topics into your terminal it is easy to miss a few gems in there.

One of these gems is the enable builtin:

enable: enable [-a] [-dnps] [-f filename] [name ...]
    Enable and disable shell builtins.
    
    Options controlling dynamic loading:
      -f        Load builtin NAME from shared object FILENAME
      -d        Remove a builtin loaded with -f

Dynamic loading of builtins from a file! This means that we can hook into the shell and access its internal state.

While the bash manual doesn't go into depth as to how to create a builtin, the bash source code distribution is a bit more helpful:

bash-master $ ls examples/loadables/
basename.c  id.c             mypid.c     README      stat.c       uname.c
cat.c       ln.c             necho.c     realpath.c  strftime.c   unlink.c
dirname.c   loadables.h      pathchk.c   rm.c        sync.c       whoami.c
fdflags.c   logname.c        perl        rmdir.c     tee.c
finfo.c     Makefile.in      print.c     seq.c       template.c
head.c      Makefile.inc.in  printenv.c  setpgid.c   truefalse.c
hello.c     mkdir.c          push.c      sleep.c     tty.c

The loadables directory contains a whole host of examples of how to write your own builtins.

The README doesn't try to hide the fact that we are venturing into less traveled territory here:

Many of the details needed by builtin writers are found in hello.c, the canonical example. There is no real `builtin writers' programming guide'. The file template.c provides a template to use for creating new loadable builtin

Creating a new builtin

After some digging around in the bash source code, I found a way to compile a loadable builtin on Ubuntu. Build instructions on other systems will be different but if you obtain a copy of the bash source you should have everything you need.

Let's start by creating an new directory for this project and copying the file examples/loadables/template.c from the bash source repository into our new directory.

bash-plugin $ cp ~/Downloads/bash-master/examples/loadables/template.c ./hello.c

Inspecting hello.c, we'll find the following structural elements:

int
template_builtin (list)
     WORD_LIST *list;

/* Called when `template' is enabled and loaded from the shared object.  If this
   function returns 0, the load fails. */
int
template_builtin_load (name)
     char *name;

/* Called when `template' is disabled. */
void
template_builtin_unload (name)
     char *name;

char *template_doc[] = {
	"Short description.",
	""
	"Longer description of builtin and usage.",
	(char *)NULL
};

struct builtin template_struct = {
	"template",			/* builtin name */
	template_builtin,		/* function implementing the builtin */
	BUILTIN_ENABLED,		/* initial flags for builtin */
	template_doc,			/* array of long documentation strings. */
	"template",			/* usage synopsis; becomes short_doc */
	0				/* reserved for internal use */
};

Most of this is either self-explanatory or explained enough by the comments.

We'll use this template to implement a simple builtin that will

set the variable GREETING to the value "hello, world",
define a builtin function hello, that will display the value of the variable GREETING

Before proceeding, we'll replace all occurrences of template with hello, since that is how we want to call our new builtin.

bash-plugin $ sed -i 's/template/hello/g' hello.c

Magic build incantations

If you just want to compile your new builtin, jump to the end of this section

In order to actually use our new builtin, we need to compile it first to shared object.

Let's try the naive thing and just tell GCC to created a shared object hello.so from our source file:

bash-plugin $ gcc -shared -fPIC -o hello.so hello.c
hello.c:5:10: fatal error: config.h: No such file or directory
    5 | #include <config.h>
      |          ^~~~~~~~~~
compilation terminated.

This fails because the necessary bash header files are not found. On Ubuntu, after installing bash-builtins, we can find the config.h header file under /usr/include/bash

bash-plugin $ dpkg-query -L bash-builtins | grep -F config.h
/usr/include/bash/config.h

We need to tell GCC about this directory by adding the -I flag:

bash-plugin $ gcc -shared -fPIC -I /usr/include/bash -o hello.so hello.c
hello.c:14:10: fatal error: loadables.h: No such file or directory
   14 | #include "loadables.h"
      |          ^~~~~~~~~~~~~
compilation terminated.

Another compilation error! Luckily bash-builtins also provides this file, albeit in a different location:

$ dpkg-query -L bash-builtins | grep -F loadables
/usr/lib/bash/loadables.h

Extending our compile command with this new directory, we get this:

bash-plugin $ gcc -I /usr/include/bash -I/usr/lib/bash -shared -fPIC -o hello.so hello.c
In file included from /usr/include/bash/builtins.h:33,
                 from /usr/lib/bash/loadables.h:29,
                 from hello.c:14:
/usr/include/bash/command.h:25:10: fatal error: stdc.h: No such file or directory
   25 | #include "stdc.h"
      |          ^~~~~~~~
compilation terminated.

Again the same type of error, so we can apply the same type of solution again:

bash-plugin $ dpkg-query -L bash-builtins | grep -F stdc.h
/usr/include/bash/include/stdc.h
bash-plugin $ gcc -I /usr/include/bash/include -I /usr/include/bash -I/usr/lib/bash -shared -fPIC -o hello.so hello.c
In file included from hello.c:14:
/usr/lib/bash/loadables.h:31:10: fatal error: bashgetopt.h: No such file or directory
   31 | #include "bashgetopt.h"
      |          ^~~~~~~~~~~~~~
compilation terminated.
bash-plugin $ dpkg-query -L bash-builtins | grep -F bashgetopt
/usr/include/bash/builtins/bashgetopt.h

This was the last time we'd run into this issue. The final build command for our plugin is thus:

bash-plugin $ gcc -I /usr/include/bash/builtins \
  -I /usr/include/bash/include \
  -I /usr/include/bash \
  -I/usr/lib/bash \
  -shared -fPIC -o hello.so hello.c
bash-plugin $ echo $?
0

Let's put this into a makefile so that we don't have to repeat ourselves all the time:

INCLUDE := /usr/include/bash/builtins \
           /usr/include/bash/include \
           /usr/include/bash \
           /usr/lib/bash

hello.so: hello.c
	gcc $(foreach dir,$(INCLUDE),-I $(dir)) -shared -fPIC -o $@ $<

Now we can build our plugin by just invoking make:

bash-plugin $ make
gcc -I /usr/include/bash/builtins -I /usr/include/bash/include -I /usr/include/bash -I /usr/lib/bash -shared -fPIC -o hello.so hello.c
bash-plugin $ make 
make: 'hello.so' is up to date.

Adding functionality

With a working build process in place, we can add functionality to our builtin now.

Unfortunately I couldn't find any documentation about the internal APIs of bash. This turned out not to be much of a problem though, since the functions are all well named and mostly do what their name says.

Before getting into how to work with bash variables, we change the definition of hello_builtin to just print hello, world. That way we can see that our builtin actually works:

int
hello_builtin (list)
     WORD_LIST *list;
{
  printf("hello, world\n");
  return EXECUTION_SUCCESS;
}

Seeing our new builtin in action is pretty straightforward:

bash-plugin $ make --quiet
bash-plugin $ ls *.so
hello.so
bash-plugin $ enable -f ./hello.so hello
bash-plugin $ hello
hello, world

We'll probably want run this a few more times, so adding a test target to our makefile seems like a good idea:

.PHONY: test
test: hello.so
	bash -c 'enable -f ./hello.so hello; hello'

By having test depend on hello.so, we can just invoke make test to test our builtin.

We also need to tell make that test is not an actual file, so that make will always build this target. This is what the .PHONY: test directive achieves:

bash-plugin $ make test --quiet
hello, world

Setting a variable in bash

Not having a manual or tutorials detailing how to program bash itself, we'll have to resort to reading header files.

Having a rough idea of what we want, this should be too difficult.

bash-plugin $ grep -F set /usr/include/bash/variables.h | grep variable
/* Reinitialize some special variables that have external effects upon unset
   variable is set. */
bash-plugin $ grep -F bind /usr/include/bash/variables.h | grep variables
                                   bind_variable. */
extern SHELL_VAR *bind_variable __P((const char *, char *, int));
extern SHELL_VAR *bind_global_variable __P((const char *, char *, int));
extern SHELL_VAR *bind_variable_value __P((SHELL_VAR *, char *, int));
extern SHELL_VAR *bind_int_variable __P((char *, char *, int));
extern int unbind_variable __P((const char *));
extern int check_unbind_variable __P((const char *));
extern int unbind_variable_noref __P((const char *));

This looks much better! Let's try bind_global_variable:

/* Called when `hello' is enabled and loaded from the shared object.  If this
   function returns 0, the load fails. */
int
hello_builtin_load (name)
     char *name;
{
  bind_global_variable("GREETING", "hello, world", 0);
  return (1);
}

Now a variable GREETING should be set to hello, world. Changing our test, we can verify this:

.PHONY: test
test: hello.so
	bash -c 'enable -f ./hello.so hello; printf "GREETING=%s\n" "$$GREETING"; hello'

Note the $$ to prevent make from treating $GREETING as a make variable.

Our test is successful:

bash-plugin $ make test
gcc -I /usr/include/bash/builtins -I /usr/include/bash/include -I /usr/include/bash -I /usr/lib/bash -shared -fPIC -o hello.so hello.c
bash -c 'enable -f ./hello.so hello; printf "GREETING=%s\n" "$GREETING"; hello'
GREETING=hello, world
hello, world

With this change in place, we can change hello to display the value of GREETING:

int
hello_builtin (list)
     WORD_LIST *list;
{
  SHELL_VAR * greeting = find_variable("GREETING");
  if (greeting == NULL) {
    printf("hello, world\n");
    return EXECUTION_SUCCESS;
  }

  printf("%s\n", get_variable_value(greeting));

  return EXECUTION_SUCCESS;
}

The functions find_variable and get_variable_value were found by inspecting variables.h.

After rebuilding the plugin we can play around with this more interesting version:

bash-plugin $ enable -f ./hello.so hello
bash-plugin $ hello
hello
bash-plugin $ GREETING=foo
bash-plugin $ hello
foo

Where to go from here

While the hello builtin in itself is not particularly interesting, we've covered the full build process and some basic bash functions in this article.

The code for a skeleton bash plugin is available here: https://github.com/dhamidi/bash-plugin-hello

Since plugins have access to all internal bash APIs, they can do things that normal bash functions and eval cannot do.

Some ideas for what to do with these powers:

bind to a JSON parser library and allow parsing JSON by dynamically defining corresponding bash variables for each JSON entry (similar to what recutils offer),
create a builtin that allows defining "dynamic" variables like bash's $RANDOM,
add an JSON-RPC server exposing bash internals so that other processes can directly access them without the need for another plugin,
hook into the parser to allow substitutions to use another interpreted language,
hook into cd to parse .env file to set local variable definitions,
create dynamic variables that are persisted in an SQLite database.