Exporting Lots of Bash Functions in Environment

High level

How does BASH environment appear to behave with 10K+ sourced functions? -> Quite well.

Exported functions appear to start break things past about 15-20K exported functions (depending on the name length of functions).

Deeper dive

My BASH environment has a few thousand hand-rolled and now VIBED shell functions.

As I am continuously adding more, the question comes up: will I start hitting some limits where this strategy is going to give me pain in the future? And whether I should focus more on cleaning up functions that may have lost their use cases.

To test this, we can try to declare a bunch of functions:

main() {
  for i in {1..20000} ; do
    export VAR1="${i:?}"
    eval "function_name_${VAR1}(){
      echo \"Hello World - ${VAR1}\"
    }
    export -f function_name_${VAR1}
    "
  done
}

main "${@}"

When I try to source the above I start to get UGLY errors like:

-bash: /opt/homebrew/opt/coreutils/libexec/gnubin/basename: Argument list too long

When using functions like basename $PWD

Well that is of course not nice, to have core utility functions breaking on us.

However, if I try to do the same thing without exporting the function (without export -f), things look much happier.

main() {
  for i in {1..50000} ; do
    export VAR1="${i:?}"
    eval "function_name_${VAR1}(){
      echo \"Hello World - ${VAR1}\"
    }
    "
  done
}

main "${@}"

Environment works with having 50K+ functions in it. However, certain environment functions appear to have non linear scaling. For example:

With 10K functions in environment

compgen -ac | wc -l
10366

Getting a list of functions is around 100ms

time compgen -ac

real	0m0.137s
user	0m0.034s
sys	0m0.042s

But with 60K functions in environment

❯compgen -ac | wc -l
60366

Getting a list of functions is not 600ms but is about 6 seconds!

time compgen -ac > /dev/null

real	0m5.983s
user	0m5.600s
sys	0m0.059s

While 20K functions still perfroms well with getting a list of defined functions at respectable 400ms

m:mac d:vintrin-env b:master ○❯compgen -ac  | wc -l
20367
m:mac d:vintrin-env b:master ○❯time compgen -ac > /dev/null

real	0m0.386s
user	0m0.345s
sys	0m0.031s

End note

While I am sitting at about 10K functions in my environment right now, it appears I have another 10K functions I can add. And that should be plenty good amount of space. (As long as I don't export -f all of them).

Metadata

Above testing done on M2-PRO macbook with 10 cores, in iTerm using GNU bash, version 5.2.37