Compilation speed of CL implementations

Published on
Tags: lisp, performance

These last few days, I worked a lot on make-website, including support for both Clozure CL and ECL in addition to the de facto reference implementation that is SBCL.

One of the underlying goals was to see if I could speed it up, because while SBCL is the leading impl. concerning execution speed, its compilation speed suffers from all the optimizations. And I have good reasons to think that my bottleneck is the macro-expansion of all the spinneret tags and the associated GC stress, here.

So without further ado, here's how good (or bad) these three fare when used to (re)build this website (durations in seconds):

SBCLCCLECLSBCL's interpreter
From scratch70.133.0261.12.5
One doc changed0.60.42.20.0
No change0.10.20.20.0

Looks like putting in one hour to support CCL was worth it, but switching to SBCL's interpreter obsoleted all the others for this use case! Keep in mind that adding a new page or changing a title guarantees an almost full rebuild (because of the inlined and page-specific sidebar, can't use Iframes here), so this matters more than you may have thought.

Might ECL be hampered by its use of boehm-gc? Apparently, it's using gcc to compile everything, I must try to use its bytecode compiler but ECL seems to be currently broken on my machine… I'll update this page if I manage to fix it.

Benchmark details

  • Versions used: SBCL 2.5.4, CCL 1.13, ECL 24.5.10
  • CPU: AMD 5900X
  • RAM: 64 GB DDR4
  • OS: Gentoo Linux

For fairness, I added a global (optimize (compilation-speed 3)) declaration in make-website.lisp (where spinneret is called) and I tried giving each a heap size of 32 GiB and kept the best result (SBCL' interpreter got almost two times slower with it).

I also tried after rebuilding ECL with --enable-{gen,precise}gc, to no avail; it actually got ~10% slower, even with half the "gc count"…

Raw benchmark log
[sbcl] From scratch
sbcl --dynamic-space-size 32768 --script make.lisp
Evaluation took:
  70.088 seconds of real time
  67.258411 seconds of total run time (63.604077 user, 3.654334 system)
  [ Real times consist of 1.676 seconds GC time, and 68.412 seconds non-GC time. ]
  [ Run times consist of 1.676 seconds GC time, and 65.583 seconds non-GC time. ]
  95.96% CPU
  8,416 forms interpreted
  310,403 lambdas converted
  259,332,400,267 processor cycles
  35,628,858,640 bytes consed

[sbcl] One file touched
sbcl --dynamic-space-size 32768 --script make.lisp
Evaluation took:
  0.647 seconds of real time
  0.632841 seconds of total run time (0.536662 user, 0.096179 system)
  97.84% CPU
  28 forms interpreted
  2,845 lambdas converted
  2,398,210,352 processor cycles
  311,402,848 bytes consed

[sbcl] No change
sbcl --dynamic-space-size 32768 --script make.lisp
Evaluation took:
  0.097 seconds of real time
  0.081726 seconds of total run time (0.049787 user, 0.031939 system)
  84.54% CPU
  383 lambdas converted
  362,481,008 processor cycles
  31,102,912 bytes consed

[ccl] From scratch
</dev/null ccl -R 34359738368 -b -Q -l make.lisp
(LET ((*SRC-ROOT* #P"src/") ... (MAKE-WEBSITE))
took 32,960,000 microseconds (32.960000 seconds) to run.
      1,654,613 microseconds ( 1.654613 seconds, 5.02%) of which was spent in GC.
During that period, and with 24 available CPU cores,
     30,746,420 microseconds (30.746420 seconds) were spent in user mode
        810,946 microseconds ( 0.810946 seconds) were spent in system mode
 5,115,814,944 bytes of memory allocated.
 328,817 minor page faults, 0 major page faults, 0 swaps.

[ccl] One file touched
</dev/null ccl -R 34359738368 -b -Q -l make.lisp
(LET ((*SRC-ROOT* #P"src/") ... (MAKE-WEBSITE))
took 393,344 microseconds (0.393344 seconds) to run.
      40,406 microseconds (0.040406 seconds, 10.27%) of which was spent in GC.
During that period, and with 24 available CPU cores,
     356,686 microseconds (0.356686 seconds) were spent in user mode
      29,940 microseconds (0.029940 seconds) were spent in system mode
 52,122,400 bytes of memory allocated.
 12,137 minor page faults, 0 major page faults, 0 swaps.

[ccl] No change
</dev/null ccl -R 34359738368 -b -Q -l make.lisp
(LET ((*SRC-ROOT* #P"src/") ... (MAKE-WEBSITE))
took 168,695 microseconds (0.168695 seconds) to run.
      35,505 microseconds (0.035505 seconds, 21.05%) of which was spent in GC.
During that period, and with 24 available CPU cores,
     143,708 microseconds (0.143708 seconds) were spent in user mode
      20,069 microseconds (0.020069 seconds) were spent in system mode
 13,007,568 bytes of memory allocated.
 11,336 minor page faults, 0 major page faults, 0 swaps.

[ecl] From scratch
ecl --norc --quiet --heap-size 34359738368 --shell make.lisp
;;; Loading #P"/home/anon/.local/lib/quicklisp/setup.lisp"
;;; Loading #P"/usr/lib64/ecl-24.5.10/asdf.fas"
real time : 261.131 secs
run time  : 55.901 secs
gc count  : 296 times
consed    : 6167662912 bytes

[ecl] One file touched
ecl --norc --quiet --heap-size 34359738368 --shell make.lisp
;;; Loading #P"/home/anon/.local/lib/quicklisp/setup.lisp"
;;; Loading #P"/usr/lib64/ecl-24.5.10/asdf.fas"
real time : 2.165 secs
run time  : 0.532 secs
gc count  : 5 times
consed    : 99081392 bytes

[ecl] No change
ecl --norc --quiet --heap-size 34359738368 --shell make.lisp
;;; Loading #P"/home/anon/.local/lib/quicklisp/setup.lisp"
;;; Loading #P"/usr/lib64/ecl-24.5.10/asdf.fas"
real time : 0.187 secs
run time  : 0.198 secs
gc count  : 3 times
consed    : 51961600 bytes

[sbcl-interp] From scratch
sbcl --script make.lisp
Evaluation took:
  2.503 seconds of real time
  1.440874 seconds of total run time (0.899308 user, 0.541566 system)
  [ Real times consist of 0.050 seconds GC time, and 2.453 seconds non-GC time. ]
  [ Run times consist of 0.053 seconds GC time, and 1.388 seconds non-GC time. ]
  57.57% CPU
  1,876,713 forms interpreted
  170 lambdas converted
  9,263,076,688 processor cycles
  917,200,752 bytes consed

[sbcl-interp] One file touched
sbcl --script make.lisp
UPDATING TARGETS - 1ST PASS...
UPDATING TARGETS - 2ND PASS)...
Evaluation took:
  0.019 seconds of real time
  0.018245 seconds of total run time (0.014359 user, 0.003886 system)
  94.74% CPU
  15,898 forms interpreted
  91 lambdas converted
  67,733,976 processor cycles
  12,539,152 bytes consed

[sbcl-interp] No change
sbcl --script make.lisp
Evaluation took:
  0.003 seconds of real time
  0.002838 seconds of total run time (0.002838 user, 0.000000 system)
  100.00% CPU
  10,497,714 processor cycles
  556,320 bytes consed