language benchmarks without too much effort put into them
Guess luajit is its own language now.
ooh, pathfinding.  thanks waraya!
add lua, remove shebangs

heads

tip
browse log

clone

read-only
https://hg.sr.ht/~icefox/casual-benchmarks
read/write
ssh://hg@hg.sr.ht/~icefox/casual-benchmarks

All models are wrong, some models are useful.

Okay I keep wishing for what the Programming Language Shootout (now aka the Benchmarks Game) used to be, a pile of order-of-magnitude benchmarks of various programming languages and implementations without too much bullshit in them. Now there's things in there that use identity for a hash function, things that use bleeding-edge algorithms or hand-written SIMD, or other bullshit like that; basically, they're massively overfitted.

So the goal is to try to make a set of idiomatic, casual benchmark programs that showcase basic operations, used in basic ways, with basic libraries. We want bog-standard, pretty reasonable code that doesn't do anything especially fancy, so we can see see what kind of performance we expect from the first pass of the program.

Actually, "casual benchmarks" is a pretty good name.

For example we might have:

  • Comparisons between different languages: Python and Rust hashtables
  • Comparisons between language implementations, like gcc and clang or cpython and pypy
  • simple comparisons between Rust's built-in hashtables and something like FNVHashMap (honestly this might end up being out of scope, it's too easy to go deep into the paint)

We will not have:

  • Comparisons that use profiled and hand-optimized code that isn't already part of some large common library like, say, numpy
  • Comparisons that use uncommon, high-power language features like Rust unsafe or C++ template metaprogramming more than needed to solve the problem
  • Comparisons that use SIMD more than the language implementations can do on its own
  • Comparisons that use multithreading without saying so -- they'll be treated as separate algorithms

The point of this is to have a vague idea of the performance you can expect from normal code.

See also: https://programming-language-benchmarks.vercel.app/ Probably best to just cherry-pick programs from that and make it generate actual graphs, tbh. The Benchmarks Game repo is also here: https://salsa.debian.org/benchmarksgame-team/benchmarksgame

Both are MIT licensed, so this should be too.

#Guidelines

Hmmm.

  • Implement the same algorithms, as far as one can
  • Implement the algorithms in the most natural way for each language, as far as one can. This doesn't mean to be obtuse or wasteful, or to avoid handy things the language gives you, just stop and think for a moment and ask yourself "am I being too tryhard here?"
  • Use the most normal datatypes, built into the language -- so some dictionary-heavy benchmark might use a hashmap in Rust or a binary tree of some kind in Haskell.
  • Use normal heckin' optimization settings without annotations, appropriate for release. So on Rust this would be --release without trying to fiddle with LTO at all or twist other knobs, for C this would be -O2 or maybe -O3 but again no knobs beyond that.

#Languages I have

  • C/gcc
  • Rust
  • Crystal (incomplete)
  • Python (incomplete)
  • Lua/lua (incomplete)

#Languages I want

  • C/clang
  • Common Lisp/SBCL
  • C#
  • D
  • Elixir
  • Erlang
  • F#
  • Go
  • Hare
  • Java
  • Javascript/v8
  • Julia
  • Kotlin
  • LFE?
  • Lua/luajit
  • Nim
  • OCaml
  • Ruby
  • Zig

Suggestions welcome but also each new toolchain is gonna add a certain amount of work, mostly in the setup and installation of all these things on a particular system, so I reserve the right to say "sorry but I don't care that much".

#Problems I have

  • Hello world
  • n-body simulation
  • some circle math thing
  • dijkstra pathfinding on a randomized grid using a tree-based heap

#Problems I want

Maybe divide them into "number-crunching", "data-mongling" and "IO"? I want a representative sample of things beyond just heckin' KNN

#Contributors

  • icefox
  • wareya
  • wukong