BitCrafts Toolkit

Role

The bitcrafts-dev repository holds the bc-* ecosystem: six C libraries that form the foundation and three CLI tools built on top. The BitCrafts Vigil desktop frontend lives in its own repository (bitcrafts-vigil) and drives these tools. Pure C11, zero external runtime dependencies, baseline x86-64-v3 (AVX2 + BMI2 + FMA + LZCNT + MOVBE), GPLv3 (tools) / LGPLv3 (libraries).

Numbers at a glance

On a Ryzen 7 5700G (16 SMT threads, Linux 6.x, warm cache) against the Linux 6.12 + Node.js v22 source corpus (113k files, 2.2 GB):

bcduplicate peaks at 758k files/s — 18× rmlint on the same thread count. Full methodology, datasets, and reproduction scripts live under benchmarks/ in the repo.

Layout

bitcrafts-dev/
├── subprojects/        # libraries (LGPLv3)
│   ├── bc-core         # CPU primitives: hash, SIMD memory, checked arithmetic
│   ├── bc-allocators   # pool, arena, slab, context allocators
│   ├── bc-containers   # vector, hashmap, set, ring buffer, bitset
│   ├── bc-concurrency  # worker pool, lock-free MPMC queue, signal-safe shutdown
│   ├── bc-io           # streams, filesystem helpers, mmap, io_uring wrappers
│   └── bc-runtime      # lifecycle, layered config, structured logging, metrics, CLI
├── tools/              # CLI binaries (GPLv3)
│   ├── bc-hash         # recursive file-tree hashing (SHA-256 / xxh3 / BLAKE2b ...)
│   ├── bc-duplicate    # duplicate detection via size → fast-hash → full-hash funnel
│   └── bc-integrity    # JSONL filesystem manifests, verify, diff
└── benchmarks/         # comparator runs, datasets, hyperfine outputs

The GTK4 + libadwaita desktop app, BitCrafts Vigil, is maintained in a separate repository (bitcrafts-vigil) and depends on these tools at runtime.

Highlights

• Adaptive parallel I/O: io_uring batched reads with per-worker queues, fallback to threadpool. Adaptive dispatch decides serial vs parallel per workload — break-even at roughly 90 files or 1 MB on Zen 3.
• Lock-free MPMC queue (Vyukov sequence-based) at the heart of every parallel walk; zero shared mutable state between workers except through the queue and atomic counters.
• Hardware-accelerated hashing: SHA-NI for SHA-256, AVX2 paths selected at runtime. ISA-neutral public headers — no intrinsics leak across the API boundary.
• Reproducible benchmarks vs sha256sum, openssl, hashdeep, jdupes, rmlint, czkawka_cli, mtree. Numbers, datasets, and commands committed under benchmarks/.
• CI matrix: debug, ASan + UBSan, TSan, release. TSan is load-bearing — every parallel path is validated under it.

Install

Debian / Ubuntu, x86-64-v3 baseline (any Haswell-or-later CPU). One-time — trust the bit-crafts package registry:

sudo install -d /etc/apt/keyrings
curl -fsSL https://pkgs.bit-crafts.com/api/packages/bit-crafts/debian/repository.key \
  | sudo tee /etc/apt/keyrings/bit-crafts.asc >/dev/null
echo "deb [signed-by=/etc/apt/keyrings/bit-crafts.asc] https://pkgs.bit-crafts.com/api/packages/bit-crafts/debian trixie main" \
  | sudo tee /etc/apt/sources.list.d/bit-crafts.list
sudo apt update

Install the CLI tools — bchash, bcduplicate, bcintegrity land in /usr/bin, statically linked:

sudo apt install bitcrafts-tools-x86-v3

Building your own program against the bc-* libraries? Install the development package (public headers, .pc files, shared objects):

sudo apt install bitcrafts-dev-x86-v3

The registry index is signed; apt verifies it against the key fetched above (signed-by=), so no separate checksum/signature step is needed.

Build from source

Debian 13 (Trixie) baseline. liburing, xxHash and BLAKE3 are vendored via Meson wraps and statically linked in release builds — no system packages required for them.

git clone https://git.bit-crafts.com/bit-crafts/bitcrafts-dev.git
cd bitcrafts-dev
scripts/install-deps.sh build       # compiler, meson, cmocka
scripts/bx build release            # build all libs + tools
sudo meson install -C build/release

The scripts/bx wrapper exposes named build variants — debug, release, asan, tsan, ubsan, bench, coverage — and a matrix command iterates the default set.

Origins

The libraries and tools used to live in separate repositories — one per project, each with its own CI, its own version, its own packaging. Coordinating cross-cutting changes across that DAG became the bottleneck. In 2026-05 the ecosystem was consolidated into a single Meson workspace (meson.override_dependency lets each subproject resolve siblings without pkg-config), one CI pipeline, one source of truth — now self-hosted at git.bit-crafts.com/bit-crafts/bitcrafts-dev.

Older articles were written when each bc-* project was its own repo; the equivalent code now lives under bitcrafts-dev/subprojects/bc-X (libraries) or tools/bc-X (CLI binaries).