Install goblin-slurp — goblin-slurp

What you need

goblin-slurp is a host-local, latency-sensitive daemon. Build it on the machine that will run it.

A C++23 compiler: recent Clang or GCC
CMake 3.25 or newer
simdjson
Boost 1.81 or newer
OpenSSL development headers
POSIX threads

Goblin Core is the primary runtime target. Its native shared-memory ringbuffer is the recommended low-latency path from goblin-slurp into the Redis-compatible server. The Goblin Core client headers, generated SBE codecs, and fast_float are already vendored under third_party/goblin-core, so you do not need a separate source checkout to compile the default build. RESP2 over TCP or a Unix-domain socket remains available as a compatibility path to another Redis-compatible server.

The build enables -march=native and -mtune=native. A binary built on one CPU generation may not run on an older machine. Build on the deployment host or on a machine with the same instruction set.

Install build dependencies

macOS with Homebrew

brew install cmake simdjson boost openssl@3

Apple Clang comes with Xcode Command Line Tools. Install them first if c++ --version is unavailable:

xcode-select --install

Debian or Ubuntu

Use a release whose repositories provide CMake 3.25 or newer and Boost 1.81 or newer:

sudo apt update
sudo apt install build-essential cmake libsimdjson-dev libboost-dev libssl-dev

Check the versions before configuring:

c++ --version
cmake --version

If the distribution packages are older than the minimums, install a current toolchain through the distribution's supported backports or the upstream CMake packages.

Get the source

git clone https://github.com/adamdeprince/goblin-slurp.git
cd goblin-slurp

If you already have this checkout, run the remaining commands from its root.

Configure a release build

The default build uses the vendored goblin-core client:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release

CMake reports the goblin-core revision it found. A normal line looks like:

text

-- goblin-core: vendored (v0.10.0-1-g17857c7)

To compile against a live goblin-core checkout instead, point CMake at that checkout explicitly:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
  -DGOBLIN_CORE_ROOT=/path/to/goblin-core

Use the checkout that matches the server you will deploy. The shared-memory ring ABI and SBE schema are versioned contracts.

Build and test

cmake --build build --parallel
ctest --test-dir build --output-on-failure

The roundtrip test needs a live goblin-core binary. It exits as skipped when one is unavailable. To include it, provide the binary:

GOBLIN_CORE_BIN=/path/to/goblin-core \
  ctest --test-dir build --output-on-failure

Run the independent Python decoder cross-check as well:

python3 -m pytest reference/test_golden.py

The test requires pytest. It is a verification dependency, not a runtime dependency.

Verify the binary

./build/goblin-slurp --version
./build/goblin-slurp --help
./build/goblin-slurp --type stocks --dry-run IBM AMD SPY

--version reports the goblin-slurp release, the goblin-core revision used at build time, the ring ABI, both SBE schemas, and the compiled transports. --dry-run resolves the feed and subscriptions without contacting Massive or a Redis server, so it does not need an API key.

Install the binaries

For a user-local installation:

cmake --install build --prefix "$HOME/.local"
"$HOME/.local/bin/goblin-slurp" --version

For a system installation:

sudo cmake --install build --prefix /usr/local
/usr/local/bin/goblin-slurp --version

The install also includes:

goblin-jsonl-dump, a raw Redis Pub/Sub-to-JSONL capture process
gs-latency-monitor, a live latency monitor
public goblin-slurp headers
share/goblin-slurp/market_data.xml, the market-data SBE schema

Optional transport builds

The normal build includes the verified shared-memory ring, TCP, and Unix-domain socket transports. RDMA and ExaSock are compile-time options:

cmake -S . -B build-rdma -DCMAKE_BUILD_TYPE=Release \
  -DGOBLIN_SLURP_RDMA=ON

cmake -S . -B build-exasock -DCMAKE_BUILD_TYPE=Release \
  -DGOBLIN_SLURP_EXASOCK=ON

These paths require their platform SDKs and hardware. They are wired through but marked unverified by this project; the daemon prints that status at runtime.

Keep goblin-core compatibility visible

There are two supported ways to update the client side:

Configure with -DGOBLIN_CORE_ROOT=/path/to/goblin-core to build directly against a checkout.
Refresh the vendored snapshot:

python3 tools/vendor_goblin_core.py /path/to/goblin-core
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel
ctest --test-dir build --output-on-failure

Always re-run goblin-slurp --version after an upgrade. A stale ring ABI is a hard compatibility failure, not a performance detail.

Installation troubleshooting

Symptom	What to check
`CMake 3.25 or higher is required`	Install a newer CMake and remove the incomplete build directory before configuring again.
`simdjson not found`	Install the simdjson development package and make sure its prefix is visible to CMake.
`Could NOT find Boost`	Confirm Boost 1.81 or newer is installed. Set `BOOST_ROOT` only when it lives outside the platform's normal prefix.
`Could NOT find OpenSSL`	Install development headers. On Homebrew, CMake normally finds `openssl@3`; otherwise pass its prefix through `CMAKE_PREFIX_PATH`.
`goblin-core headers missing`	Restore `third_party/goblin-core`, use `-DGOBLIN_CORE_ROOT`, or run the vendoring tool against a valid checkout.
Binary fails with an illegal instruction	Rebuild on the target host; the project intentionally compiles for the build machine's CPU.
`roundtrip` is skipped	Set `GOBLIN_CORE_BIN` to a built goblin-core executable. The remaining unit tests still run.

Continue with SETUP.md to connect a Massive feed and observe the first internal Redis Pub/Sub message.