Set up goblin-slurp

Before you connect

You need:

• A Massive.com API key with authorized WebSocket access for the asset class you intend to consume
• A built or installed goblin-slurp binary
• Either Goblin Core for the shared-memory ringbuffer path, or a reachable Redis-compatible TCP or Unix-domain socket
• A downstream Redis Pub/Sub client

Finish INSTALL.md first if the binary is not built.

goblin-slurp is designed for internal routing into infrastructure you control. It is not a market-data redistribution service and does not grant data rights. Keep the feed and every downstream consumer within the entitlements and terms of your Massive agreement.

One slurper owns one feed. A feed is the WebSocket host plus one asset class. Use separate processes for stocks and crypto, or for real-time and delayed stocks. Massive may permit only one connection to a given feed, so do not try to scale one feed by starting duplicate readers.

Choose the Goblin Core or Redis path

Goblin Core is the primary target. Its native shared-memory ringbuffer is the recommended low-latency ingest path; consumers can then use its familiar Redis-compatible Pub/Sub interface. The server transport and the payload delivered to internal subscribers remain independent choices.

Primary: shared-memory ringbuffer to Goblin Core

This is the default and lowest-overhead local path. Start Goblin Core with both a ringbuffer for goblin-slurp and a Redis-compatible listener for internal subscribers:

goblin-core \
  --tcp-listen 127.0.0.1:6379 \
  --ring /tmp/goblin-slurp 8mb \
  --unsolicited-output-buffer-bytes 4mb

The ringbuffer path matches goblin-slurp's default GOBLIN_RING_PATH=/tmp/goblin-slurp. The larger unsolicited output buffer is important for Pub/Sub: overflowing it disconnects a slow subscriber.

Compatibility: RESP2 over TCP

For an ordinary Redis-compatible endpoint, bypass the Goblin Core ringbuffer:

./build/goblin-slurp \
  --transport tcp \
  --target 127.0.0.1:6379 \
  --type stocks IBM AMD SPY

The default wire is RESP2, so each event uses the normal binary-safe Redis command PUBLISH channel payload. Here, PUBLISH is the literal name of the internal Redis fan-out command; it does not mean goblin-slurp redistributes the feed publicly. goblin-slurp currently has no Redis AUTH option, so use a trusted internal endpoint or enforce access at the network and server boundary.

Compatibility: RESP2 over a Unix-domain socket

./build/goblin-slurp \
  --transport unix \
  --target /run/goblin-core/goblin.sock \
  --type stocks IBM AMD SPY

Use a socket path that the goblin-slurp process can read and write.

Provide the Massive credential

Export the key in the process environment:

export MASSIVE_API_KEY="your-key-here"

The key is required for a live connection. It is deliberately not accepted as a command-line flag, keeping it out of shell history and process listings.

For a service manager, load the value from a protected environment or credential file. Do not copy it into the unit's ExecStart command.

Preview the resolved plan

Run a dry check before opening either connection:

./build/goblin-slurp \
  --type stocks \
  --streams trades,quotes \
  --dry-run \
  IBM AMD SPY

--dry-run prints the Massive endpoint, event subscriptions, Redis channel prefixes, transport target, wire protocol, payload format, queue policy, and CPU placement. It does not need MASSIVE_API_KEY.

Use a quoted wildcard when you explicitly want an entire feed:

./build/goblin-slurp --type stocks --dry-run '*'

The quotes prevent your shell from expanding * into filenames.

Start a feed

With Goblin Core listening on the default ringbuffer:

export MASSIVE_API_KEY="your-key-here"
./build/goblin-slurp --type stocks IBM AMD SPY

Or request the complete eligible feed:

./build/goblin-slurp --type crypto '*'

By default the event bus requests trades and quotes where both exist, routes lossless JSON, sends RESP2 commands, reports statistics every 10 seconds, and reconnects retryable WebSocket failures with backoff.

Pick an asset class

Examples:

./build/goblin-slurp --type stocks '*'
./build/goblin-slurp --type options O:SPY261218C00700000
./build/goblin-slurp --type options --trades-only '*'
./build/goblin-slurp --type crypto '*'
./build/goblin-slurp --type forex '*'
./build/goblin-slurp --type futures --delayed '*'
./build/goblin-slurp --type indices '*'

Real-time is the default. --delayed selects Massive's 15-minute delayed host for any asset class.

Select event streams

Without a stream flag, goblin-slurp asks for trades and quotes where they exist. Choose a smaller or different set with --streams:

./build/goblin-slurp \
  --type stocks \
  --streams trades,quotes,agg-minute,agg-second,luld \
  IBM

./build/goblin-slurp \
  --type crypto \
  --streams agg-minute,agg-second \
  BTC-USD ETH-USD

Available names are:

• trades
• quotes
• agg-minute
• agg-second
• luld, for stocks only

--trades-only is an alias for --streams trades, and --quotes-only is an alias for --streams quotes. Do not combine either alias with --streams. Forex and indices have no trade tape. A request for an unsupported explicit stream fails before connecting.

Choose JSON or SBE payloads

JSON is the default:

./build/goblin-slurp --type stocks IBM

Each payload is the individual Massive object, not the WebSocket array. Its original bytes are preserved and three ingest fields are appended:

Enable fixed-layout SBE market-data payloads with:

./build/goblin-slurp --type stocks --sbe-out IBM

Channel names stay identical. The SBE schema is sbe/market_data.xml, and reference/codec.py is an independent Python decoder.

Choose the Redis command wire

--wire controls how goblin-slurp sends the Redis PUBLISH command to Goblin Core. PUBLISH is Redis protocol terminology for internal fan-out; it does not mean the feed is made public. This option does not control the payload that subscribers receive.

For example, this is binary on both layers:

./build/goblin-slurp \
  --type crypto \
  --wire sbe \
  --sbe-out \
  '*'

Use --wire sbe only with a compatible Goblin Core target. Normal Redis servers understand RESP2, not the Goblin Core SBE envelope.

Subscribe from Redis

For JSON output, inspect internal messages with redis-cli:

redis-cli --raw PSUBSCRIBE 'T:*' 'Q:*'

The supplied Python subscriber understands both JSON and SBE payloads:

python3 reference/subscriber.py \
  --port 6379 \
  --patterns 'T:*' 'Q:*'

Any Redis client can subscribe. The useful routing contract is the channel:

• Subscribe to an exact instrument such as T:IBM.
• Pattern-subscribe to a family such as Q:*.
• Expect exactly one event in each Pub/Sub message.
• Treat the payload as opaque bytes until you choose the JSON or SBE decoder.

Tune the live process

Start with the defaults and watch the periodic statistics. Then tune only from measured evidence.

The frame queue drops the oldest frame when full so the socket can keep draining. Drops are counted and reported. A growing drop count means the Goblin Core output path cannot keep up; reduce the subscription, fix the server bottleneck, or measure CPU and NUMA placement.

On Linux, keep the feed and Goblin Core output threads on different physical cores near the NIC and Goblin Core memory. The two CPU flags are rejected when they name the same CPU.

Run under a service manager

The checked-in systemd/ files document a real deployment, but they contain host-specific usernames, paths, CPUs, NUMA policy, and capture filenames. Do not install them unchanged.

For a production unit:

• Order goblin-slurp after the network and Redis-compatible server.
• Load MASSIVE_API_KEY from a protected environment or credential file.
• Give each asset class its own process and, for rings, its own ring path.
• Keep Restart=always with a short delay; goblin-slurp applies its own WebSocket reconnect backoff.
• Stop the old process before starting a replacement for the same feed.
• Preserve periodic stderr statistics in the service journal.
• Increase the server's Pub/Sub output allowance for the expected fan-out.

Environment reference

The CLI --realtime, --delayed, and --url options resolve the WebSocket endpoint explicitly.

Runtime troubleshooting

For the complete wire schema, data-type rationale, parser behavior, live verification record, and operational experiments, continue into README.md.