| Safe Haskell | None |
|---|---|
| Language | GHC2021 |
Ecluse.Core.Queue
Description
The mirror-queue handle: the durable hand-off from the request path to the mirror worker.
Mirroring is demand-driven: when a client fetches an artifact whose version
passes the rules, the proxy enqueues a MirrorJob and serves the artifact
immediately, never blocking on the mirror. A separate worker receives jobs,
fetches and verifies the artifact, publishes it to the mirror target, and acks
the job (see docs/architecture/cloud-backends.md → "Mirror Queue").
The queue is the one cloud surface with materially different APIs per provider
(AWS SQS SendMessage/ReceiveMessage+visibility-timeout/DeleteMessage; GCP
Pub/Sub Publish/Pull+ack-deadline/Acknowledge), so it is its own handle --
a record of functions (the Handle pattern). Both providers fit the same
receive → process → ack shape; their differences (visibility timeout vs ack
deadline, batch limits, dead-letter wiring) stay behind the handle, and
ReceiptHandle is opaque so neither leaks.
Like the other handles, the effectful fields return IO, not App, so an
adapter stays decoupled from the proxy's Env/App (see
docs/architecture/technology-stack.md → "Key Decisions").
Conventions
The two cloud backends both give at-least-once delivery, which is safe here because publishing is idempotent (a registry treats versions as immutable). The handle's contract reflects that:
enqueueis best-effort. It runs on the request hot path (enqueue, then serve immediately), so a failure must be logged/metered and __never fail the client response__ -- the artifact is already served, and a later pull re-enqueues.- Retry is "don't
ack". A job that fails processing transiently (a flaky fetch, a registry blip) is simply not acked; the visibility timeout / ack deadline redelivers it, and it may succeed next time. There is deliberately nonackfor the transient case. deadLetteris the terminal terminus. A job that can never succeed (an artifact past the plan-sized byte cap) is a terminal verdict, and each backend realises it its own way: the in-memory backend drops the delivery (its only terminus), the SQS backend returns the message with a backoff visibility timeout without deleting it, so it rides the operator's redrive policy to the dead-letter queue for forensic retention rather than being silently discarded. This is not anack(a retry) and not anack(a clean retire); it is the third, terminal outcome.extendVisibilitylets the worker hold a long publish (a large artifact) past the visibility window. It is an optimization, not correctness-critical, since idempotency already makes redelivery harmless.
This module provides the handle, its payload types, and the building blocks a
backend implementation reaches for; the STM-backed bounded, best-effort
production backend mirroring rolls over to when no ECLUSE_QUEUE__URL is set
lives in Ecluse.Core.Queue.Memory.
It also provides newEnqueueBuffer, a bounded producer-side hand-off buffer
wrapped in front of any backend so the serve path's enqueue completes in
microseconds while a composition-root drain loop delivers to the (possibly slow)
backend off the request path.
Synopsis
- data MirrorQueue = MirrorQueue {
- enqueue :: MirrorJob -> IO (Either QueueFault ())
- receive :: IO (Either QueueFault [QueueMessage])
- ack :: ReceiptHandle -> IO (Either QueueFault ())
- extendVisibility :: ReceiptHandle -> Seconds -> IO (Either QueueFault ())
- deadLetter :: ReceiptHandle -> IO (Either QueueFault ())
- noMirrorQueue :: MirrorQueue
- data QueueFault = QueueFault {}
- queueTransportFault :: TransportFault -> QueueFault
- data MirrorJob = MirrorJob {}
- data RemoteSpanContext = RemoteSpanContext {}
- data QueueMessage = QueueMessage {}
- data ReceiptHandle
- mkReceiptHandle :: Text -> ReceiptHandle
- unReceiptHandle :: ReceiptHandle -> Text
- newtype Seconds = Seconds Int
- writeOrDrop :: TBQueue MirrorJob -> TVar Int -> MirrorJob -> STM (Maybe Int)
- reportWorthy :: Int -> Int -> Bool
- newEnqueueBuffer :: Int -> (Int -> IO ()) -> (Int -> Text -> IO ()) -> MirrorQueue -> IO (MirrorQueue, IO ())
Queue handle
data MirrorQueue Source #
The mirror-queue handle -- a record of functions over a backend whose private
state the closures capture. See the module header for the enqueue /
don't-ack-to-retry / no-nack conventions; all fields are IO, and each
reports its backend failures as a QueueFault value, so no queue outage
ever rides the exception channel through a caller.
Constructors
| MirrorQueue | |
Fields
| |
noMirrorQueue :: MirrorQueue Source #
The inert queue a deployment with zero mirroring mounts carries, so the
composition-root Env keeps its total shape without a backend. It is unreachable
by construction (no serve path enqueues on a mount that never mirrors, and no
worker runs to poll it); reached anyway, enqueue is a typed, counted refusal --
never a crash -- and receive is the empty healthy poll.
Faults
data QueueFault Source #
Why a queue operation could not be delivered to the backend, reported as a
value on every handle field: the closed transport cause a consumer branches
on, and the backend's rendered detail for its log line. The cause vocabulary is
Ecluse.Core.Fault's (TransportCause); a cloud backend's service-level
refusal (a throttle, an access denial) classifies as TransportProtocol with
the service detail carried. Build one by adopting an already-classified
transport fault (transportFault) with queueTransportFault,
so the detail stays bounded.
Every fault is safe to absorb under the handle's contract: an enqueue fault is the documented best-effort loss (re-enqueued on the next demand), a receive fault is a failed poll (retried after backoff), and an ack or visibility fault just means the message redelivers (idempotent). The typed channel exists so each caller makes that absorption decision explicitly, with the cause in hand.
Constructors
| QueueFault | |
Fields
| |
Instances
| Show QueueFault Source # | |
Defined in Ecluse.Core.Queue Methods showsPrec :: Int -> QueueFault -> ShowS # show :: QueueFault -> String # showList :: [QueueFault] -> ShowS # | |
| Eq QueueFault Source # | |
Defined in Ecluse.Core.Queue | |
queueTransportFault :: TransportFault -> QueueFault Source #
Adopt an already-classified TransportFault (an adapter edge's
classification of its client library's exception) as a QueueFault. The
TransportFault side (transportFault) truncates the detail
to the shared log-line budget, so the two vocabularies cannot drift on what
"bounded" means.
Payloads
A mirror job: everything the worker needs to back-fill one artifact into the mirror target. The version was gated by the rules at serve time (when the job was enqueued); the worker re-evaluates current policy through the same shared admission oracle before mirroring (see Ecluse.Core.Worker.Job), then fetches the bytes, verifies them against the digests of the artifact that re-evaluation re-admitted, and publishes.
The queue payload is a trust boundary, so it carries __selection keys, never
authority__: the filename (jobArtifactFilename) names the artifact the worker's
ingest re-evaluation selects and gates under current policy, and the payload
carries no digest or size at all -- the descriptor the tamper gate and the publish
document consume (MirrorArtifact) is derived entirely from
the artifact that re-evaluation re-admits.
Constructors
| MirrorJob | |
Fields
| |
data RemoteSpanContext Source #
A serialised W3C trace-context carrier riding on a MirrorJob: the
traceparent (and any tracestate) of the span that enqueued the job, in the
standard wire encoding. It is captured at enqueue time and read back by the worker's
tracing port to re-establish a span link from the per-job span to the enqueueing
request, so the asynchronous mirror hand-off is navigable in a trace.
The two fields are the W3C header values verbatim; the queue carries them opaquely (it neither parses nor validates them -- an unparseable carrier simply yields no link), so this type names what is carried without coupling the queue to any tracing backend.
Constructors
| RemoteSpanContext | |
Fields
| |
Instances
| Show RemoteSpanContext Source # | |
Defined in Ecluse.Core.Queue Methods showsPrec :: Int -> RemoteSpanContext -> ShowS # show :: RemoteSpanContext -> String # showList :: [RemoteSpanContext] -> ShowS # | |
| Eq RemoteSpanContext Source # | |
Defined in Ecluse.Core.Queue Methods (==) :: RemoteSpanContext -> RemoteSpanContext -> Bool # (/=) :: RemoteSpanContext -> RemoteSpanContext -> Bool # | |
data QueueMessage Source #
A received message: the MirrorJob to process together with the
ReceiptHandle used to ack it (or extendVisibility on it) once processed.
Constructors
| QueueMessage | |
Fields
| |
Instances
| Show QueueMessage Source # | |
Defined in Ecluse.Core.Queue Methods showsPrec :: Int -> QueueMessage -> ShowS # show :: QueueMessage -> String # showList :: [QueueMessage] -> ShowS # | |
| Eq QueueMessage Source # | |
Defined in Ecluse.Core.Queue | |
Opaque receipt
data ReceiptHandle Source #
An opaque handle identifying a received message for ack /
extendVisibility. It carries the backend's own delivery token -- an SQS receipt
handle or a Pub/Sub ackId -- as text; the constructor is hidden so neither
provider's representation leaks into worker code, and a handle is only ever
obtained from a QueueMessage returned by receive. Build one (in a backend)
with mkReceiptHandle and read the token back with unReceiptHandle.
Instances
| Show ReceiptHandle Source # | |
Defined in Ecluse.Core.Queue Methods showsPrec :: Int -> ReceiptHandle -> ShowS # show :: ReceiptHandle -> String # showList :: [ReceiptHandle] -> ShowS # | |
| Eq ReceiptHandle Source # | |
Defined in Ecluse.Core.Queue Methods (==) :: ReceiptHandle -> ReceiptHandle -> Bool # (/=) :: ReceiptHandle -> ReceiptHandle -> Bool # | |
| Ord ReceiptHandle Source # | |
Defined in Ecluse.Core.Queue Methods compare :: ReceiptHandle -> ReceiptHandle -> Ordering # (<) :: ReceiptHandle -> ReceiptHandle -> Bool # (<=) :: ReceiptHandle -> ReceiptHandle -> Bool # (>) :: ReceiptHandle -> ReceiptHandle -> Bool # (>=) :: ReceiptHandle -> ReceiptHandle -> Bool # max :: ReceiptHandle -> ReceiptHandle -> ReceiptHandle # min :: ReceiptHandle -> ReceiptHandle -> ReceiptHandle # | |
mkReceiptHandle :: Text -> ReceiptHandle Source #
Wrap a backend's delivery token (an SQS receipt handle, a Pub/Sub ackId)
as an opaque ReceiptHandle. For backend implementations only -- worker code
obtains handles from receive, never builds them.
unReceiptHandle :: ReceiptHandle -> Text Source #
Recover the backend's delivery token from a ReceiptHandle, to pass back to
the backend on ack / extendVisibility. For backend implementations only.
Durations
A duration in whole seconds, for extendVisibility. A 'newtype' so a raw
Int of seconds is never confused with some other count.
Backend building blocks
writeOrDrop :: TBQueue MirrorJob -> TVar Int -> MirrorJob -> STM (Maybe Int) Source #
Hand a job to a bounded queue within the caller's transaction: write it when
there is room, or drop it at the cap (drop-newest) and return the incremented
running drop total for the caller's report policy. Dropping rather than blocking
keeps the producer non-blocking, and the loss is safe: a dropped job is
re-enqueued on the next demand for its artifact. A backend building block, shared
by the bounded in-memory backend (Ecluse.Core.Queue.Memory) and
newEnqueueBuffer's hand-off so the two cannot drift on the drop policy.
reportWorthy :: Int -> Int -> Bool Source #
Whether the n-th event in a rate-limited series should be reported: the first
(n == 1), then every interval-th. Shared by the bounded queue's drop reporting and
the composition root's enqueue-buffer reporting so the two cannot drift.
Buffered producer hand-off
Arguments
| :: Int | Buffer depth: how many undelivered jobs the hand-off retains before dropping the newest. |
| -> (Int -> IO ()) | Invoked on every hand-off drop, with the running drop total. |
| -> (Int -> Text -> IO ()) | Invoked on every backend delivery failure, with the running failure total and the failure's detail. |
| -> MirrorQueue | The backend whose |
| -> IO (MirrorQueue, IO ()) |
Wrap a bounded producer-side hand-off buffer in front of a queue, so the
serve path's enqueue is an in-process STM write (microseconds) no matter how slow
the backend's own producer call is.
The motivating case is the SQS backend: its enqueue is an HTTP round trip
(SendMessage), and the serve path runs the mirror enqueue after the response body
has been sent but before the handler returns -- so on a keep-alive connection those
milliseconds hold the connection's turn and tax the next request on it. Buffered,
the handler hands the job off and returns; the returned drain loop -- which the
composition root runs alongside the server -- delivers buffered jobs to the backend
at the backend's own pace. The consumer fields (receive, ack,
extendVisibility) pass through untouched.
Loss stays safe, so the buffer keeps the handle's best-effort producer contract (mirroring is demand-driven: a lost job is re-enqueued on the next demand for its artifact -- the same argument Ecluse.Core.Queue.Memory's bounded backend makes):
- Drop-newest on overflow. A hand-off finding the buffer full drops the job and
invokes
onDropwith the running drop total. The callback fires on every drop (metric-grade); the caller owns any log rate-limiting. - A backend failure inside the drain loop invokes
onDeliveryFailurewith the running failure total and the failure's detail, then the loop backs off (bounded, growing with consecutive failures) before the next job so a persistently-unreachable backend is retried at a bounded rate rather than hot-looping; the failed job is not redelivered here, and the monotonic failure count is the operator's degraded-hand-off surface. - Cancellation loses the buffer. The drain loop never returns, so the composition root races it against the services; shutdown cancels it and any still-buffered jobs are dropped -- the same safe loss.
The wrapped enqueue never fails: it is always Right () (a drop is the
documented safe loss, reported through onDrop, not a fault), so the never-fails
producer contract is visible in the type.