ecluse:ecluse-core
Safe HaskellNone
LanguageGHC2021

Ecluse.Core.Queue

Description

The mirror-queue handle: the durable hand-off from the request path to the mirror worker.

Mirroring is demand-driven: when a client fetches an artifact whose version passes the rules, the proxy enqueues a MirrorJob and serves the artifact immediately, never blocking on the mirror. A separate worker receives jobs, fetches and verifies the artifact, publishes it to the mirror target, and acks the job (see docs/architecture/cloud-backends.md → "Mirror Queue").

The queue is the one cloud surface with materially different APIs per provider (AWS SQS SendMessage/ReceiveMessage+visibility-timeout/DeleteMessage; GCP Pub/Sub Publish/Pull+ack-deadline/Acknowledge), so it is its own handle -- a record of functions (the Handle pattern). Both providers fit the same receive → process → ack shape; their differences (visibility timeout vs ack deadline, batch limits, dead-letter wiring) stay behind the handle, and ReceiptHandle is opaque so neither leaks.

Like the other handles, the effectful fields return IO, not App, so an adapter stays decoupled from the proxy's Env/App (see docs/architecture/technology-stack.md → "Key Decisions").

Conventions

The two cloud backends both give at-least-once delivery, which is safe here because publishing is idempotent (a registry treats versions as immutable). The handle's contract reflects that:

  • enqueue is best-effort. It runs on the request hot path (enqueue, then serve immediately), so a failure must be logged/metered and __never fail the client response__ -- the artifact is already served, and a later pull re-enqueues.
  • Retry is "don't ack". A job that fails processing transiently (a flaky fetch, a registry blip) is simply not acked; the visibility timeout / ack deadline redelivers it, and it may succeed next time. There is deliberately no nack for the transient case.
  • deadLetter is the terminal terminus. A job that can never succeed (an artifact past the plan-sized byte cap) is a terminal verdict, and each backend realises it its own way: the in-memory backend drops the delivery (its only terminus), the SQS backend returns the message with a backoff visibility timeout without deleting it, so it rides the operator's redrive policy to the dead-letter queue for forensic retention rather than being silently discarded. This is not a nack (a retry) and not an ack (a clean retire); it is the third, terminal outcome.
  • extendVisibility lets the worker hold a long publish (a large artifact) past the visibility window. It is an optimization, not correctness-critical, since idempotency already makes redelivery harmless.

This module provides the handle, its payload types, and the building blocks a backend implementation reaches for; the STM-backed bounded, best-effort production backend mirroring rolls over to when no ECLUSE_QUEUE__URL is set lives in Ecluse.Core.Queue.Memory.

It also provides newEnqueueBuffer, a bounded producer-side hand-off buffer wrapped in front of any backend so the serve path's enqueue completes in microseconds while a composition-root drain loop delivers to the (possibly slow) backend off the request path.

Synopsis

Queue handle

data MirrorQueue Source #

The mirror-queue handle -- a record of functions over a backend whose private state the closures capture. See the module header for the enqueue / don't-ack-to-retry / no-nack conventions; all fields are IO, and each reports its backend failures as a QueueFault value, so no queue outage ever rides the exception channel through a caller.

Constructors

MirrorQueue 

Fields

  • enqueue :: MirrorJob -> IO (Either QueueFault ())

    Producer. Best-effort: runs on the request hot path, so a Left is counted/logged by the caller and never fails the client response (see the header); the lost job is re-enqueued on the next demand for its artifact.

  • receive :: IO (Either QueueFault [QueueMessage])

    Consumer. One long-poll for a batch of messages; Right [] on timeout (an empty, healthy poll), so the worker loop simply polls again. A Left is a failed poll: the worker logs it and backs off, and -- unlike an empty poll -- it does not advance the liveness heartbeat, so a persistently failing backend still surfaces through /livez.

  • ack :: ReceiptHandle -> IO (Either QueueFault ())

    Acknowledge a processed message so it is not redelivered. Not acking is how a failed job is retried (the header's "retry is don't ack"), so a Left here is absorbed after logging: the processed message redelivers, and idempotent publishing makes the repeat harmless.

  • extendVisibility :: ReceiptHandle -> Seconds -> IO (Either QueueFault ())

    Extend a received message's visibility window to hold a long publish. An optimization, not correctness-critical (redelivery is harmless), so a Left is absorbed silently by the caller.

  • deadLetter :: ReceiptHandle -> IO (Either QueueFault ())

    Realise a terminal fault: a job that can never succeed (an artifact past the plan-sized byte cap), decided as a verdict at the read site. Each backend routes it to its own dead-letter terminus -- the in-memory backend drops the delivery (its only terminus; observability is the worker's log and metric), the SQS backend returns the message with a backoff visibility timeout __without deleting it__, so it rides the operator's redrive policy to the dead-letter queue rather than being silently discarded. Distinct from ack (a clean retire) and from not-acking (a transient redelivery); see the header's terminus convention. A Left is absorbed after logging, like ack.

noMirrorQueue :: MirrorQueue Source #

The inert queue a deployment with zero mirroring mounts carries, so the composition-root Env keeps its total shape without a backend. It is unreachable by construction (no serve path enqueues on a mount that never mirrors, and no worker runs to poll it); reached anyway, enqueue is a typed, counted refusal -- never a crash -- and receive is the empty healthy poll.

Faults

data QueueFault Source #

Why a queue operation could not be delivered to the backend, reported as a value on every handle field: the closed transport cause a consumer branches on, and the backend's rendered detail for its log line. The cause vocabulary is Ecluse.Core.Fault's (TransportCause); a cloud backend's service-level refusal (a throttle, an access denial) classifies as TransportProtocol with the service detail carried. Build one by adopting an already-classified transport fault (transportFault) with queueTransportFault, so the detail stays bounded.

Every fault is safe to absorb under the handle's contract: an enqueue fault is the documented best-effort loss (re-enqueued on the next demand), a receive fault is a failed poll (retried after backoff), and an ack or visibility fault just means the message redelivers (idempotent). The typed channel exists so each caller makes that absorption decision explicitly, with the cause in hand.

Constructors

QueueFault 

Fields

  • qfCause :: TransportCause

    The closed classification a consumer or an operator reads.

  • qfDetail :: Text

    The backend's rendered detail, bounded to a log-line-sized budget. Diagnostic text only: it is never parsed, and no decision may branch on it.

Instances

Instances details
Show QueueFault Source # 
Instance details

Defined in Ecluse.Core.Queue

Eq QueueFault Source # 
Instance details

Defined in Ecluse.Core.Queue

queueTransportFault :: TransportFault -> QueueFault Source #

Adopt an already-classified TransportFault (an adapter edge's classification of its client library's exception) as a QueueFault. The TransportFault side (transportFault) truncates the detail to the shared log-line budget, so the two vocabularies cannot drift on what "bounded" means.

Payloads

data MirrorJob Source #

A mirror job: everything the worker needs to back-fill one artifact into the mirror target. The version was gated by the rules at serve time (when the job was enqueued); the worker re-evaluates current policy through the same shared admission oracle before mirroring (see Ecluse.Core.Worker.Job), then fetches the bytes, verifies them against the digests of the artifact that re-evaluation re-admitted, and publishes.

The queue payload is a trust boundary, so it carries __selection keys, never authority__: the filename (jobArtifactFilename) names the artifact the worker's ingest re-evaluation selects and gates under current policy, and the payload carries no digest or size at all -- the descriptor the tamper gate and the publish document consume (MirrorArtifact) is derived entirely from the artifact that re-evaluation re-admits.

Constructors

MirrorJob 

Fields

  • jobPackage :: PackageName

    The package whose artifact is being mirrored.

  • jobVersion :: Version

    The specific version to mirror.

  • jobArtifactUrl :: RegistryUrl

    Where to fetch the artifact bytes from (the public upstream), carried as the validated https egress witness rather than bare text; the SQS wire decode re-forms it, since the queue payload is a trust boundary.

  • jobArtifactFilename :: Text

    The serve-time-admitted artifact's filename: the selection key the worker's ingest re-evaluation gates by, cross-checked against current metadata by the shared admission gate rather than trusted.

  • jobTraceContext :: Maybe RemoteSpanContext

    The trace context of the serve-time span that enqueued the job, captured at enqueue time so the worker's per-job span can link back to the request that produced the work across the asynchronous hop. Nothing when tracing was off at enqueue time (or for a job from a producer that carried none). The queue treats it as opaque transport; only the tracing port reads it.

Instances

Instances details
Show MirrorJob Source # 
Instance details

Defined in Ecluse.Core.Queue

Eq MirrorJob Source # 
Instance details

Defined in Ecluse.Core.Queue

data RemoteSpanContext Source #

A serialised W3C trace-context carrier riding on a MirrorJob: the traceparent (and any tracestate) of the span that enqueued the job, in the standard wire encoding. It is captured at enqueue time and read back by the worker's tracing port to re-establish a span link from the per-job span to the enqueueing request, so the asynchronous mirror hand-off is navigable in a trace.

The two fields are the W3C header values verbatim; the queue carries them opaquely (it neither parses nor validates them -- an unparseable carrier simply yields no link), so this type names what is carried without coupling the queue to any tracing backend.

Constructors

RemoteSpanContext 

Fields

  • rscTraceparent :: Text

    The W3C traceparent header value of the enqueueing span.

  • rscTracestate :: Text

    The W3C tracestate header value (possibly empty) carried alongside, so vendor trace state survives the hop.

data QueueMessage Source #

A received message: the MirrorJob to process together with the ReceiptHandle used to ack it (or extendVisibility on it) once processed.

Constructors

QueueMessage 

Fields

Instances

Instances details
Show QueueMessage Source # 
Instance details

Defined in Ecluse.Core.Queue

Eq QueueMessage Source # 
Instance details

Defined in Ecluse.Core.Queue

Opaque receipt

data ReceiptHandle Source #

An opaque handle identifying a received message for ack / extendVisibility. It carries the backend's own delivery token -- an SQS receipt handle or a Pub/Sub ackId -- as text; the constructor is hidden so neither provider's representation leaks into worker code, and a handle is only ever obtained from a QueueMessage returned by receive. Build one (in a backend) with mkReceiptHandle and read the token back with unReceiptHandle.

mkReceiptHandle :: Text -> ReceiptHandle Source #

Wrap a backend's delivery token (an SQS receipt handle, a Pub/Sub ackId) as an opaque ReceiptHandle. For backend implementations only -- worker code obtains handles from receive, never builds them.

unReceiptHandle :: ReceiptHandle -> Text Source #

Recover the backend's delivery token from a ReceiptHandle, to pass back to the backend on ack / extendVisibility. For backend implementations only.

Durations

newtype Seconds Source #

A duration in whole seconds, for extendVisibility. A 'newtype' so a raw Int of seconds is never confused with some other count.

Constructors

Seconds Int 

Instances

Instances details
Show Seconds Source # 
Instance details

Defined in Ecluse.Core.Queue

Eq Seconds Source # 
Instance details

Defined in Ecluse.Core.Queue

Methods

(==) :: Seconds -> Seconds -> Bool #

(/=) :: Seconds -> Seconds -> Bool #

Ord Seconds Source # 
Instance details

Defined in Ecluse.Core.Queue

Backend building blocks

writeOrDrop :: TBQueue MirrorJob -> TVar Int -> MirrorJob -> STM (Maybe Int) Source #

Hand a job to a bounded queue within the caller's transaction: write it when there is room, or drop it at the cap (drop-newest) and return the incremented running drop total for the caller's report policy. Dropping rather than blocking keeps the producer non-blocking, and the loss is safe: a dropped job is re-enqueued on the next demand for its artifact. A backend building block, shared by the bounded in-memory backend (Ecluse.Core.Queue.Memory) and newEnqueueBuffer's hand-off so the two cannot drift on the drop policy.

reportWorthy :: Int -> Int -> Bool Source #

Whether the n-th event in a rate-limited series should be reported: the first (n == 1), then every interval-th. Shared by the bounded queue's drop reporting and the composition root's enqueue-buffer reporting so the two cannot drift.

Buffered producer hand-off

newEnqueueBuffer Source #

Arguments

:: Int

Buffer depth: how many undelivered jobs the hand-off retains before dropping the newest.

-> (Int -> IO ())

Invoked on every hand-off drop, with the running drop total.

-> (Int -> Text -> IO ())

Invoked on every backend delivery failure, with the running failure total and the failure's detail.

-> MirrorQueue

The backend whose enqueue is being decoupled from its callers.

-> IO (MirrorQueue, IO ()) 

Wrap a bounded producer-side hand-off buffer in front of a queue, so the serve path's enqueue is an in-process STM write (microseconds) no matter how slow the backend's own producer call is.

The motivating case is the SQS backend: its enqueue is an HTTP round trip (SendMessage), and the serve path runs the mirror enqueue after the response body has been sent but before the handler returns -- so on a keep-alive connection those milliseconds hold the connection's turn and tax the next request on it. Buffered, the handler hands the job off and returns; the returned drain loop -- which the composition root runs alongside the server -- delivers buffered jobs to the backend at the backend's own pace. The consumer fields (receive, ack, extendVisibility) pass through untouched.

Loss stays safe, so the buffer keeps the handle's best-effort producer contract (mirroring is demand-driven: a lost job is re-enqueued on the next demand for its artifact -- the same argument Ecluse.Core.Queue.Memory's bounded backend makes):

  • Drop-newest on overflow. A hand-off finding the buffer full drops the job and invokes onDrop with the running drop total. The callback fires on every drop (metric-grade); the caller owns any log rate-limiting.
  • A backend failure inside the drain loop invokes onDeliveryFailure with the running failure total and the failure's detail, then the loop backs off (bounded, growing with consecutive failures) before the next job so a persistently-unreachable backend is retried at a bounded rate rather than hot-looping; the failed job is not redelivered here, and the monotonic failure count is the operator's degraded-hand-off surface.
  • Cancellation loses the buffer. The drain loop never returns, so the composition root races it against the services; shutdown cancels it and any still-buffered jobs are dropped -- the same safe loss.

The wrapped enqueue never fails: it is always Right () (a drop is the documented safe loss, reported through onDrop, not a fault), so the never-fails producer contract is visible in the type.