ecluse:ecluse-core
Safe HaskellNone
LanguageGHC2021

Ecluse.Core.Osv.Stream

Description

Streaming ingest of the osv.dev export archive Pilot compiles osv.db from.

The feed aggregates many upstream databases, so the bounds here are per entry: a drop is counted in IngestStats and the rest of the archive keeps flowing. ilMaxAdvisoryBytes applies before the bytes are retained and before the JSON decodes, and the refused entry drains to its boundary so the entries after it stay aligned. The aggregate verdict is systemicDrop, which the compiler reads once the stream completes.

Synopsis

Documentation

streamOsvUrl :: forall (m :: Type -> Type) i. (MonadResource m, MonadThrow m, KatipContext m) => Maybe TracerProvider -> Manager -> OsvIngest -> String -> ConduitT i ExtractedOsv m ContentDigest Source #

Fetch the OSV zip and stream its contents, bounded by ingest. It returns the digest of the whole response body.

parseOsvStream :: forall (m :: Type -> Type). (MonadResource m, MonadThrow m, KatipContext m) => Maybe TracerProvider -> OsvIngest -> ConduitT ByteString ExtractedOsv m () Source #

Parse the zip stream and emit ExtractedOsv, bounded by ingest.

Ingest bounds and drop accounting

newtype IngestLimits Source #

The per-advisory byte bound one ingest pass holds every zip entry to.

Constructors

IngestLimits 

Fields

  • ilMaxAdvisoryBytes :: Int

    The largest decompressed advisory JSON one zip entry may hold, in bytes. It bounds memory and decode cost.

Instances

Instances details
Show IngestLimits Source # 
Instance details

Defined in Ecluse.Core.Osv.Stream

Eq IngestLimits Source # 
Instance details

Defined in Ecluse.Core.Osv.Stream

defaultIngestLimits :: IngestLimits Source #

An 8 MiB per-advisory ceiling.

data IngestStats Source #

The running tally of one ingest pass, which systemicDrop reads once the stream completes.

Constructors

IngestStats 

Fields

Instances

Instances details
Show IngestStats Source # 
Instance details

Defined in Ecluse.Core.Osv.Stream

Eq IngestStats Source # 
Instance details

Defined in Ecluse.Core.Osv.Stream

data OsvIngest Source #

The context one ingest pass threads through the stream.

newOsvIngest :: MonadIO m => IngestLimits -> OsvEcosystem -> EpssScores -> UTCTime -> m OsvIngest Source #

A fresh ingest context with the given bounds, feed, EPSS table and clock, and a zeroed tally.

readIngestStats :: MonadIO m => OsvIngest -> m IngestStats Source #

Read the current drop tally.

resetIngestStats :: MonadIO m => OsvIngest -> m () Source #

Zero the tally before a retry re-streams, so it reflects only the final attempt.

systemicDrop :: IngestStats -> Bool Source #

Whether the drop tally requires Pilot to refuse publication.

newtype PilotIngestAborted Source #

Systemic drops or zero relevant rows refused the pass, whose tally this carries. Nothing replaces the last-good artifact.

What one attempt learned about its source

data OsvAttempt Source #

What one ingest attempt learned about its source. A retry replaces it whole, so it matches the tally beside it.

Constructors

OsvAttempt 

Fields

Instances

Instances details
Show OsvAttempt Source # 
Instance details

Defined in Ecluse.Core.Osv.Stream

Eq OsvAttempt Source # 
Instance details

Defined in Ecluse.Core.Osv.Stream

readOsvAttempt :: MonadIO m => OsvIngest -> m OsvAttempt Source #

Read what the current attempt learned about its source.

resetOsvAttempt :: MonadIO m => OsvIngest -> m () Source #

Forget the source metadata alongside resetIngestStats, so a failed attempt's dates never reach the artifact.