Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ For each `[Job]` class (see `Templates/Job.sbntxt`), nested in the user's partia
- **`Invoker`**: a singleton `IJobInvoker` that deserializes the payload and runs the handler through
the Immediate.Handlers pipeline.
- **`JobDefinition`**: the runtime description (name, queue, attempts, timeout, backoff, cron,
overlap and misfire policies, concurrency).
overlap and misfire policies, and concurrency).
- **`PayloadJsonContext`**: hand-emitted System.Text.Json metadata for the payload and context
types, so serialization never uses reflection.

Expand All @@ -44,8 +44,11 @@ configuration surface (`ConfigureStorage`, `ConfigureWorkers`, `UseFairQueues`,

`JobSchedulingService` (`Internals/JobSchedulingService.cs`) is one hosted service per process:

- **Startup.** Initializes storage and submits the complete definition catalogue and code-defined
schedules through `MergeJobDefinitionsListAsync`, even when workers are disabled. The storage
reconciles the complete catalogue. Metadata has no invoker or CLR type.
- **Polling loop.** Each iteration materializes due recurring schedules, builds a
`JobAcquisitionRequest` from free worker capacity and queue/job concurrency limits, calls
`JobAcquisitionRequest` from local definitions, free worker capacity, and queue/job concurrency limits, calls
`IJobStorage.AcquireDueJobsAsync`, and writes the claimed records to an unbounded channel. On
`PurgeInterval` it also purges job and batch history. It then waits `PollingInterval`.
- **Workers.** `WorkerCount` tasks read the channel and run `ExecuteJobAsync`: a new DI scope, a
Expand Down
8 changes: 4 additions & 4 deletions docs/conventions.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,10 +47,10 @@ assembly's `LibraryEventIds`:

| Assembly | Range in use |
| --- | --- |
| `Immediate.Jobs.Shared` | 11000–11093 (scheduler, in-memory, single-server) |
| `Immediate.Jobs.EntityFrameworkCore` | 11500–11542 |
| `Immediate.Jobs.LinqToDB` | 11600–11642 |
| `Immediate.Jobs.Redis` | 11700–11728 |
| `Immediate.Jobs.Shared` | 11000–11097 (scheduler, in-memory, single-server) |
| `Immediate.Jobs.EntityFrameworkCore` | 11500–11544 |
| `Immediate.Jobs.LinqToDB` | 11600–11644 |
| `Immediate.Jobs.Redis` | 11700–11730 |

Append new ids after the current maximum of the assembly's range; never reuse or renumber an id,
because users filter on them. `EventName` is `Immediate.Jobs.{Assembly}.{Name}`.
Expand Down
7 changes: 4 additions & 3 deletions docs/recurring-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,10 @@ A `RecurringJobSchedule` has a unique `Name`, the `JobName` it materializes, a `
`TimeZoneInfo.FindSystemTimeZoneById`. The analyzers validate code-defined expressions at compile
time (`IJOB0007`).

At startup `JobSchedulingService.InitializeAsync` calls `MergeRecurringSchedulesListAsync` with every
code-defined schedule. Storage replaces the code-defined set (removing schedules for deleted jobs)
and leaves dynamic schedules untouched.
At startup `JobSchedulingService.InitializeAsync` submits all definitions and their code-defined schedules
through `MergeJobDefinitionsListAsync`. Storage atomically reconciles the catalogue, removing code-defined schedules when a definition disappears or loses its cron expression. It preserves
dynamic schedules. Unchanged cron/time zone retains the next occurrence; updates
preserve the pause state and last occurrence.

## Materialization

Expand Down
48 changes: 45 additions & 3 deletions docs/storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
Two interfaces in `src/Immediate.Jobs.Shared/Storage`:

- **`IJobStorage`**: queueing, acquisition (including fair-queue policies), leases, completion and
failure, recurring schedules, wait-for-trigger updates and triggers, queries, monitoring snapshots,
failure, definition catalogues, recurring schedules, wait-for-trigger updates and triggers, queries, monitoring snapshots,
heartbeats, purging. Every provider implements all of it.
- **`IJobGraphStorage : IJobStorage`**: batches, continuations, dynamic additions from running jobs,
batch triggers, batch reads and purging, plus the two single-server recovery reads
Expand All @@ -30,6 +30,45 @@ monitoring snapshots and health-check data.
is preserved through settlement and dynamic additions, and `BatchState.WaitingForTrigger` is
non-terminal.

## Definition reconciliation

`MergeJobDefinitionsListAsync(JobDefinitionRegistration)` receives the complete local definition list,
and all code-defined schedules. It inserts or updates supplied definitions and removes stored
definitions absent from that list. An empty list clears the catalogue.
Definition deletion never cascades to invocations, running leases, or execution history.

Definition names are case-insensitive identities and retain their readable spelling. In-memory
and single-server use `OrdinalIgnoreCase`; SQL Server inherits database collation; PostgreSQL uses
a nondeterministic ICU collation (`und-u-ks-level2`); SQLite uses `NOCASE` (ASCII-only database
comparison). EF tracking uses an ordinal case-insensitive key comparer. EF applications with custom
provider conventions can pass `definitionNameCollation` to `AddImmediateJobs`.
Definition names must not have leading or trailing whitespace: the job analyzer reports `IJOB0008`,
and storage rejects invalid manually supplied names before modifying the catalogue.

Relational providers store definition configuration in individual columns in
`immediate_job_definition_metadata`; the name key is the only persisted definition name.
`TimeoutTicks` and `BackoffBaseTicks` store exact .NET ticks (100 nanoseconds), and policy enums
use small integers. Redis continues storing serialized definition records.

Concurrent startup snapshots commit atomically. In-memory uses its gate; EF Core takes an early write
lock on a singleton catalogue row. LinqToDB uses a serializable transaction and merges supplied
records directly into each table on SQL Server and PostgreSQL 15 or later. Matched updates preserve
canonical names and recurring progress using target values. SQL Server removes obsolete rows in the
merge; PostgreSQL uses a separate set-based delete in the same transaction because PostgreSQL 15 has
no WHEN NOT MATCHED BY SOURCE clause. Only SQLite reads a snapshot for client-side reconciliation; it uses transactional upserts and deletes because
it does not support SQL `MERGE`. Serialization conflicts retry the complete LinqToDB snapshot.
Redis compares a version and applies metadata and
schedule changes in one Lua script, recomputing after a competing startup. Single-server delegates
metadata reads and reconciliation to durable storage, then refreshes the primary's code-defined
schedules. Pause and unchanged schedule progress survive startup.

Each application's list is authoritative for the catalogue. Concurrent applications with inconsistent
lists can remove or replace each other's definitions; the last successful reconciliation wins.

Custom providers must implement the catalogue operations. Relational users must add the definition metadata table
through their normal schema update process; EF Core also requires the singleton catalogue table. The
bootstrap helpers do not upgrade existing production databases.

## Providers

| Provider | Location | Concurrency idiom |
Expand All @@ -51,7 +90,8 @@ back a job transition.

Tables: `immediate_jobs`, `immediate_job_executions`, `immediate_job_continuations`,
`immediate_job_batches`, `immediate_recurring_jobs`, `immediate_job_servers`, and
`immediate_fair_queue_groups`.
`immediate_fair_queue_groups`, and `immediate_job_definition_metadata`. EF Core also uses
`immediate_job_definition_catalog` to serialize catalogue updates.

- **EF Core**: applications call `modelBuilder.AddImmediateJobs()` and own their migrations. A schema
change is a breaking change for every EF Core user, and needs a migration note in the pull request.
Expand All @@ -75,7 +115,9 @@ may build keys from the root):
| `leases` | sorted set | Active job ids by lease expiry. |
| `executions:index:{id}` / `executions:data:{id}` | sorted set / hash | Execution history. |
| `recurring:*` | various | Schedules, due index, names, materialization de-duplication. |
| `server:{worker}` / `servers` | hash / sorted set | Heartbeats. |
| `server:{worker}` / `servers` | hash / sorted set | Scheduler heartbeats. |
| `definition-metadata` | hash | All definition metadata keyed by stable job name. |
| `definition-catalog-version` | string | Version used to atomically reconcile a complete catalogue. |
| `fair:*` | various | Fair-queue indexes, active counts, and cursors; see [fair queues](fair-queues.md). |

The payload lives only inside `record`. Lua must not re-encode it with `cjson` (key order and escaping
Expand Down
7 changes: 3 additions & 4 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,8 @@

**Guidance for adding tests:**

- **Storage behaviour belongs in the conformance suite**, not in provider-specific tests, so every
provider is held to it. Write a provider-specific test only for behaviour that exists in that
provider alone.
- **Every storage behaviour test belongs in the conformance suite**, so every provider is held to
the same contract. Do not write standalone or provider-specific storage tests.
- **Functional tests cover only what the layer above storage adds**, for example a scheduler turning
`TryTriggerAsync`'s `false` into an exception, or argument validation. Don't re-assert storage
semantics there.
Expand All @@ -25,7 +24,7 @@
`src/Immediate.Jobs.Testing/Storage` ships in the `Immediate.Jobs.Testing` package, so custom
providers can run it too.

- **Case files.** `QueueStorageConformance`, `RecurringStorageConformance`,
- **Case files.** `QueueStorageConformance`, `DefinitionCatalogStorageConformance`, `RecurringStorageConformance`,
`FairQueueStorageConformance`, `TriggerStorageConformance`, `GraphStorageConformance`, and
`ReplicaStorageConformance`.
- **Case shape.** Each case is `new(Name, RequiredCapabilities, Scenario[, PersistedJobState])`. Names
Expand Down
9 changes: 9 additions & 0 deletions readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,15 @@ public sealed class SignupService(SendWelcomeEmail.Scheduler welcomeEmail)
Register the generated handlers and jobs methods in `Program.cs`, in that order. For an assembly named `MyApp`, these
are `AddMyAppHandlers()` and `AddMyAppJobs()`.

## Persisted definitions

Startup provides the complete local catalogue to storage, which inserts, updates, and removes job
definitions and code-defined schedules. The dashboard shows every stored definition, including those
absent from its local registrations.

Job names match case-insensitively. Explicit job names must not have leading or trailing whitespace; the analyzer reports `IJOB0008`
for invalid names. Storage also rejects such names in manually supplied definition catalogues.

## Scheduling and handles

Generated schedulers use one method name for relative and absolute scheduling. Pass a `TimeSpan` for a delay or a
Expand Down
Loading
Loading