initial sqlite implementation

[nathancahill]

Gets things started with node:sqlite (in LTS since v23/Oct 16, 2024).

[jamescmartinez]

Thanks for getting this started!

We definitely need a SQLite backend. There are a few changes I need to make to the Backend interface which I should have wrapped up this weekend or early next week. Once that lands, I'll come back and update this PR on top of the new interface so everything lines up.

In the meantime, I'll have copilot do a quick pass, and I'll get a full review in as well.

Really appreciate you putting this together so quickly, especially without Backend docs!

[Copilot]

Pull Request Overview

This PR introduces an initial SQLite backend implementation for OpenWorkflow using Node.js's built-in node:sqlite module (stabilized in v22.5.0). The implementation provides an alternative to the existing PostgreSQL backend for lighter-weight deployments and testing scenarios.

Key Changes:

• Adds a new @openworkflow/backend-sqlite package with full Backend interface implementation
• Implements SQLite-specific database initialization, migrations, and query patterns
• Includes comprehensive test suite covering workflow runs, step attempts, deadlines, and cancellations

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
packages/backend-sqlite/package.json	Package configuration for the new SQLite backend module
packages/backend-sqlite/tsconfig.json	TypeScript compilation configuration extending from Node 22 and strictest presets
packages/backend-sqlite/index.ts	Main entry point exporting BackendSqlite class
packages/backend-sqlite/sqlite.ts	Core SQLite utilities including database setup, migrations, and helper functions for date/JSON conversion
packages/backend-sqlite/backend.ts	Backend interface implementation with workflow run and step attempt operations
packages/backend-sqlite/backend.test.ts	Comprehensive test suite covering all backend operations and edge cases

[laz-001]

There are a few changes I need to make to the Backend interface which I should have wrapped up this weekend or early next week.

there should be some library that handles multi-db (e.g. use common set of fields, automatic use of additional functionality, optional migration etc.). Should be worth the effort to check.

[jamescmartinez]

there should be some library that handles multi-db (e.g. use common set of fields, automatic use of additional functionality, optional migration etc.). Should be worth the effort to check.

I'm really focused on efficiency & performance for this framework, and nothing matches what the lower level drivers can achieve. A bit more (ok a lot more) work to maintain, but better in prod.

[laz-001]

A bit more (ok a lot more) work to maintain, but better in prod.

ok, then the basic thing would be to use the "common denominator" re db functionality, at least in early dev days, so more or less the same code works for postgres/sqlite and potentially others you'll use.

(my main interest is that if i use this sw, then i should be able to switch/upgrade easily from sqlite to postgres, without 'drama').

[nathancahill]

@laz-001 you'd be able to swap backends without "drama" as you describe:

Before:

import { BackendSqlite } from "@openworkflow/backend-sqlite";
import { OpenWorkflow } from "openworkflow";

const sqlitePath = process.env.DATABASE_PATH;
const backend = await BackendSqlite.connect(sqlitePath);
const ow = new OpenWorkflow({ backend });

After:

import { BackendPostgres } from "@openworkflow/backend-postgres";
import { OpenWorkflow } from "openworkflow";

const postgresUrl = process.env.DATABASE_URL;
const backend = await BackendPostgres.connect(postgresUrl);
const ow = new OpenWorkflow({ backend });

If you wanted to migrate actively running workflows from one backend to another then you'd have to write custom code for that, but it would be possible as the schemas are largely the same.

[laz-001]

you'd be able to swap backends without "drama" as you describe:

would prefer a generic way, like:

• if there's a db URL in env, instantiate postgres
• otherwise look for db PATH, instantiate sqlite

but the manual code-switch would be fine, too, in the end it's just an helper that i could provide myself.

so all good!

[mariusflorescu]

@jamescmartinez i also believe that what @laz-001 is saying is probably a better way of handling different dbs would be something like this:

1. All backends in the same package (probably rename those to adapters)
2. Use a string adapter adapter in the openworkflow class like: adapter: "postgres". So the "backend" would become something generic and will handle connection and all of this stuff based on the adapter + there won't be issues since all adapters will implement a specific interface.

This would provide less friction since users don't need to install specific package based on the db used.

There also should be no issue with bundle size since unused adapters won't ever get bundled (three-shaking).

[nathancahill]

I could see that being an alternate configuration for 2-3 built-in backends. The existing method of configuration would still work for power users, but passing a string would use a backend with default configuration:

Zero config

Possibly defaults to sqlite backend so no external service is needed to start using OpenWorkflow:

const ow = new OpenWorkflow();

Backend string option

Looks for OPENWORKFLOW_BACKEND_DB environment var:

const ow = new OpenWorkflow({ backend: 'postgres' }); // or { backend: 'sqlite' }

Backend option (current)

Customize backend with other options (like namespaceId) or use community backend:

const backend = await BackendPostgres.connect(postgresUrl);
const ow = new OpenWorkflow({ backend });

As far as tree-shaking (relevant to serverless environments), the backend string is a runtime config and wouldn't be able to tree-shake. But the backends are very minimal so not a big concern.

[jamescmartinez]

created a new issue for integrating the most popular backends as suggested above: #47