Skip to content

WIP content-sqlite: preallocate space to avoid outside diskfull events #6217

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
chu11 wants to merge 11 commits into
base: master
Choose a base branch
from
Open

WIP content-sqlite: preallocate space to avoid outside diskfull events #6217

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
bc19543
test: add 5m tmpfs
chu11 Aug 13, 2024
d6ce14b
content-sqlite: split truncate out of function
chu11 Aug 7, 2024
1f95a3d
content-sqlite: initialize variables on close db
chu11 Aug 13, 2024
ffa16c3
content-sqlite: open db given input parameters
chu11 Aug 14, 2024
cdbea4b
content-sqlite: support preallocate config
chu11 Aug 7, 2024
f366ae4
t: cover content-sqlite preallocate
chu11 Aug 13, 2024
a0e7c60
content-sqlite: disable journaling if hit ENOSPC
chu11 Aug 13, 2024
82eb65e
t: cover preallocate with journaling
chu11 Aug 15, 2024
816fae5
doc: document new content-sqlite config
chu11 Aug 15, 2024
297057f
fixup! content-sqlite: support preallocate config
chu11 Aug 20, 2024
059d388
test
chu11 Aug 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion doc/Makefile.am
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -363,7 +363,8 @@ MAN5_FILES_PRIMARY = \
man5/flux-config-ingest.5 \
man5/flux-config-kvs.5 \
man5/flux-config-policy.5 \
man5/flux-config-queues.5
man5/flux-config-queues.5 \
man5/flux-config-content-sqlite.5


MAN7_FILES = $(MAN7_FILES_PRIMARY)
Expand Down
53 changes: 53 additions & 0 deletions doc/man5/flux-config-content-sqlite.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
=============================
flux-config-content-sqlite(5)
=============================


DESCRIPTION
===========

The Flux system may configure one of several possible content backends for
storing data to long term disk. The ``content-sqlite`` module is one
such backend, using ``sqlite`` for storage.

The ``content-sqlite`` table may contain the following keys:


KEYS
====

journal_mode
(optional) Configure the sqlite journal mode. By default is set to ``WAL``
if a flux statedir is configured, otherwise defaults to ``OFF``. Legal options
are ``DELETE``, ``TRUNCATE``, ``PERSIST``, ``MEMORY``, ``WAL``, and ``OFF``.

synchronous
(optional) Configure the sqlite synchronous mode. By default is set to ``NORMAL``
if a flux statedir is configured, otherwise defaults to ``OFF``. Legal options
are ``EXTRA``, ``FULL``, ``NORMAL``, ``OFF``.

preallocate
(optional) Specify a number of bytes to preallocate to the sqlite database. This
can ensure the module can continue to function even if the disk has been full
due to other operations on the node (e.g. system logging has filled up disk). Using
preallocated space requires journaling to be disabled, which will automatically be
done if the disk is out of space and the preallocate setting is configured.


EXAMPLES
========

::

[content-sqlite]
journal_mode = "WAL"
synchronous = "NORMAL"
preallocate = 10737418240


SEE ALSO
========

:man5:`flux-config`,
`sqlite <https://www.sqlite.org/>`_,
`sqlite pragmas <https://www.sqlite.org/pragma.html>`_
Comment on lines +52 to +53
Copy link
Member

@garlick garlick Aug 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might want to put those URLs in a RESOURCES section like most of the other section 5 man pages.
The URLs are spelled out so you can cut and paste them from the man command output.
For example, in flux-config(5):

RESOURCES
=========

.. include:: common/resources.rst

Flux Administrator's Guide: https://flux-framework.readthedocs.io/projects/flux-core/en/latest/guide/admin.html

TOML: Tom's Obvious Minimal Language: https://toml.io/en/

7 changes: 7 additions & 0 deletions doc/test/spell.en.pws
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -919,3 +919,10 @@ DEEPBIND
PRELOAD
RTLD
dlopened
WAL
backend
backends
journaling
pragmas
preallocate
preallocated
4 changes: 3 additions & 1 deletion etc/rc1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,9 @@ if test $RANK -eq 0; then
rm -f ${dumplink}
fi
elif test "${backingmod}" != "none"; then
flux module load ${backingmod}
date > /tmp/timing
flux module load ${backingmod} preallocate=50000000000
date >> /tmp/timing
fi
fi

Expand Down
Loading