Skip to content

Run indexer functions in worker thread #132

@gmaclennan

Description

@gmaclennan
  • Separate worker thread that decodes hypercore blocks and writes to SQlite table via @mapeo/sqlite-indexer
  • One worker for all projects
  • sqlite in main thread needs to open db as read-only
  • needs to communicate progress
  • possibly use thread-stream for back pressure
    • would need multi-core indexer and sqlite-indexer to have stream interface.

Unrelated notes about background tasks in Android

  • background tasks in android
    • needs to be something that starts and finishes (bc android stuff)
    • use case
      • start sync, then move to different apps
      • make connections, start stuff in background
    • problem: can't have multithreaded access to a hypercore. how do we enable this but also have the main process hypercore working
    • RAF
      • from app process, only need write access to your own hypercores
      • only need read access to indexed ones (not your own)
      • in sync process, theoretically never need to write to your own hypercores
        • exception is if you cleared some of your own data (which could be applicable to blob store via clearing images)
  • could have both main process = reading and writing your own hypercores, sync process = indexing, but challenge is keeping those two tasks in sync
  • like the idea of sync being something that can be separated out from the rest of the code

Metadata

Metadata

Labels

post-mvpde-scoped to after MVP

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions