Implement event-based message fetching to replace cronJob in Discord Bot #106
Description
Description:
Currently, we are fetching Discord channel messages under two scenarios:
- When a user changes the guild period or selected channels.
- When a cronJob runs.
To improve our data currency and app efficiency, we propose to eliminate the cronJob and replace it with an event-based message fetching system (using messageCreate and messageUpdate events). This approach provides several benefits:
- Updated data: The data will be updated as soon as a message event occurs.
- Efficient processing: Unlike the cronJob which fetches channel messages for all guilds at the same time, the event-based system fetches messages only when they occur, reducing the load on the system.
- Accurate data: The messageUpdate event will ensure that any changes to messages are immediately reflected in our data.
The primary concern about the event-based system is the potential for high database interaction, which could be problematic if many events fire simultaneously. We need to consider an efficient way to handle these interactions with the MongoDB database without affecting performance.
Here are some possible solutions to address this concern:
- Batching: Collect the events and perform batch operations in the database instead of inserting or updating the records one by one.
- Queueing System: When an event is fired, instead of writing directly to the database, the data could be sent to the queue. A separate worker service can then read from the queue and write the data to the database.
- Caching: Use a caching layer (like Redis) where the events are temporarily stored and later written to the database.
Metadata
Metadata
Assignees
Labels
No labels