Major refactoring of the ChartDownloader code.

[batuakan]

1. Move away from file based caching to MBTiles based caching for better performance and reliability.
   1. Added an option to the webui to vacuum mbtiles to free disk space.
   2. Added an option to the webui to create a separate mbtiles file per region/chart combination.
   3. Added an button under region view to the webui to delete mbtiles files for selected regions/charts.
2. Estimate tile counts for progress reporting without needing to pre-generate tile lists.
3. Optimize memory usage during tile downloads. Fixed memory leak issue Plugin crashes when trying to start seeding job with too large number of tiles #48
4. Added possibility to load ChartProvider from a javascript file to be able to calculate dynamic URLs.
5. Caching of tiles stops if disk usage exceeds a certain threshold. Previously this was only checked for seeding jobs
6. Added a new endpoint for reporting cache stats for the current session for all chart providers.
7. Expanded existing cache related endpoints for more flexible usage.
8. Rewrite of the WebUI, with builtin map using leaflet,
   1. Ability to add/remove/edit regions/polygons/rectangles, and store them on the server side.
   2. Improve the download job management, with better status reporting and error handling.
   3. Added option to web ui to be able refetch tiles from remote server and update local cache. Support refreshing cached tiles / seeding jobs not preserved over restart #47
   4. More user friendly UI
   5. Better support for mobile devices
9. Various bug fixes and improvements.

Warning
This commit depends on @signalk/mbtiles library. Changes from PR #43 were copied manually

[tkurki]

Can you please rebase this on #43 ? Or I can do it for you and you continue from there.

I'd like to move forward so that we can merge #43 (that I just updated) and this and publish a new version.

[batuakan]

Can you please rebase this on #43 ? Or I can do it for you and you continue from there.

I'd like to move forward so that we can merge #43 (that I just updated) and this and publish a new version.

Hi @tkurki

Can you do it for me, I tried to base my solution on the mbtiles branch after our previous talk but i failed to do so

[tkurki]

Rebased and tests passing - you take it from here?

[batuakan]

Rebased and tests passing - you take it from here?
@tkurki Nicely done, I couldnt do this earlier because the native mbtiles branch needed to be merged into main before i could rebase my feature branch on it.

I am mostly happy with the code as it is just the modal welcome page/documentation needs to be improved. In the meanwhile can you run my feature branch and see the new features and fixes before we move on to actually reviewing the code. I am excited to hear your feedback on the new features

[tkurki]

Reload should not take you to the beginning: save chart viewport and the selected visible layers

The modal initial help is less useful than it could be, as when you first see it you don't know what the page is going to look like, so it is hard to understand what it is saying => make the help panel always initially visible at bottom left corner for example, with the ability to minimise (and remaximise afterwards)

Vacuuming is only useful if the mbtiles already exists and you choose to refresh also existing tiles, right? I would change the checkboxes to

•  use separate file for this region (default: single file per chart provider)
• refresh all tiles
• compact file after seeding

and refresh all tiles would be gray if there are no existing cached tiles, and compact would be gray unless you select refresh all. Vacuum is not end user friendly terminology....

The jobs button is really low key. It could show an (animated) indicator if there are jobs running.

Why can you start a job that is already completed? You can't change its parameters.

Start and stop buttons seem a little unresponsive - like I need to press them multiple times? We could implement a websocket mechanism for sending the status of the jobs instead of polling.

The Serif font in the help panel hurts my brain 😁

[torevalley]

Installed the branch and have a few questions comments.

First of all great effort and function, I really like the way your regions work especially in edit mode, real easy to e.g. follow a coastline.

• I would appreciate the possibility to name the region on creation and edit, not only in seed job creation. I now created my first region and did not name it. I then edited the region and did a new seed, where I named the region Öresund. That did not stick, the mbtiles file and the seedjob still says unnamed --> file is unnamed_eniro.mbtiles.

• Is there any plans to show the existing caches (I guess mbtiles will be the only choice in the near future). I think it is useful to be able to keep inventory and also use that to refresh the content as a seeding job every now and then assuming that the chart providers keep their chart up-to-date.

• I have a chart of type mapstyleJSON that is setup to be cached, it show up as a candidate for seeding job in the "main branch" but on this fork it does not show up. Will that type be supported?

• for the mapstyleJSON a max I use a zoom of 18 or 20 would be really useful (that chart itself has 24 as max but that is overdoing it), realizing quite a few tiles for that but if you are willing to spend the time and the diskspace is there...

• Agree with Teppo on the initial modal, maybe create a default map from the ones already seeded, I might be wrong but I guess most people will create off-line mbtiles mainly from one chart provider, that could eventually be pre-selected.

Please let me know if this is not the correct way to submit things, e.g. the first might be a bug with the regions --> maybe better as an issue.....

Hope I can help testing, so let me know of any special directions you want me to exercise :)

[tkurki]

@batuakan just to be clear - are you planning to address my feedback or do you expect this to be merged as is, then published as beta for testing or a release?

[batuakan]

Lets get this going again, and sorry responding late, was busy with other SK projects

@tkurki I can add a settings endpoint similar to freeboardsk, to save and load the viewpoint and the zoom and the active regions.

I added a new button on the bottom corner, but the contents of the help dialog is still a matter of procrastination :D

Changed the check boxes, for separate mbtiles creation, I should explicitly say in the documentation that first the tiles are cached into the global tile cache, once that is finished a new mbtiles is created and the tiles are copied to the new one. If you use the layer you want to seed there will definitely be some tiles in the cache for the area you are browsing.

I changed the icon on the jobs button, I can look into how to add a badge to the jobs button to show the number of ongoing jobs.

I will remove the start/stop button from the jobs view then, the user can cancel and remove a job by using the X button at the top right corner.

The buttons are bit unresponsive depending on when you click because at every poll to the server, the whole jobs view is discarded and recreated, but once we remove the buttons the situation should improve

Any font recommendations?

I don't think you would approve to merge this code before going through a strict code review :D. I am expecting many rounds of back and forth

[batuakan]

@torevalley

I am happy it worked for you.

• Region settings dialog pops up when you finish creating a region, I think it is the best time to name the region

• I wanted to have a feature like that, but i couldn't think of a good way to visualize that. Also if you can see the tile when you are online, then you have it in the cache, if you dont see the tile when you are offline then you dont have it in the cache :D.

• regarding the mapstyleJSON, apparently I filter out everything else other than tilelayer. Did the caching work nicely with mapstyleJSON? I wonder if it would cache into mbtiles file?

[torevalley]

About regions, I understand what you are saying but changing the name in edit still makes sense.
I created an unnamed region and it seems it cannot be named after creation.

On visualize regions, does the cache hold info on the region it was created from?
If the region is available and time for latest seed on that area could be stored in my head that could be visualized.
Another way could be to just persist the seed jobs per region and make if possible to display that list. Thinking that as a user you would know your regions.

Mapstyle json did not cache as tiles, but you can add it and check the proxy box. I have no clue whats happening behind the scenes with that type of source/provider. I was merely hoping it would be cachable :)

[batuakan]

Hi @tkurki

I added the help page as well, shall we begin the code review process?