[Docs] Restructuring Documentation and Re-classifying public APIs

[Tobotimus]

Some stuff on our public developer documentation, which is aimed at cog creators, should really only be documented for internal use and subsequent to changes at any time. This is following on from a bit of a rant of mine over in #2774.

What should be moved?

As I mentioned there, with respect to APIs which are already documented, candidates for moving to these internal docs would be:

• The Downloader API and cog reference
• The drivers API reference
• The cog manager API reference
• A bit of the data manager API reference (creating instances, storage details and all that)
  • Things like cog_data_path() should obviously remain public
  • Things like load_basic_configuration() should be documented internally but we should reconsider how our testing framework handles stuff like this.

Obviously there are lots of things we could add to these docs, which don't already exist in the public docs but would be useful in these internal docs. But I'm not setting any milestones for adding that stuff right now, because it can be quite a lot of work.

Where should these internal docs be located?

On the same RTD site is fine, just under a new section. The sections are currently a bit muddled up - I think we need to redefine them as four clear sections:

• Installation and Setup Docs (stuff done on the host system)
• User Docs (stuff done through Discord, @retke has done some great work here with [V3] User documentation #1757 and [Docs] Getting started guide #2659)
• Cog Developer Docs (stuff available for use by 3rd party devs)
• Red Developer Docs (what this issue is about)

[mikeshardmind]

I don't mind if we have a section of the docs which is strictly for internal Red use, but I think there's a larger issue doing this somewhat masks.

We have a lot of things which are exposed publicly. If we don't want developers using them and relying on them, we should be following the conventions for this and marking them private.

We can't do this in a micro release based on our versioning.

Right now, I'm a little more lax on breaking this for things which aren't documented, but for anything which is documented and exposed publicly which we don't want to be locked into a design on, we should properly mark them private in the next minor release.

[Tobotimus]

I agree this shouldn't be done in micro releases, only minor.

The problem is that Python doesn't have a package-protected scope, only module/class/subclass protected. So I'd prefer leaving them as "public" to other internal Red modules, but not as part of our APIs for 3rd party cogs.

[Tobotimus]

Like I know that encapsulation in Python is only really by convention with leading underscores, but I would argue that a single leading underscore on a module-level function implies module-level protection...

[mikeshardmind]

I can put a draft PR up later today for modification and use with 3.2 which may make it a bit more clear the level of changes I'd be arguing for here.

I'm fine with a separate section of the docs for any fully internal use classes/modules so long as those classes/modules are properly documented as not for consumer use in the main section, but anything where we would have mixed parts, the convention method and not splitting up the docs makes more sense to me.

[mikeshardmind]

I think the parts of this we were most concerned about for 3.2 are handled at this point, but I don't think this is fully resolved yet.