Make entry points/inferred Windows subsystem conform better with conventions

[squeek502]

Previously, start.zig would always export wWinMainCRTStartup as the entry symbol, regardless of whether main or wWinMain was used as the main. This meant that the linker was unable to differentiate between main/wWinMain and so anything with wWinMainCRTStartup would get the .console subsystem inferred. In other words, the added test would fail for winmain_inferred as it would have the .console subsystem instead of the expected .windows subsystem.

This commit changes the relevant logic:

• Only export wWinMainCRTStartup when using wWinMain
• Export wmainCRTStartup for non-wWinMain main functions when libc is not linked
• When libc is linked, then the libc runtime entry point is used and main is exported (this part hasn't changed)
• Infer .windows subsystem when exporting WinMainCRTStartup/wWinMainCRTStartup/WinMain/wWinMain
• Infer .console subsystem when exporting main/mainCRTStartup/wmainCRTStartup

This is a breaking change in the sense that a compiler built before this commit will be unable to compile a non-wWinMain exe using the updated start.zig. This is because the previous code did not look for mainCRTStartup/wmainCRTStartup at all, so it'd fall back to assuming the entry point is wWinMainCRTStartup which is no longer exported when using a normal main and therefore hit error: lld-link: <root>: undefined symbol: wWinMainCRTStartup

---

These are some changes I had sitting around from the time of #17808 that I've updated and added a standalone test for. Ultimately this is some fairly minor convenience stuff that mostly allows users to not have to specify subsystem explicitly when defining wWinMain, so the breaking nature of this change may not be worth it.

cc @castholm

[castholm]

I've not invested any time into investigating this and am mainly going off quickly reading the MSVC linker docs, but I'm wondering if the way Zig decides the subsystem and entry point is correct:

https://learn.microsoft.com/en-us/cpp/build/reference/subsystem-specify-subsystem?view=msvc-170#remarks

The /SUBSYSTEM option specifies the environment for the executable.

The choice of subsystem affects the entry point symbol (or entry point function) that the linker will select.

I also found this blog post from Raymond Chen:

https://devblogs.microsoft.com/oldnewthing/20241004-00/?p=110338

If you did not specify the /DLL flag, then the linker looks at your /SUBSYSTEM flag. If you asked for /SUBSYSTEM:CONSOLE, then it looks for wmain and main. If you asked for /SUBSYSTEM:WINDOWS, then it looks for wWinMain and WinMain. If you didn’t specify a subsystem, then it looks for all four symbols, and whichever symbol it finds first determines what the implied /ENTRY is:

If linker finds	Then entry point is
wmain	wmainCRTStartup
main	mainCRTStartup
wWinMain	wWinMainCRTStartup
WinMain	WinMainCRTStartup

This reads to me that if no subsystem is specified by the user, it should be .console if any w?main symbol is exported, otherwise .windows if any w?WinMain main symbol.

Then, if the resolved subsystem is .console, the entry symbol should be wmainCRTStartup or mainCRTStartup, otherwise if it's .windows it should be wWinMainCRTStartup or WinMainCRTStartup. (I have no idea what the convention is for other subsystems or if there is any.) Currently the order in src/link/Lld.zig appears to be the other way around, prioritizing the w?WinMain entrypoints.

Also, from a quick test, the MSVC linker behavior seems to be to fail the link with LNK1561: entry point must be defined or LNK1221: a subsystem can't be inferred and must be defined if the above logic doesn't resolve any entry point/subsystem.

[castholm]

I wonder if always exporting the default Zig main as wmainCRTStartup might be a problem in some instances. If the user does zig build-exe foo.zig --subsystem windows, linking would fail under the MSVC linker behavior laid out in my above comment.

Is it possible to export the same symbol under two names, wmainCRTStartup and wWinMainCRTStartup, without any negative side effects? This way it would work regardless of which subsystem is specified at the final link.

[squeek502]

Yeah, the "subsystem determines the entry point" thing of the MSVC tooling is something we discussed here and that I promptly forgot about.

Will have to think more about this. A framing that might be helpful is thinking about "what do we want to work?" and let the behavior be dictated by that, i.e. stuff like:

• Do we want an object file with a main function built by Zig to be able to be linked by MSVC tooling without explicitly specifying the entry point symbol?